PaperlayerView exact revision on arXiv

Source-linked AI summary

Extracting Forgotten Prompts from Targeted Unlearned Models

Au Ashley Hoi-Ting, Meghdad Kurmanji, William F. Shen, Nicholas D. Lane, Ligang He

arXiv:2609.03662v1cs.LG

TL;DR

Existing attacks typically assume forgotten prompts are known and recover their answers, but forgotten prompts themselves can remain discoverable through behavioral traces. TAS uses retained prompts and black-box access to identify forgotten entities and reconstruct prompts, achieving high recovery across multiple settings while using far fewer queries than exhaustive probing.

  • Problem

    Existing attacks assume the adversary already knows forgotten prompts, leaving the problem of discovering those prompts themselves insufficiently addressed; prompt content may encode sensitive information.

  • Method

    TAS is a black-box active-search method that uses retained prompts and behavioral traces to identify forgotten entities and reconstruct forgotten prompts under a limited query budget.

  • Results

    100% entity identification and up to 95% forgotten-prompt reconstruction were achieved across three datasets, three model families, and three unlearning methods using only a small fraction of exhaustive probing.

  • Takeaways & Limitations

    Successful answer suppression does not imply target concealment, so conventional answer-suppression metrics can overlook leakage of forgotten prompts.

  • Takeaways & Limitations

    Perfect ranking does not guarantee accurate multi-entity recovery; remaining errors require determining when to terminate the ranked candidate list.

Abstract

from arXiv · show

Recent unlearning methods (e.g. NPO, DPO, LUNAR) make use of refusal alignment to suppress forgotten data. However, it has been shown that refusal responses might leave traces of unlearning, and recent attacks have been able to successfully recover some of the unlearned knowledge. In this paper, we uncover a new vulnerability. Existing attacks typically assume that the forgotten prompts are already known to the adversary and focus on recovering their answers. However, we show that the forgotten prompts themselves can be extracted by using the retained data and black-box access to the model. Our attack, Targeted Active Search (TAS), first identifies the forgotten entities by constructing canonical templates and entity pool, and selectively querying the model using the most informative template-entity pair under a limited query budget. Once the entities are identified, TAS instantiates prompt templates with those entities to probe the unlearned model and reconstruct the forgotten prompts. Experiments across three unlearning methods with three datasets and three LLMs shows that TAS recovers the forgotten entity with $100\%$ accuracy and reconstructs up to $95\%$ of forgotten prompts, all while using up to $99.7\%$ fewer queries than naive probing.

1 Introduction

The paper identifies forget-prompt discovery as a black-box vulnerability overlooked by attacks that assume forgotten prompts are known. It introduces TAS to recover forgotten entities and prompts from retained data and refusal traces under a limited query budget.

  • Existing attacks assume the adversary already knows the forgotten prompt and recover its answer rather than discovering the prompt itself.
  • Forgotten prompts can encode sensitive clinical or legal associations, making prompt recovery a distinct privacy and security risk even when answers remain suppressed.
  • TAS constructs candidate entities and reusable templates from retained prompts, then searches their combinations using black-box refusal behaviour.
  • 100% entity identification and up to 95% forgotten-prompt reconstruction were achieved across three unlearning methods, model families, and benchmarks.The reported experiments used DPO, NPO, and LUNAR; Llama2-7B, Llama3-8B, and Gemma-7B; and PISTOL, DUSK, and TOFU.
  • TAS uses up to 99.7% fewer queries than exhaustive probing while recovering hidden forgotten prompts with high precision and recall.
  • Conventional forget-set metrics fail to predict black-box discoverability, motivating evaluations that test whether unlearning makes forgotten prompts recoverable.

2 Threat Model

The threat model gives the attacker retained prompts and black-box query access but withholds the forget set, original model, internals, and implementation. The goal is to identify forgotten entities and reconstruct their associated prompts under a finite budget.

  • The attacker seeks to identify forgotten entities and construct the associated forgotten prompt set under a limited query budget B.
  • Forgotten targets may be single entities or ordered entity tuples, with ordering relevant when the underlying relation is directional.
  • Success requires the identified entity to match the true forgotten entity and the constructed prompt set to match the true forget prompt set.
  • The adversary can issue arbitrary black-box queries, observe decoded text outputs, and repeat interactions subject to a finite budget.
  • The attacker has retained prompts but lacks the forget set, pre-unlearned model, model internals, and unlearning implementation.

3 Targeted Active Search

TAS turns forget-prompt recovery into structured adaptive search over entities and templates. It builds reusable candidates, scores refusal-like outputs, maintains posterior states, warms up coverage, and then adaptively probes promising combinations.

  • 3 Targeted Active Search: TAS alternates between identifying likely forgotten entities and finding retained templates that recreate forgotten prompts from black-box outputs.
  • 3 Targeted Active Search: The pipeline comprises structured search construction, posterior maintenance, warm-up coverage, adaptive active search, and forgotten-prompt reconstruction.
  • 3.1 Candidate construction: Retained questions yield candidate entities and reusable templates with ordered placeholders, separating substitutable content from query structure.
  • 3.1 Candidate construction: TOFU canonicalizes semantically duplicated templates, whereas DUSK and PISTOL templates are canonical by construction.
  • 3.1 Candidate construction: Each query instantiates a template t with an ordered entity tuple z, with one slot for DUSK and TOFU and two for PISTOL.
  • 3.2 Response scoring: TAS observes only decoded completions and scores them using a lexical refusal detector plus a semantic similarity score against reference refusals.
  • 3.3 Posterior states: TAS maintains Beta posteriors over slot-wise entities and templates, updating them with refusal scores so evidence accumulates directionally.
  • 3.4 Coverage before adaptation: Warm-up queries broadly cover the search space before Thompson sampling adaptively selects promising template-entity combinations.

4 Experimental Setup

The experiments evaluate TAS across unlearning benchmarks, models, and search baselines, using a staged search procedure and metrics for entity recovery, prompt reconstruction, and query efficiency.

  • Benchmarks and models: Experiments use DUSK, TOFU, and PISTOL, where forgotten data are created by splitting selected entities’ records into retain and forget sets.
  • Benchmarks and models: TAS is evaluated with DPO, NPO, and LUNAR across Llama2-7B, Llama3-8B, and Gemma-7B.
  • Search methods: Baselines include brute-force, Random, Greedy, and UCB, differing in exhaustive coverage, posterior guidance, exploitation, or confidence-adjusted refusal scoring.
  • TAS procedure: TAS constructs entities and canonical templates from retained prompts, initializes posterior beliefs, performs warm-up coverage, then adaptively probes entity-template pairs.
  • TAS procedure: TAS completes adaptive search with confirmation of shortlisted entities, cardinality estimation, and prompt reconstruction by instantiation and refusal-based ranking.
  • Evaluation metrics: Evaluation asks whether TAS identifies forgotten entities, reconstructs forget prompts, and searches efficiently, reporting entity accuracy and MRR alongside prompt recall and precision.
  • Evaluation metrics: Query efficiency is measured by queries used, normalized cost, hit rate, and first hit to capture discovery cost under noisy refusal evidence.

5 Results

TAS identifies forgotten entities reliably across datasets and unlearning settings, then reconstructs forgotten prompts with high recall under substantially reduced query budgets. Its gains arise from posterior-guided search, canonical templates, and strategic evidence accumulation, while multi-entity stopping remains a limitation.

  • Entity Identification: 1.000 match accuracy and MRR: TAS identifies the forgotten entity exactly across DUSK, PISTOL, and TOFU, with zero variance across models and unlearning methods.No baseline matches this performance across all three datasets.
  • Entity Identification: TAS reaches the true forgotten entity and ranks it first in every PISTOL run, unlike Random and Greedy, which are often misled by collateral refusals.Strategic revisiting accumulates enough refusal evidence to separate the target from plausible distractors.
  • Prompt Reconstruction: 0.861, 0.954, and 0.873 prompt recall: TAS recovers most of the forget set on DUSK, PISTOL, and TOFU, respectively.On PISTOL, TAS exceeds Brute-force prompt recall, 0.954 versus 0.843.
  • Prompt Reconstruction: Unlearning methods leave different reconstruction footprints: LUNAR has higher precision but lower recall, whereas DPO and NPO have higher recall but lower precision.LUNAR reports 0.799 precision and 0.780 recall; DPO and NPO report recall of 0.966 and 0.943, respectively.
  • Discoverability: 0.000 Forget ROUGE-1 on NPO with PISTOL does not prevent TAS from reconstructing 96.1% of the corresponding forgotten prompts.Answer suppression and prompt-target concealment therefore measure different security properties.
  • Query Efficiency: 5.49% of PISTOL queries and 0.6% of TOFU queries suffice for perfect entity identification, while TAS uses 5,000 queries instead of TOFU’s 804,138-query brute-force space.All baselines receive the same budget, but TAS benefits from concentrating probes on candidates accumulating consistent refusal evidence.

6 Related Work

Prior work studies prompt discovery, privacy inference, and recovery after unlearning, but generally assumes a known target or candidate pool. This paper instead frames forgotten-target discovery as an open black-box search problem.

  • Existing prompt-discovery research spans mining, optimization, LLM-based search, evolutionary methods, and security-oriented extraction.
  • Unlearning benchmarks such as PISTOL and DUSK model relations or overlap between forget and retain data, but typically assume knowledge of the forget target.
  • Membership inference, dataset inference, and FUMA remain verification problems because the possible target or candidate pool is supplied.
  • Prior recovery attacks use black-box querying or model comparisons to revive or extract forgotten information, rather than discover an unknown forget target.
  • This work studies an adversary with no forget set, target-containing candidate pool, pre-unlearning model, or model internals.

7 Discussion

The discussion distinguishes target leakage from answer leakage and analyzes when TAS remains effective, where it is limited, and what defenses would require. It argues that secure unlearning must conceal the intervention target while preserving retained utility.

  • Target leakage remains possible even when unlearning suppresses the forgotten answer, because behavioral changes can reveal what was targeted.
  • Strong answer-suppression scores do not guarantee target concealment; DPO, NPO, and LUNAR produce different reconstruction footprints.
  • Collateral refusals make search noisier, but TAS can separate the true target by accumulating and revisiting evidence.
  • Broadening refusals is insufficient as a defense because narrow changes reveal targets while broad changes create collateral refusals and harm retained behavior.
  • A stronger defense would seek behavioral indistinguishability between forgotten and ordinary entity-template combinations while preserving answer suppression and retained utility.
  • TAS depends on retained prompts providing sufficient coverage of entities and prompt structure surrounding the forgotten target.
  • The attack assumes text-only black-box access and therefore targets unlearning methods that produce observable changes in generated text.
  • Increasing the number of forgotten entities makes cardinality estimation less reliable, especially in the denser TOFU search space.

8 Conclusion

The paper concludes that forgotten prompts can remain discoverable through refusal traces even when answers are suppressed, and presents TAS as a black-box method for exposing this target leakage. Its algorithm combines posterior-guided search with confirmation and template handling, while motivating behavioral indistinguishability as a security goal.

  • TAS achieves 100% entity identification and reconstructs up to 95% of forgotten prompts across three datasets, model families, and unlearning methods.
  • TAS uses retained prompts and black-box outputs to identify forgotten entities and reconstruct prompts from behavioral traces.
  • Each entity-template pair maintains a Beta posterior updated by probes, with posterior means used to rank candidates for probing.
  • Refusal scores are binarized into refusal, compliance, or no-evidence outcomes using threshold rules before posterior updates.
  • Because refusals identify multi-slot tuples rather than individual members, positive credit is divided by soft responsibility across entities.
  • Compliance evidence updates entity candidates but not template utility, while indiscriminately refusing templates lose sampling weight.
  • Confirmation uses equal-allocation mean refusal scores over shared templates to make candidate comparisons comparable after unequal Thompson sampling.

B Unlearning Methods

The paper compares three unlearning mechanisms: preference-based suppression in DPO and NPO, and representation redirection in LUNAR. Despite these differences, all three can induce distinctive behavioral changes on forgotten examples.

  • DPO: DPO increases abstention likelihood while decreasing the likelihood of the forgotten answer for forget-set prompts.It constructs preference pairs that treat the original answer as rejected and an abstention response as preferred.
  • NPO: NPO suppresses the forgotten response directly without specifying a preferred replacement response.Its adaptive weighting reduces further updates as the forgotten response becomes less likely than under the reference model.
  • LUNAR: LUNAR redirects forget-set activations toward representations associated with the model expressing ignorance.It optimizes activation matching while restricting updates to a single MLP down-projection matrix at a selected layer.
  • LUNAR: LUNAR reference prompts need only evoke ignorance and need not contain information related to the forget set.Examples can involve fictitious or otherwise unknown entities.
  • Comparison: DPO specifies both the suppressed response and its replacement, NPO specifies only the response to reduce, and LUNAR changes internal representations.The three methods therefore implement forgetting through distinct output-preference or representation-level mechanisms.

C Experimental Details

The evaluation uses strict and topical LLM judges to determine whether reconstructed prompts match forgotten prompts. The paper uses topical matching as the default criterion.

  • Judging protocol: Llama3-8b serves as the judge for whether each reconstructed prompt belongs to the true forget set.The judge evaluates reconstructed prompts against the reference forgotten prompts.
  • Judging protocol: The strict judge requires the same information and the same specific person, while the topical judge permits different facts about the same person and topic.A different person or unrelated topic is not considered a match under the topical criterion.
  • Judging protocol: The topical judge is the paper’s default evaluation criterion.It credits candidate prompts that concern the same specific person and underlying topic, even when the exact fact differs.

C.2 Standard forget-set metrics

The standard forget-set metrics quantify residual overlap with forgotten answers and the model’s residual confidence in producing them. Lower values indicate stronger forgetting under conventional evaluation.

  • Metrics: Forget ROUGE-1 measures recall overlap between the model output and the reference forgotten answer.Higher overlap means the response still resembles the forgotten answer, whereas lower overlap indicates greater suppression.
  • Metrics: Forget probability measures the likelihood assigned to the forgotten answer, capturing residual confidence even without verbatim generation.Lower forget probability and lower Forget ROUGE-1 jointly indicate stronger forgetting under conventional evaluation.

C.4 Ablation Study: Query Budget

The query-budget ablation shows that TAS reaches stable recovery performance well before the maximum budget on all datasets. Confirmation improves reliability beyond posterior ranking alone, while prompt precision remains lower than recall.

  • Budget sensitivity: TAS performance rises to a maximum at a fraction of the budget ceiling and remains flat afterward on every dataset.Figure 4 reports entity identification in the left column and prompt reconstruction in the right column across query budgets.
  • Budget sensitivity: DUSK reaches Accuracy and MRR of 1.000 from B = 100 onward, making its B = 500 ceiling substantially larger than required.This is the earliest saturation point reported across the datasets.
  • Budget sensitivity: PISTOL Accuracy increases from 0.33 at B = 100 to 0.89 at B = 200 and plateaus near 1.000 by B ≈300.The plateau occurs well inside the 1000-query ceiling.
  • Early stopping: Early stopping triggers near the stabilization points: DUSK at 285, PISTOL at 868, and TOFU at 2488.The paper relates these stopping points to posterior concentration and the point where further queries no longer improve recovery.
  • Confirmation phase: Posterior-only ranking is less reliable than the confirmed pipeline, with PISTOL remaining below 0.75 exact-match and TOFU below the confirmed curve.The persistent gap indicates that confirmation resolves ambiguity among near-indistinguishable high-refusal candidates rather than merely adding queries.
  • Prompt reconstruction: Prompt reconstruction follows similar thresholds but occurs slightly later and at lower performance than entity identification.Prompt recall saturates near the entity-recovery threshold, while precision plateaus lower.

D.3 Training- and knowledge-gap disclaimers

The section defines refusal-detection rules for training- and knowledge-gap disclaimers, answer avoidance, and canned refusals. It also adds an optional embedding-based score and suppresses false positives from ordinary factual absence statements.

  • Training- and knowledge-gap disclaimers: Training- and knowledge-gap disclaimers detect claims that the model lacks information, access, training, or knowledge.These patterns include explicit statements about limited resources, databases, datasets, or scope.
  • Partial, mitigated competence claims: Partial competence claims capture hedged statements such as being not well versed or not the best source.The H2 pattern limits intervening text to at most 25 characters without sentence-final punctuation.
  • Suppression rules: Answer-avoidance detection distinguishes ordinary third-person factual absence from genuine unlearning refusal language.A factual absence example scores 0 unless it also contains first-person refusal language or an explicit inability verb.
  • Embedding channel: The refusal score combines regular-expression matches with optional embedding similarity to 21 canonical refusals.Cosine similarity is rescaled from [0.3,0.7] to [0,1], clipped, and combined by taking the maximum score.
Loading 2609.03662v1…