PaperlayerView exact revision on arXiv

Source-linked AI summary

Can Sensitive Information Be Deleted From LLMs? Objectives for Defending Against Extraction Attacks

Vaidehi Patil, Peter Hase, Mohit Bansal

arXiv:2309.17410v1cs.CLcs.AIcs.LG

TL;DR

Sensitive information in language models raises safety and privacy concerns, motivating direct deletion from model weights rather than merely suppressing outputs. The paper introduces an attack-and-defense framework with a candidate-budget threat model and evaluates whitebox and blackbox extraction. It finds that edited facts remain recoverable, while defenses reduce some attacks but none works universally, making reliable deletion tractable but difficult.

  • Problem

    LLMs can retain sensitive information and harmful knowledge, but it is unclear how to delete specific facts and verify deletion against prompt and whitebox extraction.

  • Method

    The paper directly edits model weights and evaluates deletion using whitebox and blackbox attacks under a threat model where recovery among B candidates counts as success.

  • Results

    Deleted facts remained extractable, with attacks succeeding as high as 89% in some experiments; defenses reduced some attacks but did not provide universal protection.

  • Takeaways & Limitations

    Truly deleting sensitive information is tractable but difficult, with potentially severe implications for deploying LLMs where privacy and safety matter.

  • Takeaways & Limitations

    The paper does not find a universally effective defense, and aggressively editing to resist paraphrasing can cause undesirable damage to model knowledge.

Abstract

from arXiv · show

Pretrained language models sometimes possess knowledge that we do not wish them to, including memorized personal information and knowledge that could be used to harm people. They can also output toxic or harmful text. To mitigate these safety and informational issues, we propose an attack-and-defense framework for studying the task of deleting sensitive information directly from model weights. We study direct edits to model weights because (1) this approach should guarantee that particular deleted information is never extracted by future prompt attacks, and (2) it should protect against whitebox attacks, which is necessary for making claims about safety/privacy in a setting where publicly available model weights could be used to elicit sensitive information. Our threat model assumes that an attack succeeds if the answer to a sensitive question is located among a set of B generated candidates, based on scenarios where the information would be insecure if the answer is among B candidates. Experimentally, we show that even state-of-the-art model editing methods such as ROME struggle to truly delete factual information from models like GPT-J, as our whitebox and blackbox attacks can recover "deleted" information from an edited model 38% of the time. These attacks leverage two key observations: (1) that traces of deleted information can be found in intermediate model hidden states, and (2) that applying an editing method for one question may not delete information across rephrased versions of the question. Finally, we provide new defense methods that protect against some extraction attacks, but we do not find a single universally effective defense method. Our results suggest that truly deleting sensitive information is a tractable but difficult problem, since even relatively low attack success rates have potentially severe societal implications for real-world deployment of language models.

1 INTRODUCTION

LLMs can contain sensitive information and generate harmful text, creating safety and information hazards. The introduction asks how to delete specific sensitive information and test whether deletion succeeded, while contrasting direct deletion with output-focused approaches such as RLHF.

  • LLMs may contain memorized personal information, harmful knowledge, outdated facts, and toxic beliefs, creating safety and information hazards.
  • The paper asks how to delete specific sensitive information from language models and how to test whether deletion succeeded.
  • Scrubbing Sensitive Info From LLM Outputs: RLHF and RLAIF are predominant approaches for suppressing sensitive outputs, but models can remain vulnerable to adversarial prompts and retain the underlying information.
  • Scrubbing Sensitive Info From LLM Outputs: Removing sensitive information from training data may require difficult, expensive retraining processes, motivating alternatives that operate directly on models.

1. Notice sensitive info

The paper frames sensitive-information deletion as an attack-and-defense problem: direct weight editing should block both prompt-based and whitebox extraction, but experiments show that deleted facts remain recoverable.

  • Notice sensitive info: Directly editing model weights is proposed to remove sensitive information, satisfy potential privacy standards, and protect against whitebox extraction from public model weights.
  • Notice sensitive info: The threat model counts deletion as incomplete when a sensitive answer can be recovered among B candidates, reflecting password attempts, parallel attacks, or regulatory demands.
  • Whitebox Attacks: 38% of deleted answers were recovered by whitebox attacks with B = 20, while extending the objective to intermediate representations reduced this rate to 2.4%.
  • Blackbox Attacks: 29% of blackbox attacks succeeded with B = 20; data augmentation did not improve the paraphrasing defense unless the model was aggressively edited.
  • Findings: The paper concludes that deletion is tractable but difficult, with potentially severe implications for deploying LLMs where privacy and safety matter.

2 RELATED WORK

Related work distinguishes targeted extraction and model editing from broader privacy attacks and machine unlearning. The paper adopts model editing because it changes particular outputs while aiming to preserve general model knowledge.

  • Prior work extracts training data broadly, whereas this paper targets specific factual information by starting from a question whose answer it seeks to recover.
  • The paper’s attacks extract factual information, unlike membership inference attacks, which test whether particular samples appeared in training data.
  • Machine Unlearning and Model Editing: Machine unlearning usually removes a training example’s influence, whereas sensitive-information deletion concerns undesirable outputs elicited by otherwise harmless prompts.
  • Machine Unlearning and Model Editing: Model editing changes outputs for selected inputs and is adopted because it has shown promising performance with minimal model damage.
  • Removing information only from forward-pass representations is insufficient here because the goal is permanent deletion from model weights.

3 PROBLEM STATEMENT

The paper formalizes sensitive-information deletion as adversarial extraction under whitebox or blackbox access, balancing removal of one fact against preservation of broader model knowledge. Attack success is measured by whether the answer appears within a candidate set of size B.

  • The framework evaluates deletion of a single undesired fact through adversarial attacks and formal metrics for attack success and model damage.
  • Adversary’s Objective: An extraction succeeds when answer A for question Q appears in candidate set C, whose size B is the attack budget.
  • Adversary’s Objective: The candidate-set threat model covers password verification, parallel malicious action, and legal demands that information be unobtainable within B candidates.
  • Adversary’s Capabilities: Whitebox attackers access model weights, architecture, forward passes, and hidden states, whereas blackbox attackers submit inputs and receive randomly sampled outputs.
  • Deletion must remove the specific target while avoiding general knowledge damage, evaluated with random and neighborhood accuracy changes.
  • Attack Success Metric: The paper reports AttackSuccess@B and accuracy changes separately because estimating a risk-performance tradeoff requires domain-specific cost-benefit analysis.
  • The Rewrite Score measures how much editing changes the target probability relative to the desired change, with 1 indicating perfect maximization and 0 no change.

4 ATTACK METHODS

The paper develops whitebox and blackbox attacks that recover supposedly deleted information from intermediate representations or rephrased prompts. These attacks construct candidate sets from logit-lens distributions, layer-to-layer probability changes, or sampled outputs for paraphrases.

  • Whitebox attacks: Whitebox attacks use the logit lens to probe intermediate Transformer representations for traces of information absent from final generated text.The logit lens converts intermediate hidden states into vocabulary distributions, enabling extraction before information disappears at later layers.
  • Whitebox attacks: The Head Projection Attack collects the top-k tokens from logit-lens distributions across selected layers, with budget B = k|L|.The attack targets single-token answers and tunes k and the layer set L under a maximum candidate budget.
  • Whitebox attacks: The Probability Delta Attack ranks tokens by probability changes between consecutive layers, selecting those that rise or fall most sharply.The candidate set can combine top-k, bottom-k, or both sets while maintaining a fixed candidate budget of |C| = 20.
  • Blackbox attack: input rephrasing: The blackbox attack samples model outputs for automatically generated paraphrases of the edited input.With R rephrases and S samples per rephrase, its candidate budget is |C| = RS.

5 DEFENSE METHODS

The paper evaluates objectives that suppress sensitive answers in final outputs, inject alternative answers, or remove traces from intermediate representations. New defenses target whitebox attacks through layer-wise hidden-state objectives and target blackbox attacks by editing across generated rephrases.

  • Existing objectives: The Empty Response Defense maximizes the probability of an empty target string, using either “I don’t know” or “dummy” instead of the original knowledge.The main experiments use “dummy,” which performs better than “I don’t know.”
  • Existing objectives: Fact Erasure minimizes the probability of the original factual answer for the edited input.This objective directly suppresses p(y|x; M) for the original fact (x, y).
  • Existing objectives: Error Injection maximizes an alternative false target, but it is not applicable when the model should avoid wrong answers to sensitive questions.The method is included to demonstrate the efficacy of injecting false information into the model.
  • Whitebox defenses: The Head Projection Defense uses a max-margin loss to keep the deleted answer outside the top-k tokens across selected intermediate layers and the final distribution.The defense tunes the layer set L to improve performance, with m denoting the margin term.
  • Whitebox defenses: The Max-Entropy Defense maximizes the entropy of each layer’s logit-lens distribution over the next token.This objective is applied across layers to make intermediate predictions less informative.
  • Blackbox defenses: The Input Rephrasing Defense adds model-generated paraphrases to the editing objective and deletes the information for the original input and its rephrases.It is designed to counter blackbox attacks that exploit incomplete generalization across paraphrases.

6 EXPERIMENT SETUP

Experiments use publicly available language models and two factual question-answer datasets chosen to test targeted information deletion. The study compares popular editing methods, with additional constrained-finetuning results reported in the appendix.

  • Models: Experiments use GPT-J, Llama-2, and GPT2-XL because of their widespread usage, public availability, and capacity for memorizing pretraining data.Results for Llama-2 and GPT2-XL are reported in Appendix B.
  • Datasets: Evaluation uses CounterFact and zsRE, which provide factual prompts or question-answer pairs and alternative false targets for model editing.CounterFact also supplies neighboring datapoints for Neighborhood ∆-Acc, while zsRE contains short Wikipedia-derived questions.
  • Datasets: The datasets are used because single-token factual answers make it easy to verify whether deleted information appears in candidate sets.This provides a concrete testbed for targeted information deletion rather than abstract sensitive-information descriptions.
  • Model editing methods: The main editing methods are ROME and MEMIT, both of which update specific MLP weight matrices in Transformer layers.For a single fact, ROME updates one MLP layer, whereas MEMIT updates multiple layers.
  • Model editing methods: Constrained finetuning is evaluated additionally in Appendix B but performs worse than ROME and MEMIT.

7 EXPERIMENT RESULTS

The experiments show that edited GPT-J models remain vulnerable to whitebox and blackbox extraction attacks, while proposed defenses reduce some attacks but do not provide universal protection.

  • 7.1 CAN WE EXTRACT A “DELETED” ANSWER FROM A LANGUAGE MODEL?: 38% attack success at B = 20 shows that ROME with the conventional Empty Response objective leaves “deleted” facts vulnerable to extraction.The experiments use GPT-J and CounterFact with three attacks: Head Projection, Probability Delta, and Input Rephrasing.
  • 7.1 CAN WE EXTRACT A “DELETED” ANSWER FROM A LANGUAGE MODEL?: 29% blackbox attack success and 18% Probability Delta success at B = 1 show that extraction can succeed with small candidate budgets.All methods appear to saturate after B = 20 candidates.
  • 7.2 HOW TO DEFEND AGAINST INFORMATION EXTRACTION ATTACKS: 89% attack success against MEMIT with Empty Response on zsRE demonstrates that a standard deletion objective can fail to fully remove facts.The result is reported for the Head Projection attack with B = 20.
  • 7.2 HOW TO DEFEND AGAINST INFORMATION EXTRACTION ATTACKS: Max-Entropy and Head Projection are the strongest defenses against whitebox attacks, with Max-Entropy reducing Head Projection success from 22.2% to 1.7% on CounterFact and from 41.4% to 2.9% on zsRE with ROME.These reductions are measured relative to the Fact Erasure baseline.
  • 7.2 HOW TO DEFEND AGAINST INFORMATION EXTRACTION ATTACKS: The Input Rephrasing defense does not reduce blackbox attack success when attack paraphrases differ from those used in the editing objective.More aggressive edits lower attack success but sharply increase Δ-Acc damage to Random and Neighborhood data.
  • 7.3 CAN WE DEFEND AGAINST UNFORESEEN EXTRACTION ATTACKS?: No single defense protects against all attacks: Max-Entropy lowers whitebox Probability Delta success to 2.4% with ROME on zsRE, but blackbox success remains 28% on CounterFact and 19% on zsRE.The Probability Delta attack is an unforeseen attack for defenses designed against Head Projection.

8 CONCLUSION

The paper concludes that model editing is a promising route for deleting sensitive information, but even edited models can reveal supposedly deleted facts under small-budget attacks. Truly deleting sensitive information is therefore tractable but difficult, with potentially severe implications for model deployment.

  • 8 CONCLUSION: Model editing is presented as the most promising approach to deleting sensitive information, yet edited models can still reveal facts under small verification budgets.The conclusion reports extraction rates as high as 89% in some experiments.

ETHICS STATEMENT

The paper frames sensitive-information deletion as an ethically consequential problem and reports tuning choices for its attack and defense procedures.

  • ETHICS STATEMENT: The paper warns that deploying LLMs capable of influencing humans may have negative moral and legal consequences.It leaves broader implications for AI, ethics, and law to future work.
  • A TUNING DETAILS: For fixed B = 20, attack hyperparameters are tuned on a separate development set of 100 samples.The tuning allocates the candidate budget across attack-specific parameters.
  • A TUNING DETAILS: The Head-Projection attack performs best for GPT-J with top-4 candidates across five tuned layers.The selected layers are 17, 18, 19, 20, and 21.
  • A TUNING DETAILS: The Probability Delta attack performs best with top-2 candidates across ten layers, using different layer ranges for GPT-J and GPT2-XL.The GPT-J layers are 8–9 and 16–23, while GPT2-XL uses layers 36–45.
  • A TUNING DETAILS: The blackbox defense uses five model-generated paraphrases, while whitebox defenses extend from attack layers through subsequent layers to the final layer.These choices are described as defense-layer and paraphrase selections.

B ADDITIONAL EXPERIMENTS

Additional experiments evaluate ROME and constrained finetuning across Llama2-7B, GPT2-XL, and GPT-J, including attack success as a function of budget.

  • B.1 LLAMA2-7B: ROME is less effective on Llama2-7B than on GPT-J under the reported experimental setup.The authors modify ROME hyperparameters for Llama2-7B to obtain reasonable rewrite score and delta accuracy.
  • B.1 LLAMA2-7B: Table 3 reports attack success rates for three attacks on CounterFact after deleting information from Llama2-7B with defended ROME.The table evaluates ROME augmented with the defense strategies.
  • B.2 GPT2-XL: GPT2-XL experiments report attack success across three editing methods and defense strategies, including its dependence on attack budget.Figure 5 shows the budget relationship, while Table 4 reports GPT2-XL results.
  • B.3 CONSTRAINED FINETUNING: Constrained finetuning optimizes the same singular MLP weight matrix edited by ROME under an ℓ∞-norm constraint.The optimization uses the Adam optimizer.
  • B.3 CONSTRAINED FINETUNING: Table 5 reports attack success rates for GPT-J after constrained finetuning with the defense strategies.The table concerns CounterFact facts edited using constrained finetuning.

C.1 MODEL EDITING METHODS

The paper compares model-editing methods and describes how its attack and evaluation procedures operationalize information deletion and extraction.

  • C.1 MODEL EDITING METHODS: ROME edits a specific MLP layer with a rank-one update derived by treating the MLP weight as a linear associative memory.Its default objective maximizes the probability of a desired output with input data augmentation and regularization.
  • C.1 MODEL EDITING METHODS: MEMIT spreads an update over multiple MLP layers when editing a single fact, unlike ROME’s single-layer update.For GPT-J, MEMIT updates layers 5–7.
  • C.1 MODEL EDITING METHODS: Rewrite Score measures the fraction of the possible desired change achieved in the target probability.For minimization objectives, it becomes 1 − p(y|x; M*)/p(y|x; M).
Loading 2309.17410v1…