Knowledge Sanitization of Large Language Models

• Web-trained LLMs can expose personal and confidential information through extraction attacks, with leakage risks increasing as models scale.Source: §1 · p. 1
• Existing privacy and unlearning approaches can conceal confidential knowledge but may generate substitute information, hallucinations, or other secondary harms.Source: §1 · pp. 1–2
• Knowledge sanitization replaces targeted answers with predefined harmless phrases such as “I don’t know” rather than alternative factual content.Source: §1 · p. 2, Figure 1 · p. 2
• The method selectively fine-tunes MLP layers of pretrained LLMs, allowing deployment without retraining the full model.Source: §1 · p. 2
• Experiments with LLaMA and GPT-J show consistent harmless responses for targeted knowledge, preserved performance on other domains, and robustness against extraction attacks.Source: §1 · p. 2

2.1 Preliminaries

• Knowledge pairs split a prompt prefix x<t from its subsequent answer sequence x≥t, enabling the method to distinguish targeted knowledge from its context.Source: §2.1 · pp. 2–3

2.2 Method

• Sanitization replaces target answers in KF with safe phrases in KS, while combining KS with KR prevents unnecessary sanitized responses on retained knowledge.Source: §2.2 · p. 3
• LoRA updates only MLP weight matrices because these layers are implicated in storing and modifying factual associations, reducing trainable parameters while targeting knowledge.Source: §2.2 · p. 3

2.3 Dataset

• The evaluation uses TriviaQA as a closed-book dataset, with separate sets for forgotten knowledge, sanitized targets, and retained knowledge.Source: §2.3 · pp. 3–4
• Ten dataset constructions vary the forgetting targets by changing the random seed, and the retention set is sized proportionally to the forgetting set.Source: §2.3 · p. 4

3.1 Experimental Setup

• Sanitization used LoRA on MLP layers across LLaMA and GPT-J, with comparisons against Negative Gradient, Negative Task Vector, ROME, and fine-tuning baselines.The evaluation covered closed-book TriviaQA plus common-sense reasoning and reading comprehension benchmarks.Source: §3.1 · p. 5

3.2 Main Results: Comparison on Task Performance

• TriviaQA forgetting-target accuracy fell from 74% to 7% for LLaMA after sanitization, while retention accuracy remained 49.9% versus 49.8%.Sanitization without KR reduced retention accuracy to 11.8%, showing the importance of mixing forgetting and retaining data.Source: §3.2 · p. 6
• Performance remained nearly unchanged on common-sense reasoning and reading comprehension tasks after sanitization.These benchmarks are multiple-choice, so their preserved performance may be easier to maintain than generation performance on TriviaQA.Source: §3.2 · p. 6

3.3 Leakage Rate in Entire Generation

• The entire-generation leakage rate for forgetting targets was approximately 8% while retention-target performance remained maintained.This evaluation checks whether forgotten answers appear anywhere in the full generated text, rather than only in the extracted answer span.Source: §3.3 · p. 6

3.4 Quality of Generated Texts

• Sanitization changed perplexity little on WikiText-2, whereas Negative Gradient increased perplexity and reduced generation quality.The comparison evaluates generation quality using perplexity before and after sanitization.Source: §3.4 · pp. 6–7, Table 4 · p. 7

4.1 Categorization of LM Outputs

• The evaluation classifies outputs as correct answers, sanitization phrases, or other responses potentially involving hallucinations.Examples include the confidential address, “I don’t know,” and an alternative address respectively.Source: §4.1 · p. 7

4.2 Results

• About 80% of forgetting-target outputs became sanitization phrases, while retaining-target correct answers remained stable and category-C outputs decreased.The three categories distinguish correct answers, the predefined phrase, and other outputs that may involve hallucinations.Source: §4.2 · p. 7, §4 · p. 7
• ROME replaced almost all outputs with sanitization phrases for both forgetting and retaining targets, indicating poor knowledge retention.The comparison concludes that simple knowledge replacement is insufficient for jointly achieving forgetting and retention.Source: §4.2 · p. 7

5.1 Experimental Setup

• Table 5 categorizes TriviaQA outputs as correct answers, sanitization phrases, or other outputs including hallucinations, with values averaged over five runs.This evaluation measures whether designated forgetting and retaining targets are replaced by the intended sanitization phrases.Source: Table 5 · p. 8
• The experiment applies automatically generated adversarial prompts about Julius Caesar, Calpurnia, and Cleopatra, alongside control prompts about Agrippina the Younger and Pompeii.A separate prompt tests Cleopatra in contexts unrelated to Julius Caesar.Source: §5.1 · p. 8

5.2 Results

• Sanitization significantly mitigates extraction of Julius Caesar-related information, producing “I don’t know” responses or blanks instead of specific details.The model remains cautious even when prompts contain contextually rich sentences that could otherwise elicit exploitable information.Source: §5.2 · p. 8
• Sanitization preserves accurate information about other contexts, including Cleopatra and Pompeii, without compromising the sanitization process.The results indicate a balance between withholding targeted knowledge and retaining broader knowledge-generation ability.Source: §5.2 · p. 8

6 Conclusion

• Knowledge sanitization generates predefined harmless phrases for prompts seeking sensitive or confidential information, reducing the potential for data leakage.The approach is presented as a method for enhancing security and reliability during knowledge extraction.Source: §6 · p. 8
• The approach addresses risks from confidential information entering training data through web texts, user inputs, and later-generation model datasets.The conclusion emphasizes that training sources are not limited to web text and may incorporate sensitive user information.Source: §6 · p. 8
• Adversarial examples collected during the research process are proposed as a basis for developing more robust sanitized LLMs in the future.This future direction responds to the potential vulnerabilities associated with sensitive information in training data.Source: §6 · p. 8
• The study characterizes knowledge sanitization as a step toward more secure and reliable LLM deployment aligned with safety and responsibility.Source: §6 · p. 8

Appendix

• Table 7 evaluates accuracy as the proportion of KR instances mixed into sanitization training data changes while the number of KS instances remains fixed.Source: Table 7 · p. 12
• Table 8 reports LLaMA output distributions for different sanitization phrases across correct answers, sanitization phrases, and other outputs including hallucinations.The table also identifies original-model results with “Orig.”Source: Table 8 · p. 13