PaperlayerView exact revision on arXiv

Source-linked AI summary

Precise In-Parameter Concept Erasure in Large Language Models

Yoav Gur-Arieh, Clara Suslik, Yihuai Hong, Fazl Barez, Mor Geva

arXiv:2505.22586v2cs.CL

TL;DR

Removing undesirable pretrained knowledge matters because harmful, irrelevant, or legally protected generations can hinder utility and raise safety and legal risks. PISCES disentangles and edits concept-related directions in model parameters. It improves specificity and robustness over existing methods while maintaining or slightly improving efficacy.

  • Problem

    Undesirable pretrained knowledge can produce harmful, irrelevant, or legally protected generations that hinder utility and introduce safety and legal risks.

    Source: §1 · p. 1

  • Method

    PISCES disentangles MLP parameters into features, identifies target-concept features using vocabulary projection, and ablates them from the parameters.

    Source: §1 · p. 2, §4.2 · p. 5

  • Results

    PISCES achieves 5%–31% higher specificity and 28%–38% greater robustness than state-of-the-art approaches, while maintaining or slightly improving efficacy.

    Source: §1 · p. 2, §7 · p. 9

  • Takeaways & Limitations

    The results establish in-parameter erasure as a state-of-the-art approach for fine-grained and robust conceptual knowledge removal in LLMs.

    Source: §7 · p. 9

    The current implementation targets only MLP parameters, and its SAE-based disentangler can erase only concepts captured as features while introducing imperfect reconstructions.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Large language models (LLMs) often acquire knowledge during pretraining that is undesirable in downstream deployments, e.g., sensitive information or copyrighted content. Existing approaches for removing such knowledge rely on fine-tuning, training low-rank adapters or fact-level editing, but these are either too coarse, too shallow, or ineffective. In this work, we propose PISCES (Precise In-parameter Suppression for Concept EraSure), a novel framework for precisely erasing entire concepts from model parameters by directly editing directions that encode them in parameter space. PISCES uses a disentangler model to decompose MLP vectors into interpretable features, identifies those associated with a target concept using automated interpretability techniques, and removes them from model parameters. Experiments on Gemma 2 and Llama 3.1 over various concepts show that PISCES achieves modest gains in efficacy over leading erasure methods, reducing accuracy on the target concept to as low as 7.7%, while dramatically improving erasure specificity (by up to 31%) and robustness (by up to 38%). Overall, these results demonstrate that feature-based in-parameter editing enables a more precise and reliable approach for removing conceptual knowledge in language models.

1 Introduction

PISCES targets conceptual knowledge in model parameters because existing erasure methods can damage unrelated capabilities or leave knowledge recoverable. It disentangles and edits concept-related features, achieving gains in specificity and robustness over existing methods.

1 Introduction

  • PISCES addresses the need to remove undesirable pretrained knowledge, such as information that creates safety or legal risks in deployment.The introduction gives a child-facing chatbot that should not discuss guns as an example.Source: §1 · p. 1
  • Existing concept-unlearning methods can damage semantically related concepts and general capabilities, while erased knowledge may return through adversarial prompting or fine-tuning.These shortcomings motivate more precise and robust conceptual erasure.Source: §1 · pp. 1–2
  • PISCES locates concept-related directions in model parameters, identifies target features with vocabulary projection, and ablates them from MLP parameters.The method uses a disentangler to separate MLP parameters into fine-grained features and implements the disentangler with sparse autoencoders.Source: §1 · p. 2
  • PISCES achieves 5%–31% higher specificity and 28%–38% greater robustness than state-of-the-art approaches.The paper also reports that PISCES slightly outperforms existing methods in efficacy.Source: §1 · p. 2
  • PISCES’s success depends on the disentangler identifying coherent concept-related features; stronger disentanglers could further improve erasure performance.Source: §1 · p. 2

2 Related Work

The related work distinguishes conceptual erasure from attribute removal and fact editing, and contrasts PISCES’s parameter edits with fine-tuning and activation steering.

Concept erasure

  • Representation-based erasure typically removes linearly decodable attributes or biases, whereas this work targets conceptual knowledge in model parameters.Source: §2 · p. 2

Knowledge editing

  • Fact-editing methods can be difficult to apply to whole concepts because they would require enumerating and editing every relation in which the concept appears.The paper gives Uranium as an example.Source: §2 · p. 2

Concept unlearning

  • Concept-unlearning methods often fine-tune on forget-sets, which can affect unrelated parameters and leave knowledge intact; SAE steering can also degrade coherence, incur computational overhead, and fail to persist under white-box threat models.PISCES instead edits concept-encoding parameters directly.Source: §2 · p. 3

3 In-Parameter Concept Erasure

The paper defines concept erasure as removing a target concept’s knowledge while retaining other knowledge and capabilities. Because concepts may be distributed across or entangled within MLP vectors, PISCES disentangles features to edit concept-specific directions.

Problem setup

  • The goal is to prevent a transformer language model from generating correct information about a target concept while retaining other knowledge and capabilities.A concept is treated as a human-understandable group of features, examples, or words sharing a property and localizable in model representations.Source: §3 · p. 3

Erasure approach

  • Individual MLP vectors can be polysemantic, so a concept may span multiple vectors or be entangled within one, limiting precise removal without harming unrelated capabilities.Source: §3 · p. 3
  • PISCES addresses this entanglement by disentangling neurons into fine-grained, interpretable features and removing target-concept directions across neurons.Source: §3 · p. 4

PISCES

PISCES applies feature-level editing to concept-bearing MLP parameters: it identifies target features, selects vectors that activate them, and reconstructs edited vectors in place.

4.1 Framework

  • PISCES identifies concept features, finds MLP vectors activating them, ablates those features, and reconstructs the edited vectors in place.Source: §4.1 · p. 4, Figure 3 · p. 4

4.2 Implementation

  • PISCES implements its disentangler as a sparse autoencoder and applies SAEs trained on MLP outputs directly to MLP vectors to preserve alignment with the training subspace.Source: §4.2 · p. 4
  • Vocabulary projection ranks feature vectors by their unembedding logits; automated token-density selection across layers is followed by manual filtering.Source: §4.2 · p. 5
  • A per-feature activation threshold selects strongly activating MLP vectors, because editing all vectors can hurt specificity and coherence.Source: §4.1 · p. 4, §4.2 · p. 5
  • For selected vectors, PISCES sets target-feature activations to negative scaled values, then reconstructs each edited vector and replaces it in place.Source: §4.2 · p. 5

5 Experiments

The experiments compare PISCES with four concept-erasure methods on two models using efficacy, specificity, coherence, and relearning-robustness evaluations. PISCES achieves a stronger efficacy-specificity balance and lower relearning accuracy than the compared methods.

5.1 Experimental Setting

  • Efficacy measures target-concept question answering, while specificity tests general and similar-domain knowledge and coherence tests instruction following and text coherence.Efficacy uses 50 open-style questions; specificity includes MMLU and similar-domain questions, while coherence uses 50 Alpaca-Eval tasks.Source: §5.1 · pp. 5–6

5.2 Results

  • 7.7% retained accuracy on Llama and 14.3% of original accuracy on Gemma accompany PISCES's stronger efficacy-specificity balance; Gemma similar-domain accuracy is 84.1%.The reported results also show higher specificity and coherence than the baselines.Source: Table 1 · p. 6, Figure 4 · p. 6, §5.2 · p. 7

5.1 Experimental Setting

  • Evaluation covers eleven concepts in Gemma-2-2B-it and Llama-3.1-8B-it.The table reports results averaged across questions and normalized so that 100% equals baseline performance, with 95% confidence intervals.Source: Table 1 · p. 6, §5.1 · p. 7
  • Robustness is evaluated by fine-tuning edited models on concept-related text without evaluation answers, then measuring whether efficacy recovers.The evaluation excludes adversarial attacks because preliminary tests found their effect negligible.Source: §5.1 · p. 6
  • PISCES is compared with RMU, ELM, MEMIT, and AlphaEdit using 100-configuration hyperparameter sweeps selected on validation data disjoint from testing.The evaluation also includes a supervised difference-in-means disentangler counterpart.Source: §5 · p. 5, §5.1 · p. 7

5.2 Results

  • In Gemma, PISCES retains 14.3% of original accuracy while maintaining 84.1% similar-domain performance and near-perfect MMLU and AlpacaEval scores; other methods show poorer tradeoffs, including 30% similar-domain and 8% MMLU drops.Source: §5.2 · p. 7
  • 51.5% relearning accuracy for PISCES versus 85.4% for the next-best Gemma method; on Llama, PISCES reaches 65.4% versus 93.2%–103.1% for other methods.Relearning fine-tunes edited models on concept-related text that excludes evaluation answers.Source: §5.1 · p. 6, §5.2 · p. 7

6 Analysis

The analysis relates disentangler feature quality and quantity to erasure outcomes, and compares computational costs across methods. PISCES is reported to be substantially more efficient when applied to multiple concepts.

6.1 Effect of Disentangler Performance on Erasure Success

  • Feature alignment is negatively correlated with erasure accuracy (−0.72, p-value 0.01), and selected-feature count is negatively correlated with MMLU performance (−0.64, p-value 0.03).The analysis uses the final selected feature set in Gemma-2-2B-IT.Source: Figure 5 · p. 8, §6.1 · p. 8
  • Features with high alignment and coherence tend to yield better evaluation performance, while concepts with more selected features often have lower MMLU and Alpaca scores.The authors suggest accumulated reconstruction error as a likely explanation for the lower scores associated with more selected features.Source: §6.1 · p. 8
  • PISCES performs best when the disentangler identifies precise, coherent features, while its performance declines when such features are absent.The examples given are golf, Republic of Ireland, and baseball for strong features, and Uranium when they are absent.Source: §6.1 · p. 8

6.2 Computational Efficiency

  • PISCES requires 5 · 10^14 FLOPs for Gemma and 1.1 · 10^15 FLOPs for Llama; across multiple concepts, it becomes 1–2 orders of magnitude more efficient than other methods.The comparison excludes SAE training, which is a significant additional cost, although training can be done once rather than per concept.Source: §6.2 · p. 8, Table 2 · p. 8

7 Conclusion

PISCES targets conceptual knowledge in model parameters by disentangling parameter directions and editing those that encode a concept. Across two models and diverse concepts, it improves specificity and robustness while maintaining or slightly improving efficacy.

7 Conclusion

  • PISCES disentangles model parameters to isolate concept-representing directions and removes them through targeted edits.The framework is presented as an alternative to approaches based on fine-tuning or fact-level editing.Source: §7 · p. 9
  • PISCES achieves higher robustness and specificity than existing methods while maintaining or slightly improving efficacy.The evaluation covers two models and diverse concepts.Source: §7 · p. 9
  • The conclusion presents in-parameter erasure as a state-of-the-art approach for fine-grained and robust conceptual knowledge removal in LLMs.Source: §7 · p. 9

Limitations

The current PISCES implementation is limited to MLP parameters and relies on SAE features and vocabulary projection for concept identification. The authors identify extensions to attention components and alternative feature-discovery approaches as future directions.

Limitations

  • The current implementation edits MLP parameters only, although attention heads also contribute to knowledge storage.The authors suggest that extending PISCES to those components could enable more comprehensive erasure.Source: Limitations · p. 9
  • Reliance on SAEs limits erasure to concepts captured as features and also entails imperfect reconstructions.The authors propose new parameter-disentangling methods and supervised disentanglement as possible alternatives for future work.Source: Limitations · p. 9
  • Vocabulary projection is less reliable in early layers, motivating possible use of complementary automated interpretability techniques.Source: Limitations · p. 9

Ethical Considerations

The paper acknowledges that concept erasure could be misused for censorship or suppressing legitimate knowledge, while describing potential benefits for safer deployment, copyright compliance, and studying how concepts are encoded in parameters.

Ethical Considerations

  • The authors acknowledge that erasing undesirable or sensitive concepts could be misused for censorship or suppression of legitimate knowledge.They also cite safer deployment, copyright compliance, and understanding concept encoding as potential benefits.Source: Ethical Considerations · p. 9

A.1 SAE Feature Selection

  • Concept-specific token selection tokenizes the target concept’s forget set, removes stop words, applies TF-IDF to identify informative tokens, then manually selects 2–5 highly correlated tokens, preferably among top-ranked tokens.For the Culture of Greece example, TF-IDF ranked Greek and Greece as the top two tokens, with Athens ranked 11th.Source: §A.1 · p. 14
  • Candidate SAE features are selected by VocabProj token overlap, manually filtered for concept alignment, then pruned if ablation significantly degrades MMLU validation performance.The overlap threshold used was α = 4, and manual filtering typically took under a minute.Source: §A.1 · p. 14

A.2 Setting SAE Feature Activations

  • Feature edits account for whether a feature promotes or suppresses the concept and whether its activation is positive or negative in concept-context data.The feature direction is inferred from concept tokens in its vocabulary projection; activation sign is determined by the majority sign on forget-set data.Source: §A.2 · p. 15

A.3 Evaluating Feature Selection Agreement

  • Four annotators assessed 158 candidate features across 10 concepts, with Fleiss’ κ = 0.574 indicating moderate to near-substantial agreement.Source: §A.3 · pp. 15–16

B Hyperparameter Selection

  • Each method’s per-concept hyperparameters were selected from 100 configurations using a disjoint validation set and the efficacy, specificity, and coherence harmonic mean.Source: B Hyperparameter Selection · p. 16

C Justifying use of LLM-as-a-Judge

  • An LLM judge’s evaluations were checked against three human annotators on 120 questions; its winning rate was ω = 0.67 (p-value 0.027).The authors report this as justification for using the LLM judge in their evaluation protocol.Source: C Justifying use of LLM-as-a-Judge · p. 16

D.1 Generating Questions; D.2 Generating Relearning Data

  • The authors manually validated 5% of generated question-answer pairs and found them accurate; relearning data was filtered to exclude sentences likely to contain evaluation answers.Manual review of 5% of sampled relearning paragraphs found no answers to the test questions.Source: §D.1 · p. 17, §D.2 · p. 17, Relearning Data Generation Prompt · p. 18

E Feature Analysis

Feature analysis reports examples of feature annotations, summarizes selected-feature attributes by concept, and finds a weak relationship between coherence and erasure accuracy.

E Feature Analysis

  • A weak relationship between feature coherence and accuracy suggests that more coherent features tend to enable more effective concept erasure.Figure 8 illustrates the relationship between coherence and accuracy scores.Source: E Feature Analysis · p. 18
  • Table 6 summarizes each concept’s number of selected features and their average alignment, coherence, and normalized erasure scores.Concepts are sorted by feature count, and sensitive concepts are marked with an asterisk.Source: E Feature Analysis · p. 18
  • Table 5 provides feature-annotation examples for different alignment–coherence score combinations.Examples include tokens associated with Uranium as a sub-concept of Nuclear.Source: E Feature Analysis · p. 18

F Adversarial Evaluation

The adversarial evaluation tested prompting attacks and a universal suffix on unlearned models, but found negligible or slightly negative effects and omitted them from evaluation.

F Adversarial Evaluation

  • The evaluation used a Lynch et al. (2024) adversarial prompt and a per-concept universal suffix trained on three validation-set questions.The attacks were tested on unlearned models as part of the robustness evaluation.Source: F Adversarial Evaluation · p. 18
  • Across five concepts, adversarial prompting and a universal GCG suffix had negligible or slightly negative effects on retained accuracy for PISCES, ELM, and RMU.Mean effect on retained accuracy ranged between −0.06 and 0.003.Source: F Adversarial Evaluation · pp. 18–19
  • The authors omitted these attacks from their evaluation because their effects were negligible or counterproductive.Source: F Adversarial Evaluation · p. 19

G Difference-In-Means

The difference-in-means disentangler provides an alternative to the SAE-based implementation, while paired tests report significant PISCES advantages in specificity and robustness.

G Difference-In-Means

  • The difference-in-means alternative forms a concept-specific vector from forget- and retain-set MLP-output means, then edits the top-k most similar vectors by weight orthogonalization.Across concepts on Gemma-2-2b-it, it struggled to balance concept erasure with retained model performance; the authors describe the SAE-based disentangler as stronger in precision and robustness.Source: G Difference-In-Means · p. 19

H Statistical Significance Testing

I Resources and Packages

  • Experiments used transformers, datasets, TransformerLens, and SAELens, and ran on a single H100 80GB GPU.The authors also used ChatGPT to help implement specific helper functions.Source: I Resources and Packages · p. 19
Loading 2505.22586v2…