PaperlayerView exact revision on arXiv

Source-linked AI summary

Intrinsic Test of Unlearning Using Parametric Knowledge Traces

Yihuai Hong, Lei Yu, Haiqin Yang, Shauli Ravfogel, Mor Geva

arXiv:2406.11614v3cs.CLcs.AI

TL;DR

A key challenge in developing unlearning methods is evaluating their performance, particularly validating whether unlearned information has been erased. The paper uses vocabulary projections to identify concept vectors and builds CONCEPTVECTORS, a behavioral and intrinsic benchmark covering 285 concepts in LLaMA and OLMo. Existing methods make only negligible changes to concept vectors despite suppressing generation, whereas direct intervention removes traces and jailbreaks can reactivate them by increasing vector activations.

  • Problem

    A key challenge in developing unlearning methods is evaluating their performance, particularly validating whether unlearned information has been erased.

    Source: §1 · p. 1

  • Method

    The paper uses vocabulary projections to identify concept vectors and builds CONCEPTVECTORS, a behavioral and intrinsic benchmark covering 285 concepts in LLaMA and OLMo.

    Source: §1 · p. 1

  • Results

    Existing methods make only negligible changes to concept vectors despite suppressing generation, whereas direct intervention removes traces and jailbreaks can reactivate them by increasing vector activations.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Ablating concept vectors effectively removes associated knowledge and reduces susceptibility to adversarial attacks, supporting evaluation of parametric erasure alongside external behavior.

    Source: §7 · p. 8

    The benchmark examines MLP layers rather than all knowledge-bearing parameters, and concept superposition can make editing affect unrelated concepts.

    Source: Limitations · p. 9

Abstract

from arXiv · show

The task of "unlearning" certain concepts in large language models (LLMs) has attracted immense attention recently, due to its importance in mitigating undesirable model behaviours, such as the generation of harmful, private, or incorrect information. Current protocols to evaluate unlearning methods largely rely on behavioral tests, without monitoring the presence of unlearned knowledge within the model's parameters. This residual knowledge can be adversarially exploited to recover the erased information post-unlearning. We argue that unlearning should also be evaluated internally, by considering changes in the parametric knowledge traces of the unlearned concepts. To this end, we propose a general evaluation methodology that leverages vocabulary projections to inspect concepts encoded in model parameters. We use this approach to localize "concept vectors" - parameter vectors that encode concrete concepts - and construct ConceptVectors, a benchmark dataset containing hundreds of common concepts and their parametric knowledge traces within two open-source LLMs. Evaluation on ConceptVectors shows that existing unlearning methods minimally impact concept vectors and mostly suppress them during inference, while directly ablating these vectors demonstrably removes the associated knowledge and significantly reduces the model's susceptibility to adversarial manipulation. Our results highlight limitations in behavioral-based unlearning evaluations and call for future work to include parameter-based evaluations. To support this, we release our code and benchmark at https://github.com/yihuaihong/ConceptVectors.

1 Introduction

Behavioral unlearning tests do not establish that target knowledge has been removed from model parameters, and residual knowledge may still be elicited. The paper introduces a parameter-based benchmark and finds that existing methods mostly suppress, rather than erase, parametric knowledge traces.

1 Introduction

  • Behavioral tests largely assess answering or completing queries, while growing evidence shows that steering can still elicit supposedly unlearned information.The cited examples include question answering and query completion.Source: §1 · p. 1
  • The paper presents CONCEPTVECTORS as the first benchmark for internal, parameter-based evaluation of unlearning methods.The conclusion argues that evaluation should consider parametric erasure alongside external performance.Source: §1 · pp. 1–2
  • The proposed parameter-based evaluation uses vocabulary projections to locate concept vectors in MLP layers that strongly affect corresponding concepts and minimally affect unrelated ones.The authors describe these vectors as a subset of parameters encoding target knowledge and their removal as a necessary condition and lower bound for thorough unlearning.Source: §1 · p. 1
  • Existing unlearning methods prevent concept generation but make only negligible changes to parametric knowledge traces; direct concept-vector intervention erases traces and affects generation.The benchmark evaluates 285 concepts in LLaMA and OLMo.Source: §1 · pp. 1–2
  • Jailbreaks increase concept-vector activations, bypass unlearning, and can generate supposedly unlearned knowledge; better trace removal can reduce jailbreak success.The paper evaluates multiple adversarial attacks after unlearning.Source: §1 · p. 2

2 Related Work

Prior work studies behavioral unlearning benchmarks, knowledge localization, and jailbreak robustness, but the cited evaluation work does not examine parametric knowledge traces during unlearning. This paper connects knowledge localization to unlearning and evaluates that connection across methods.

Evaluating Unlearning

  • Existing unlearning benchmarks address forgetting fictional-author knowledge, harmful knowledge, and jailbreak robustness, but none examine parametric knowledge traces during evaluation.Examples include the Harry Potter task, TOFU, and WMDP.Source: §2 · p. 2

Knowledge Localization in LLMs

  • Prior localization studies find factual associations in MLP weights and show that manipulating concept-specific traces can alter model responses.The paper builds on these findings to connect knowledge localization with unlearning concepts.Source: §2 · p. 2
  • Unlike work that trains parameters to inject artificial concept knowledge or deletes memorized sequences, this approach uses traces formed through natural LLM training.Source: §2 · p. 3

LLM Safety and Adversarial Attacks

  • Earlier safety and attack studies report jailbreak recovery of erased concepts, while this work evaluates the issue across unlearning methods and links erasing parametric knowledge with reduced attack susceptibility.The cited prior work includes recovery of information after model editing and alignment.Source: §2 · p. 3

3 Parametric Knowledge Traces for Unlearning Evaluation

The section treats unlearning a concrete concept as removing its associated knowledge from model parameters, not merely changing behavior. It proposes evaluating this through vocabulary projections and concept-specific MLP parameter vectors, while treating those vectors as a lower-bound trace rather than a complete representation of knowledge.

3 Parametric Knowledge Traces for Unlearning Evaluation

  • Unlearning a concrete concept requires checking whether its associated information has been removed from parameters, rather than relying only on behavioral tests.For example, erasing Harry Potter knowledge includes associations such as his friends and creator.Source: §3 · p. 3
  • Vocabulary projections expose interpretable token patterns in MLP parameter vectors, which can indicate the concepts those vectors promote.Prior work describes MLP outputs as linear combinations of parameter vectors.Source: §3 · p. 3
  • The paper calls MLP parameter vectors with clear concept-related projection tokens concept vectors and uses them as knowledge traces for evaluating unlearning.A successful method is expected to substantially change vectors associated with the target concept so its specific associations are no longer visible.Source: §3 · p. 3
  • A concept vector may encode only part of the target knowledge or also relate to other concepts, so modifying such vectors is a necessary lower-bound condition, not sufficient by itself for complete erasure.Ablating one vector alone may not completely erase the knowledge.Source: §3 · p. 3

4 The CONCEPTVECTORS Benchmark

CONCEPTVECTORS pairs intrinsic evaluation of parameter traces with behavioral tests, using localized concept vectors to study whether unlearning erases or merely suppresses knowledge. Its construction selects vectors with clear concept patterns and target-specific causal effects, yielding 285 vectors across LLaMA and OLMo.

4 The CONCEPTVECTORS Benchmark

  • The benchmark combines intrinsic and behavioral evaluations to test unlearning against localized parametric concept vectors.It is built for LLaMA 2 7B-chat and OLMo 7B.Source: §4 · p. 4

4.1 Benchmark Construction Methodology

  • Vocabulary projections, GPT-4 scoring of top tokens, and manual review identify vectors with clear, concrete concept patterns.The search excludes 30% of candidates per layer before GPT-4 scores the remaining vectors; the authors select middle-upper-layer vectors.Source: §4.1 · pp. 4–5
  • Behavioral tests pair GPT-4-generated concept questions with text-completion queries sampled from Wikipedia paragraphs.Model-generated answers and completions serve as references; gold answers are not required.Source: §4.1 · p. 5

Step 3: Causal Validation of Concept Vectors

  • Causal validation retains vectors whose noise changes answers about the target concept substantially while leaving answers about other concepts similar.The validation perturbs one vector while keeping the model’s other parameters intact.Source: Step 3: Causal Validation of Concept… · p. 5

4.2 Benchmark Statistics

  • The final benchmark contains 285 vectors: 105 in LLaMA and 180 in OLMo; the authors also report validating vectors in other LLMs.Each benchmark concept has QA pairs and text-completion queries.Source: §4.1 · p. 4, §4.2 · p. 5

5 Experiments

Experiments compare behavioral outputs with changes in parametric concept vectors across several unlearning approaches. The reported results show that methods can suppress target-concept generation while leaving parametric traces largely intact, whereas direct vector intervention changes those traces substantially.

5.1 Unlearning Methods

  • The experiments evaluate likelihood-maximization, preference-optimization, model-editing, and representation-engineering methods, alongside the Needle oracle baseline.Needle erases information in the selected concept vector.Source: §5.1 · pp. 5–6

5.2 Evaluation Metrics

  • Intrinsic evaluation compares concept vectors before and after unlearning, while behavioral evaluation scores answers and completions about target and unrelated concepts.The behavioral metrics include BLEU and ROUGE-L; the experiment tunes hyperparameters on a validation split and reports test-set results.Source: §5 · p. 5, §5.2 · p. 6

5.3 Results

  • Needle reduces concept-vector Jaccard similarity below 0.05 while decreasing QA performance by 40%–60%.Its target–unrelated QA gap is 41 BLEU points for LLaMA and 45 for OLMo, compared with gaps below 30 points for other methods.Source: §5.3 · p. 6
  • Knowledge editing achieves lower target QA scores and higher unrelated QA scores than fine-tuning, while RMU underperforms MEMIT despite similar vector impact.The passage attributes RMU’s weaker performance to its objective of disrupting target activations rather than directly editing target knowledge.Source: §5.3 · p. 7
  • Existing methods often restrict target-concept generation but leave parametric knowledge largely intact, so behavioral evaluations overestimate erasure.Fine-tuning methods retain almost all concept-related top-projection tokens, with Jaccard similarity above 0.98 despite Target QA and text-completion scores below 0.25.Source: §5.3 · pp. 6–7, Table 1 · p. 6

6 Extraction of Parametric Knowledge with Jailbreak Attacks

The paper tests whether residual parametric knowledge can be exploited to recover supposedly unlearned information through jailbreak attacks. It reports that most attacks raise target concept-vector activations and that targeted vector edits can resist jailbreaks while preserving unrelated QA performance.

6.1 Jailbreak Attacks Activate Concept Vectors to Bypass Unlearning

  • The activation study tests ten selected concepts using benign and jailbreak questions on vanilla LLaMA and Gradient Difference- and DPO-unlearned variants.It compares each target vector’s activation with the mean activation of unrelated vectors in the same layer.Source: §6.1 · p. 7
  • All tested jailbreak attacks except low-resource-language attacks substantially increase target-vector activations, while unrelated vectors change only slightly.The low-resource-language result may reflect imperfect alignment between knowledge vectors in different languages; unlearning without jailbreak reduces target-vector activations.Source: §6.1 · p. 8

6.2 Removal of Parametric Knowledge Reduces Jailbreak Success

  • Needle and MEMIT resist jailbreaks while preserving unrelated QA performance better than baselines in the reported comparisons.For LLaMA, Needle maintains unrelated-QA BLEU of 0.7–0.8 while keeping target-QA BLEU at or below 0.05; other baselines preserving that unrelated-QA level show target-QA gaps above 0.4 for LLaMA and 0.1 for OLMo.Source: §6.2 · p. 8, Figure 3 · p. 8
  • MEMIT modifies more parameters than Needle, which may help when target knowledge is distributed across multiple vectors; fine-tuning leaves MLP traces more available to jailbreaks.Both MEMIT and Needle make targeted edits to specific MLP layers.Source: §6.2 · p. 8

7 Conclusion

CONCEPTVECTORS evaluates whether unlearning erases concept vectors, and the conclusion contrasts limited parametric changes from existing methods with effective vector ablation.

7 Conclusion

  • CONCEPTVECTORS benchmarks unlearning methods on their ability to erase concept vectors.Source: §7 · p. 8
  • Existing methods make no significant parametric changes, while ablating concept vectors removes corresponding knowledge and resists adversarial attacks.The conclusion presents vector ablation as an effective alternative to methods that leave parametric traces largely intact.Source: §7 · p. 8
  • The findings expose a gap in behavior-based evaluation and motivate more robust unlearning methods.Source: §7 · p. 9

Limitations

The benchmark’s coverage and the kinds of concepts it represents limit how broadly its findings can be generalized.

Limitations

  • The collection process does not guarantee coverage of all concept-encoding parameters, and it examines MLP layers although factual information may also reside in self-attention modules.Other concept-related parameters may therefore have been ablated by existing unlearning methods.Source: Limitations · p. 9
  • The benchmark focuses on concepts with strongly expressive vectors, whereas concepts may be encoded in superposition and editing one may affect unrelated concepts.Source: Limitations · p. 9
  • Long-tail knowledge may be stored more subtly, although current methods also struggle to parametrically unlearn easily located, frequent concepts.Source: Limitations · p. 9

Ethical Considerations

The paper identifies harmful and copyrighted concepts in model parameters and discusses both potential benefits and risks of disrupting concept vectors. Its appendix describes how concept vectors are selected and how QA questions are generated.

Ethical Considerations

  • The benchmark identifies vectors containing harmful concepts and knowledge related to copyrighted works, including Harry Potter and Star Wars.The authors connect these findings to unlearning and privacy protection.Source: Ethical Considerations · p. 9
  • Malicious actors could amplify harmful concept vectors or unlearn benign vectors that help keep outputs harmless.Source: Ethical Considerations · p. 9
  • Disrupting concept vectors is presented as substantially removing associated knowledge and strengthening defense against jailbreak attacks.Source: Ethical Considerations · p. 9

A.1 Concept Vectors Selection in CONCEPTVECTORS

  • Concept-vector selection scores the top 200 projected tokens for relevance, retains vectors scoring above 0.85, and then manually screens them for plausibility.Concepts are retained when GPT-4 provides a clear, consistent definition aligned with human annotators.Source: §A.1 · p. 12

A.2 QA Generation in CONCEPTVECTORS

  • QA evaluation generates ten simple questions about each concept.Source: §A.2 · p. 12

A.3 Concept Validation Experiments

Disrupting the identified concept vectors selectively reduced answer quality for target-concept questions, supporting their link to stored knowledge. The accompanying checks assess generated-question quality, benchmark contents, template generation, automated vector selection, and the scope needed for finer-grained evaluation.

A.3 Concept Validation Experiments

  • Noise at standard deviation 0.1 substantially reduced target-concept answer quality while leaving unrelated QA nearly unchanged, supporting the vectors’ link to target knowledge.Candidates with a target-versus-unrelated QA BLEU difference above 0.2 before and after noise were added to the benchmark.Source: §A.3 · p. 13

A.4 Quality of Generated QA Data

  • Of 284 sampled questions, 281 (99%) were reasonably simple; only 1 of 40 sampled concepts had two similar questions among its ten questions.The sampled questions were manually checked for relevance and simplicity, and all analyzed questions concerned their given concepts.Source: §A.4 · pp. 13–14

A.5 CONCEPTVECTORS Statistics

  • Each concept vector has 10 QA pairs; text-completion queries average 9.4 for LLaMA and 9.5 for OLMo, while paragraphs average 77.9 and 70.5 tokens, respectively.Source: §A.5 · p. 14

A.6 Wikidata Triplets to Templates Generation

  • GPT-4 prompt examples covered relation-specific templates for location, twin city, burial place, and native language to generate MEMIT input templates.Source: §A.6 · p. 14

A.7 Automated Approach for Selecting Concept Vectors

  • An automated selector could rank candidate vectors by vocabulary-projection probabilities for a supplied set of concept keywords, followed by causal verification.The proposed procedure is intended to reduce labor when building larger parametric benchmarks.Source: §A.7 · p. 14

A.8 Discussion on Evaluating Fine-Grained Unlearning

  • Evaluations target a specific concept and all its knowledge, whereas finer-grained unlearning would require localizing and segmenting parameters for a narrower target.The discussion contrasts removing all knowledge about “computer” with removing only harmful content such as computer-virus instructions.Source: §A.8 · p. 15

B Details of Unlearning Methods

The appendix details several unlearning approaches and robustness tests. Across the tests, jailbreaks typically raise concept-vector activations, and less parametric erasure generally corresponds to greater sensitivity to jailbreaks.

B Details of Unlearning Methods

  • Gradient ascent maximizes next-token loss on forget-set text, while Gradient Difference adds KL regularization against the original model to preserve general capability.Source: B Details of Unlearning Methods · p. 15
  • Preference methods include DPO, NPO, and NPO+KL; MEMIT editing uses empty-response or max-entropy objectives to remove or suppress target facts.DPO balances preferred and unfavored responses while limiting shift from the original distribution; NPO minimizes the probability of unfavored answers, with NPO+KL adding a KL term.Source: B Details of Unlearning Methods · p. 16
  • RMU perturbs hidden representations, whereas the Needle oracle directly adds Gaussian noise to the target concept vector.The two RMU variants use fixed layers or the concept-vector layer and its two preceding layers.Source: B Details of Unlearning Methods · p. 16

C Example Model Outputs

  • Under normal prompts, Gradient Ascent and Needle suppress target answers, but jailbreak prompts restore target answers after Gradient Ascent while Needle outputs remain nonsensical.Source: C Example Model Outputs · p. 17

D.1 Details of Unlearning Jailbreak

  • Robustness tests include handcrafted adversarial prompts, German-translated questions, a 2,000-token in-context concept passage, and automated GCG and AutoDAN suffix attacks.GCG optimizes adversarial suffixes; AutoDAN generates suffixes using hierarchical genetic algorithms.Source: §D.1 · p. 17

D.2 Details of Main Results

  • Figure 6 plots target-concept unlearning BLEU against unrelated-knowledge retention BLEU, and the scores are generally correlated.Source: §D.2 · p. 17, Figure 6 · p. 18

D.3 Activations of Concept Vectors

  • Jailbreaks typically increase concept-vector activation after DPO and Gradient Difference, though concepts with no increase had relatively low original activations.The activation distributions cover ten concepts and ten concept-related questions per concept.Source: §D.3 · p. 18

D.4 Intrinsic Evaluation Aligns with Jailbreak Success

  • Lower Jaccard similarity, indicating better parametric erasure, generally corresponds to lower jailbreak sensitivity across both models and methods.The trend has two exceptions among the highest-Jaccard LLaMA MEMIT points, where standard deviation is high.Source: §D.4 · p. 18

E Ablation Studies of Needle

Needle’s ablations show that combining vector noise with fine-tuning improves target-QA unlearning over either component alone. The appendix also gives the hyperparameter settings and reports a relationship between parametric erasure and jailbreak sensitivity.

F Unlearning Hyperparameter Tuning

  • Fine-tuning learning-rate grids differ by model; MEMIT uses a 5e-1 learning rate, 1e-3 weight decay, and four editing layers including and preceding the concept-vector layer.RMU retains original-paper hyperparameters except for the enhanced variant’s editing layers.Source: F Unlearning Hyperparameter Tuning · pp. 18–19
  • Across varying Jaccard similarity, better parametric erasure generally corresponds to a smaller target-QA score difference with versus without jailbreak.Source: Figure 9 · p. 19

E Ablation Studies of Needle

  • With unrelated-QA BLEU above 0.7, full Needle beat noise-only or fine-tuning-only by more than 0.04 target-QA BLEU; Needle with Gradient Ascent had the lowest target-QA BLEU at that retention level.The ablations compare noise alone, fine-tuning without noise, and the complete Needle approach, as well as Needle paired with different loss functions.Source: E Ablation Studies of Needle · p. 18, Figure 10 · p. 20
Loading 2406.11614v3…