PaperlayerView exact revision on arXiv

Source-linked AI summary

Detecting Edit Failures In Large Language Models: An Improved Specificity Benchmark

Jason Hoelscher-Obermaier, Julia Persson, Esben Kran, Ioannis Konstas, Fazl Barez

arXiv:2305.17553v2cs.CLcs.AIcs.LG

TL;DR

Existing specificity benchmarks can miss unwanted side effects of model edits. The paper proposes edit-conditioned COUNTERFACT+ prompts and a full-distribution metric, then finds specificity problems in ROME and MEMIT that COUNTERFACT does not detect.

  • Problem

    Existing specificity evaluation can overlook edit-induced side effects, including behavior prompted by the edit subject and changes beyond the original and edited tokens.

    Source: §1 · p. 2

  • Method

    The paper adapts neighborhood prompts to each edit in COUNTERFACT+ and adds NKL, a specificity metric based on the full probability distribution.

    Source: §1 · p. 2, §3.1 · p. 3, §5 · p. 5

  • Results

    ROME and MEMIT have specificity problems not detected by COUNTERFACT; FT-L is less vulnerable but falls short on paraphrase robustness.

    Source: §1 · p. 2, §5 · p. 5

  • Takeaways & Limitations

    Specificity benchmarks need to identify unwanted side effects to support effective and safe use of edited LLMs.

    Source: §1 · p. 1, §5 · p. 5

    The benchmark-improvement approach relies on manual inspection, which is not scalable and requires substantial time and effort.

    Source: Limitations · p. 6

Abstract

from arXiv · show

Recent model editing techniques promise to mitigate the problem of memorizing false or outdated associations during LLM training. However, we show that these techniques can introduce large unwanted side effects which are not detected by existing specificity benchmarks. We extend the existing CounterFact benchmark to include a dynamic component and dub our benchmark CounterFact+. Additionally, we extend the metrics used for measuring specificity by a principled KL divergence-based metric. We use this improved benchmark to evaluate recent model editing techniques and find that they suffer from low specificity. Our findings highlight the need for improved specificity benchmarks that identify and prevent unwanted side effects.

1 Introduction

The paper argues that existing specificity evaluation can miss edit-induced side effects, and introduces edit-conditioned prompts and full-distribution metrics to expose them.

1 Introduction

  • Model editing changes a trained model’s predicted distributions by updating its parameters, without retraining the entire model.The introduction illustrates an edit changing the Louvre’s location from Paris to Rome.Source: §1 · p. 2
  • COUNTERFACT can overlook side effects because it omits edit-conditioned prompts and considers only probabilities for the original and edited object tokens.Side effects may therefore be missed and specificity overestimated.Source: §1 · p. 2
  • The paper introduces COUNTERFACT+, a dynamic benchmark that adapts neighborhood prompts to the edit and is more sensitive than COUNTERFACT.It prepends the edit to each selected neighborhood prompt.Source: §1 · p. 2
  • Neighborhood KL divergence (NKL) measures specificity using the full probability distribution, complementing metrics focused on tokens implicated in the edit.This addresses distribution changes to tokens beyond the original and edited objects.Source: §1 · p. 2
  • ROME and MEMIT show specificity problems that COUNTERFACT does not detect, while FT-L is less vulnerable but weaker on paraphrase robustness.The conclusion describes model editing specificity as an ongoing challenge requiring further attention.Source: §1 · p. 2

2 Related work

Prior work develops methods for locating and changing stored knowledge in transformers, along with benchmarks for evaluating edited models and methods for eliciting model knowledge.

Model editing

  • Transformer knowledge-editing methods include MLP-memory updates, test-time hyper-networks, gradient-based localization, and rank-one or multi-layer edits.ROME edits one MLP layer [Meng et al., 2022a], while MEMIT distributes edits across layers to support thousands of simultaneous edits without performance degradation [Meng et al., 2022b].Source: §2 · p. 2

Model editing evaluation

Knowledge extraction from LLMs

  • Knowledge extraction commonly tests whether language models complete fill-in-the-blank prompts, with prompt diversification and fine-tuning also investigated.Constructing prompts from supervised extraction data can risk teaching new knowledge rather than eliciting what a model already recalls.Source: §2 · p. 3

3 Experimental Setup

The setup evaluates model-edit specificity using benchmark neighborhood prompts, an edit-conditioned variant, and metrics that include changes across the full next-token distribution.

3.1 Dataset

  • COUNTERFACT contains 21,919 non-factual statements targeted by edits, with neighborhood prompts used to check for unwanted changes to other associations.Its neighborhood prompts are paraphrases of true associations for nearby subjects sharing the relation and correct object.Source: §3.1 · p. 3
  • COUNTERFACT+ prepends the edit to selected neighborhood prompts, making each prompt depend on the edit under evaluation.The prompt set is selected for semantic closeness to the intended edit, then modified as a function of that edit.Source: §3.1 · p. 3

3.2 Metrics

  • Neighborhood Score counts prompts where the correct token remains more probable than the edit token; Neighborhood Magnitude measures their probability difference.High NS and NM indicate smaller unwanted side effects.Source: §3.2 · p. 3
  • NKL compares edited and unedited next-token distributions to capture changes in probabilities beyond the correct and edit tokens.A large NKL indicates that neighborhood-prompt distributions were strongly affected by the edit.Source: §3.2 · p. 4

3.3 Models and Model Editing Algorithms

  • Evaluations use GPT-2-medium, GPT-2-XL, and GPT-J with ROME, MEMIT, and constrained single-layer FT-L as a baseline.ROME updates one MLP layer; MEMIT updates several, and FT-L applies an L-infinity norm constraint.Source: §3.3 · p. 4

4 Results

COUNTERFACT+ reveals specificity problems that COUNTERFACT misses: ROME and MEMIT show larger declines than FT-L, while results across models show a shift in which editing method appears most specific. The decline also appears on the unedited model, indicating possible confounding from the test prompts.

4 Results

  • Specificity also drops for unedited GPT-J on COUNTERFACT+, suggesting test-prompt confounding, while the drop is more pronounced for ROME and MEMIT than for FT-L.The authors suggest recency bias as a possible source of confounding.Source: §4 · p. 4
  • On COUNTERFACT+, NS ranges from 33% to 54% across editing algorithms, versus close to 80% on COUNTERFACT; NKL spikes for ROME and MEMIT.FT-L shows a much smaller NKL increase between benchmarks.Source: Figure 2 · p. 4, §4 · p. 4
  • Across three models, the method with the highest measured specificity switches from MEMIT/ROME on COUNTERFACT to FT-L on COUNTERFACT+.The comparison reports Neighborhood Score, Neighborhood Magnitude, and Neighborhood KL divergence.Source: Table 1 · p. 5, Table 2 · p. 5, Table 3 · p. 5
  • The COUNTERFACT+ specificity drop holds across different model sizes and is not limited to one model.The authors report that whether the problems disappear for extremely large models remains unclear.Source: §4 · p. 5

5 Conclusion

The conclusion reports previously undisclosed specificity problems in autoregressive-transformer editing, including problems that COUNTERFACT fails to detect. COUNTERFACT+ and NKL expose these issues, but model editing remains complex and requires further attention.

5 Conclusion

  • FT-L is less vulnerable to the specificity issues than ROME and MEMIT but falls short on robustness to paraphrasing.The authors conclude that model editing still presents complexities requiring future attention.Source: §5 · p. 5
  • ROME and MEMIT show low specificity that the existing COUNTERFACT benchmark fails to detect.The authors describe these as previously undisclosed specificity problems in autoregressive-transformer editing.Source: §5 · p. 5
  • COUNTERFACT+ adapts to the edit under test and is more sensitive than the existing benchmark.NKL complements existing metrics by using the full probability distribution rather than only edit-implicated tokens.Source: §5 · p. 5

Limitations

The benchmark-improvement approach depends on manual inspection of test cases, which the authors say is not scalable and costs time and effort. They also identify untested settings and edit types as boundaries of the work.

Limitations

  • Understanding editing failure modes through manual test-case inspection is not scalable and requires significant time and effort.This is identified as the main limitation of the benchmark-improvement approach.Source: Limitations · p. 6
  • The benchmark's effectiveness remains to be assessed for complex scenarios such as dialogue and multi-turn conversations.The authors also have not investigated simultaneous multiple edits.Source: Limitations · p. 6
  • The evaluation does not cover edit types such as parameter pruning or transfer learning, nor long-term language-model properties such as discourse structure and fluency.The authors propose these long-term properties as targets for future measurement.Source: Limitations · p. 6

Ethics Statement

The ethics statement notes potential risks from autoregressive transformers and reports that no human experiments were performed. The dataset and evaluation code are open sourced.

Ethics Statement

  • The authors state that they do not perform human experiments.The passage ends mid-word after mentioning evaluation, so no further evaluation claim is included.Source: Ethics Statement · p. 6
  • The authors acknowledge risks from autoregressive transformers, including generating and manipulating text for harmful purposes.Source: Ethics Statement · p. 6
  • The dataset and evaluation code are open sourced.The authors also provide a homepage with interactive examples.Source: Ethics Statement · p. 6

A Neighborhood magnitude

Neighborhood Magnitude (NM) measures the probability difference between the correct token and the edit token. On COUNTERFACT+, ROME and MEMIT have lower specificity than on COUNTERFACT, with a larger drop for ROME.

A Neighborhood magnitude

  • NM measures the difference in probability between the correct token and the edit token.Source: Figure 3 · p. 7
  • ROME nearly matches unedited GPT-J-6B on COUNTERFACT but shows a large specificity drop on COUNTERFACT+.Source: Figure 3 · p. 7
  • MEMIT also performs significantly worse on COUNTERFACT+ than COUNTERFACT, though its drop is less dramatic than ROME’s.Source: Figure 3 · p. 7

B Scaling with model size

The specificity drop on COUNTERFACT+ persists through GPT-J (6B), while evidence that specificity improves with model size is preliminary and confounded by improvement in the unedited model. Figures 4–6 report three specificity measures across algorithms and model sizes.

B Scaling with model size

  • The specificity drop on COUNTERFACT+ persists through GPT-J (6B), but evidence of improvement with larger models is preliminary and confounded by better specificity in the unedited model.It remains unclear whether ROME and MEMIT’s specificity problems would disappear with extremely large models.Source: B Scaling with model size · p. 8
  • Figure 4 measures COUNTERFACT+ specificity with NS, the average fraction of neighborhood test prompts successfully completed after editing; larger values are better.Source: Figure 4 · p. 8
  • Figure 5 measures COUNTERFACT+ specificity with NM, the probability difference between the correct token and edit token; larger values are better.Source: Figure 5 · p. 8
  • Figure 6 measures COUNTERFACT+ specificity with NKL, the edited model’s next-token-distribution KL divergence from the unedited model across neighborhood prompts; lower values indicate higher specificity.Source: Figure 6 · p. 8

C COUNTERFACT dataset sample

The COUNTERFACT sample specifies a requested model rewrite and includes prompts for measuring specificity, alongside examples of prompts and edits.

C COUNTERFACT dataset sample

D COUNTERFACT+ dataset sample

COUNTERFACT+ derives from COUNTERFACT by modifying neighborhood prompts to prepend the model edit before the original prompt.

D COUNTERFACT+ dataset sample

Related papers

Loading 2305.17553v2…