PaperlayerView exact revision on arXiv

Source-linked AI summary

Editing Factual Knowledge in Language Models

Nicola De Cao, Wilker Aziz, Ivan Titov

arXiv:2104.08164v2cs.CLcs.AIcs.LG

TL;DR

Language models store useful but sometimes incorrect or obsolete factual knowledge implicitly in their parameters, creating a need for efficient, targeted editing. KnowledgeEditor uses a hyper-network with constrained optimization to predict parameter updates, and evaluations report reliable, consistent edits across fact-checking and question answering while updates concentrate in limited components.

  • Problem

    Language-model factual knowledge can be incorrect or obsolete, while its implicit storage in parameters makes targeted modification difficult and expensive to address through retraining.

  • Method

    KnowledgeEditor trains a hyper-network with constrained optimization to predict parameter updates that change a supplied fact while preserving other model predictions.

  • Results

    KnowledgeEditor largely meets reliability and consistency desiderata across fact-checking and question-answering tasks, with about 97% and 88% probabilities of outperforming all baselines on combined metrics.

  • Takeaways & Limitations

    The method enables efficient factual-memory edits without meta-learning, retraining, or fine-tuning the original language model, and its updates tend to concentrate in a restricted component subset.

  • Takeaways & Limitations

    The study does not fully investigate why the hyper-network focuses on particular components or how this relates to memorization mechanisms in other model domains.

Abstract

from arXiv · show

The factual knowledge acquired during pre-training and stored in the parameters of Language Models (LMs) can be useful in downstream tasks (e.g., question answering or textual inference). However, some facts can be incorrectly induced or become obsolete over time. We present KnowledgeEditor, a method which can be used to edit this knowledge and, thus, fix 'bugs' or unexpected predictions without the need for expensive re-training or fine-tuning. Besides being computationally efficient, KnowledgeEditordoes not require any modifications in LM pre-training (e.g., the use of meta-learning). In our approach, we train a hyper-network with constrained optimization to modify a fact without affecting the rest of the knowledge; the trained hyper-network is then used to predict the weight update at test time. We show KnowledgeEditor's efficacy with two popular architectures and knowledge-intensive tasks: i) a BERT model fine-tuned for fact-checking, and ii) a sequence-to-sequence BART model for question answering. With our method, changing a prediction on the specific wording of a query tends to result in a consistent change in predictions also for its paraphrases. We show that this can be further encouraged by exploiting (e.g., automatically-generated) paraphrases during training. Interestingly, our hyper-network can be regarded as a 'probe' revealing which components need to be changed to manipulate factual knowledge; our analysis shows that the updates tend to be concentrated on a small subset of components. Source code available at https://github.com/nicola-decao/KnowledgeEditor

1 Introduction

KnowledgeEditor addresses incorrect or obsolete factual knowledge stored implicitly in language-model parameters. It uses a hyper-network and constrained updates to revise targeted predictions while preserving other knowledge and improving consistency across equivalent inputs.

  • Motivation: Language models can memorize useful facts during pre-training, but those facts may be incorrect or become obsolete.The paper motivates efficient bug-fixing without expensive retraining.
  • Motivation: Neural models store factual knowledge implicitly in parameters, making their memories difficult to access, interpret, and modify.This opacity makes targeted knowledge editing challenging.
  • Method: The method targets reliability and consistency: it changes a specific fact without affecting unrelated knowledge and propagates changes across equivalent formulations.These desiderata distinguish the method from prior approaches described in the introduction.
  • Method: KnowledgeEditor trains a hyper-network to predict parameter updates for a supplied fact without requiring meta-learning, retraining, or fine-tuning of the original model.Constrained optimization encourages the edited model to retain original predictions away from the targeted fact.
  • Analysis: The learned updates concentrate on a restricted set of model components without explicitly encouraging sparsity.The hyper-network also acts as a probe, and the most-updated components differ from those receiving large gradients.
  • Results: Across fact-checking and question-answering settings, the authors report that KnowledgeEditor largely meets these desiderata while fine-tuning baselines fail.The evaluation uses different language-model architectures and knowledge-intensive tasks.

2 Task

The task is to revise a model’s preferred prediction for a targeted input while preserving predictions elsewhere. It evaluates successful edits, retention, consistency across equivalent inputs, and deterioration of downstream performance.

  • Task definition: Knowledge editing seeks revised model outputs that reflect changed facts despite knowledge being distributed opaquely across parameters and components.The operational goal is a targeted parameter change affecting predictions associated with revised facts.
  • Task definition: For revision tuple ⟨x, y, a⟩, the edited model should prefer alternative prediction a instead of original prediction y for input x.The architecture remains fixed while parameters change from θ to θ′.
  • Task definition: The preservation target is approximated with a finite set O_x of pairs ⟨x′, y′⟩ where x′ ≠ x.The model should retain other predictions, while outputs may be generated by arbitrary decision rules.
  • Equivalent inputs: Paraphrase sets P_x provide supervision for coordinated edits and evaluation of consistency across semantically equivalent inputs.Paraphrases are used during training and evaluation, not at test time.
  • Metrics: The evaluation measures success rate, retain accuracy, equivalence accuracy, and performance deterioration.These metrics compare revised predictions, preserved original predictions, equivalent-input behavior, and updated-model test performance.
  • Metrics: Retain accuracy can be computationally demanding because it may require predictions across the entirety of accessible validation or test data.Mini-batch subsampling can approximate this metric during hyper-network training.

3 Related work

Prior approaches include expensive retraining, meta-learning for editable initialization, and constrained refitting with parameter norms. Related work also studies factual recall, explicit memory, and causal mechanisms in neural networks.

  • Model modification: Retraining a language model on modified facts is often infeasible because large-scale LM training is expensive.This motivates more efficient model-editing procedures.
  • Model modification: Sinitsin et al. use meta-learning so models can be edited with a few SGD steps, but this requires expensive specialized pre-training.Their regularized objective also constrains the updated model’s deviation from the original.
  • Model modification: Zhu et al. use constrained optimization with L2 or L∞ parameter-distance constraints when refitting on altered downstream data.The approach is less computationally expensive than full retraining but constrains parameter space directly.
  • Model modification: Parameter-norm constraints may not preserve outputs effectively because neural language models are highly nonlinear.The related discussion distinguishes similarity in parameter space from similarity in function space.
  • Knowledge in LMs: Prior studies show that pretrained language models can recall factual knowledge without fine-tuning, while retrieval-based alternatives can achieve higher prediction quality.Other work investigates prompts and methods for understanding model knowledge.
  • Knowledge in LMs: Hybrid systems combining implicit and explicit memory have shown benefits for question answering, while memorization-focused methods support downstream tasks.These findings motivate editing factual knowledge stored in model parameters.
  • Causal interventions: Causal-intervention research identifies minimal neural changes associated with behaviors, often by modifying activations rather than weights.KnowledgeEditor’s update analysis connects to this broader interpretability perspective.

4 Method

KnowledgeEditor treats factual-memory editing as a constrained learning-to-update problem: a hyper-network predicts parameter changes that revise one fact while preserving other predictions. Its parameterization uses gradients and gated updates for tractable editing, with constraints approximated for large models and sequence-to-sequence outputs.

  • Method formulation: KnowledgeEditor is a hyper-network that predicts updated parameters from an atomic fact requiring revision.It replaces a handcrafted update algorithm and avoids retraining or fine-tuning the original network.
  • Method formulation: The editor minimizes loss for the desired alternative prediction while constraining outputs on other inputs to remain unchanged.The constraint is expressed through KL divergence between updated and original output distributions.
  • Method formulation: Semantically equivalent inputs can provide training supervision for coordinated edits, although they are unavailable as editor inputs at test time.They are also used to evaluate consistency across paraphrases.
  • Optimization: The model uses Lagrangian relaxation, Monte Carlo sampling, and beam search to make constrained optimization tractable.Beam search approximates KL computation for sequence-to-sequence models with unbounded output spaces.
  • Architecture: The hyper-network predicts a parameter shift rather than updated parameters directly, using gradient information to scale updates efficiently.For each weight matrix, the update is a gated sum of a scaled gradient and a bias term; the gate can leave parameters unchanged.

5 Experimental Setting

The evaluation compares KnowledgeEditor with fine-tuning and a parameter-constrained baseline on closed-book fact-checking and question answering. It uses BERT and BART models, with paraphrase annotations in question answering enabling robustness evaluation.

  • Baselines: The experiments compare KnowledgeEditor against standard fine-tuning and Zhu et al.’s method with an L∞ parameter constraint.Baselines are evaluated with either all parameters or selected layers, while KnowledgeEditor learns the update subset automatically.
  • Tasks and models: Closed-book fact-checking fine-tunes BERT base on the binary FEVER dataset from KILT.The task is evaluated using the metrics introduced for knowledge editing.
  • Tasks and models: Closed-book question answering fine-tunes BART base with a standard sequence-to-sequence objective on the zsRE dataset.Human-generated question paraphrases support measuring robustness to semantically equivalent inputs.

6 Results

KNOWLEDGEEDITOR performs strongly across fact-checking and closed-book question answering while better balancing successful edits, retained knowledge, and consistency across paraphrases than competing approaches. Its constrained updates preserve predictions more effectively than broad fine-tuning, concentrate changes in a small subset of components, and can reach near-perfect task outcomes with test-time update loops.

  • Overall results: ≈97% and ≈88% probability of outperforming all baselines across stochastic metric combinations for fact-checking and question answering, respectively.The comparison combines multiple metrics rather than selecting a single metric.
  • Success rate: 100% fact-checking success and >99% QA accuracy are achieved when KNOWLEDGEEDITOR applies multiple updates at test time.The model is not trained for multiple subsequent updates; the loop uses the baselines’ stopping criteria.
  • Success rate: KNOWLEDGEEDITOR achieves high accuracy on closed-book QA while obtaining the best success rate and retain accuracy among the compared methods.Closed-book QA has a text output space, making it more challenging than binary fact-checking; accuracy is approximately 95% or above 99% with the loop.
  • Retaining previous knowledge: Fine-tuning all layers lowers retain accuracy to ≈87% for FC and ≈68% for QA, with performance deterioration of ≈2% and ≈4%, respectively.Fine-tuning a single layer improves the trade-off but performs poorly on semantic-equivalent inputs in fact-checking.
  • Accuracy on paraphrases: Paraphrase supervision improves equivalence accuracy by >70% on FC and >30% on QA without changing success or retain accuracy.Without this supervision, KNOWLEDGEEDITOR is already competitive, but the added supervision is beneficial on both tasks.
  • Accuracy on paraphrases: KNOWLEDGEEDITOR preserves almost all predicted labels and probabilities, with most updated-versus-original logits remaining on the diagonal.Errors are concentrated near the decision boundary, where small perturbations can flip uncertain predictions.
  • Analysis of model updates: The learned updates are sparse and concentrated in a restricted set of components, unlike fine-tuning, which updates the model more uniformly.The most-updated components differ from those receiving the largest gradients, and the analysis highlights parameters affecting attention-distribution shape.

7 Conclusions

This work investigates editing factual knowledge stored implicitly in language-model parameters and proposes KnowledgeEditor, a hyper-network method for modifying that knowledge efficiently and reliably.

  • KnowledgeEditor targets factual knowledge implicitly stored within language-model parameters.
  • The method uses a hyper-network to learn efficient and reliable modifications to language-model knowledge.The authors report comprehensive evaluations against variants of fine-tuning.
  • The magnitude of predicted updates may reveal mechanisms used by language models to encode factual knowledge.The authors leave investigation of these mechanisms for future work.

Ethical Considerations

The authors argue that editing language-model knowledge can help mitigate harms, while warning that the same techniques could be misused to inject incorrect knowledge.

  • Editing language-model knowledge can correct problems after they are discovered and may mitigate inherited harms.
  • Malicious agents could use knowledge editing to inject incorrect knowledge into language models.

B Experimental setting

The experiments evaluate closed-book fact-checking and question answering, emphasizing edits that generalize across semantically equivalent inputs without retrieved evidence.

  • Fact-checking: The study evaluates closed-book fact-checking on binary FEVER, where models predict claim truth probabilities without retrieving evidence.FEVER contains 104,966 training and 10,444 validation instances.
  • Question answering: Closed-book question answering is treated as sequence-to-sequence prediction from questions to answers without retrieval or supplied evidence.
  • Question answering: The zsRE dataset is used because it includes human-generated question paraphrases for evaluating robustness to semantically equivalent inputs.Paraphrases are available for 30% of the dataset.
  • Edit targets: Alternative predictions are task-dependent: fact-checking flips the binary label, whereas question answering uses high-probability model outcomes as plausible revisions.
  • Edit targets: High-probability alternative answers do not always guarantee the same semantic type as the original answer.The paper notes that matching semantic type is likely because the model assigns them high probability.
  • Consistency: The desired edits should change predictions consistently across semantically equivalent inputs rather than producing isolated datapoint updates.The authors generate equivalent inputs because FEVER lacks paraphrases and zsRE contains them for only part of its data.

B.5 Architecture details

The experiments modify BERT and BART base models with Transformer architectures, while KnowledgeEditor uses a smaller single-layer bidirectional LSTM.

  • Original models: The original models are 12-layer Transformer-based BERT and BART base models with hidden size 768.BERT has 110M parameters and BART has 139M parameters.
  • Original models: BERT has 12 attention heads and a vocabulary of 30,522, while BART has 16 heads and a vocabulary of 50,265.
  • KnowledgeEditor: KnowledgeEditor uses a small single-layer bidirectional LSTM with 54M parameters for BERT and 67M for BART.

B.6 Training details

KnowledgeEditor training uses larger update budgets and distinct optimization settings for fact-checking and question answering, while the original models use a shorter, smaller-scale schedule.

  • Original models: The original models use batch size 256, Adam with learning rate 3e-5, weight decay 1e-2, and a 50k-update schedule with 500 warm-up updates.Training lasts up to 20 epochs, with model selection based on validation accuracy.
  • KnowledgeEditor models: KnowledgeEditor uses batch sizes of 1024 for fact-checking and 256 for question answering, with learning rate 3e-4 for parameters and 1e-1 for the Lagrangian multiplier.The optimizer also uses weight decay 1e-2 and a linear schedule with 200k updates and 1k warm-up updates.
  • KnowledgeEditor models: KnowledgeEditor training runs for up to 200 epochs and selects models using overall validation accuracy combining success rate and retain accuracy.Validation selection is approximated using mini-batches.

C Additional Results

Additional analyses show that gradient information helps KnowledgeEditor converge, while its learned updates differ from ordinary gradients and concentrate on restricted model components. A stochastic multi-metric comparison favors the full method across both tasks.

  • Update Analysis: Removing gradient information made fact-checking training converge approximately 10 times slower and prevented question-answering models from reaching the reported accuracy thresholds.Without gradients, question answering failed to exceed 75% success rate and 50% retain accuracy.
  • Update Analysis: KnowledgeEditor updates have low cosine similarity with other update methods, whereas fine-tuning is highly correlated with the gradient.The authors note that low similarity may reflect a different knowledge-preserving direction or the geometry of a very large parameter space.
  • Update Analysis: Table 2 reports average cosine similarities between update methods and the gradient, with fine-tuning applied to all layers.The comparison is intended to characterize how different update directions relate to the gradient.
  • Stochastic metric comparison: Figure 5 interprets metric aggregation probabilistically by showing weighted-sum score distributions from 1,000 Dirichlet-sampled assignments.KnowledgeEditor variants have distributions more skewed toward the high score of 100.
Loading 2104.08164v2…