PaperlayerView exact revision on arXiv

Source-linked AI summary

AnyEdit: Edit Any Knowledge Encoded in Language Models

Houcheng Jiang, Junfeng Fang, Ningyu Zhang, Guojun Ma, Mingyang Wan, Xiang Wang, Xiangnan He, Tat-seng Chua

arXiv:2502.05628v3cs.CL

TL;DR

Existing model editing methods struggle with long-form and diverse-formatted knowledge because they rely on single-token hidden-state edits. AnyEdit instead performs autoregressive chunk-wise edits grounded in the mutual-information chain rule, outperforming baselines and extending editing across lengths and formats, while remaining limited to textual knowledge and not explicitly optimized for lifelong editing.

  • Problem

    Existing model editing methods struggle with long-form and diverse-formatted knowledge because they rely on editing a single token’s hidden state.

  • Method

    AnyEdit decomposes long-form outputs into sequential chunks and iteratively edits the key token in each chunk using an autoregressive process grounded in the Chain Rule of mutual information.

  • Results

    AnyEdit outperforms all baselines across datasets, LLMs, and metrics, including over 20% higher BERT Score on UnKEBench.

  • Takeaways & Limitations

    AnyEdit acts as a plug-and-play framework that equips traditional editing methods to handle knowledge with arbitrary length and format.

  • Takeaways & Limitations

    AnyEdit is not explicitly optimized for lifelong editing and currently supports only textual knowledge, lacking multimodal integration.

Abstract

from arXiv · show

Large language models (LLMs) often produce incorrect or outdated information, necessitating efficient and precise knowledge updates. Current model editing methods, however, struggle with long-form knowledge in diverse formats, such as poetry, code snippets, and mathematical derivations. These limitations arise from their reliance on editing a single token's hidden state, a limitation we term "efficacy barrier". To solve this, we propose AnyEdit, a new autoregressive editing paradigm. It decomposes long-form knowledge into sequential chunks and iteratively edits the key token in each chunk, ensuring consistent and accurate outputs. Theoretically, we ground AnyEdit in the Chain Rule of Mutual Information, showing its ability to update any knowledge within LLMs. Empirically, it outperforms strong baselines by 21.5% on benchmarks including UnKEBench, AKEW, and our new EditEverything dataset for long-form diverse-formatted knowledge. Additionally, AnyEdit serves as a plug-and-play framework, enabling current editing methods to update knowledge with arbitrary length and format, significantly advancing the scope and practicality of LLM knowledge editing.

1. Introduction

LLM knowledge editing is efficient for short factual updates but struggles with long-form knowledge and diverse formats because existing methods rely on single-token edits. AnyEdit addresses this with autoregressive, collaborative multi-token editing.

  • LLMs can produce incorrect or outdated information, while retraining and fine-tuning are resource-intensive and risk overfitting.
  • Leading methods struggle with updates exceeding 100 tokens and mostly support structured subject-relation-object triples rather than code, mathematics, or other diverse formats.
  • Long-form diverse knowledge requires multiple interdependent tokens, making single-token hidden-state edits insufficient for consistent and accurate generation.
  • AnyEdit decomposes long-form knowledge into sequential chunks and iteratively edits each chunk’s final token to maximize generation of the subsequent chunk.
  • AnyEdit adaptively adjusts the number of edited tokens and supports poetry, code, and mathematics by decoupling editing from structure-specific constraints.
  • AnyEdit is evaluated against leading methods on standard benchmarks and EditEverything, a diverse-format dataset containing entries up to 458 tokens.

2. Preliminary

Autoregressive LLMs represent inputs through layered hidden states, while model editing locates an influential token and layers before perturbing that token to change the desired output.

  • A decoder-only LLM processes an input sequence through L layers, computing each token’s hidden state by forward propagation.
  • Model editing uses a locate-then-edit paradigm: it identifies a key input token and influential layers, then edits the token’s hidden states.
  • Current methods perturb the key token’s hidden state with a residual δ obtained by gradient descent to maximize the probability of the desired output Y given X.
  • The model parameters are updated so the key token’s hidden state aligns with the edited state.

3. Limitations of Single-token Editing

Single-token editing faces an efficacy barrier for diverse-formatted and long-form knowledge: low initial output probability and weak influence on distant tokens constrain achievable updates.

  • Single-token editing methods struggle to update long-form, diverse-formatted knowledge despite extensive study.
  • Editing Diverse-formatted Knowledge: When the desired output has low original probability, a single-token perturbation must induce a large shift, exceeding the method’s limited capacity.
  • Editing Diverse-formatted Knowledge: Code and mathematical expressions require coordinated changes across tokens because syntax, variable dependencies, and hierarchical structures propagate beyond one token.
  • Editing Diverse-formatted Knowledge: Diverse-formatted knowledge typically combines low original probability with poor editing efficacy under single-token methods.
  • Editing Long-form Knowledge: For outputs exceeding 100 tokens, weakened dependencies between distant tokens reduce the influence of input perturbations on later outputs.
  • Editing Long-form Knowledge: Experiments relating output length, probability shift, and efficacy show poor editing efficacy for long-form knowledge less affected by single-token input edits.
  • Editing Long-form Knowledge: The efficacy barrier is formalized as an upper bound on the average success rate η of making each desired output the most probable after editing.

4. AnyEdit: Autoregressive Model Editing

AnyEdit replaces simultaneous multi-token perturbation with autoregressive chunk-wise editing, using mutual-information theory to avoid interference and scale across knowledge lengths and formats.

  • Single-token editing has an upper bound that diminishes as updated knowledge becomes longer and more diverse, motivating AnyEdit’s collaborative token updates.
  • Theoretical Foundation: AnyEdit decomposes the target output into chunks and applies the mutual-information chain rule so each chunk corresponds to one perturbation target.
  • Theoretical Foundation: Autoregressive dependencies ensure later hidden states do not influence earlier outputs, while determining a chunk subsumes conditioning on its internal hidden states.
  • Theoretical Foundation: Conditioning each mutual-information term on one hidden state eliminates interference between simultaneous hidden-state perturbations.
  • Theoretical Foundation: Chunk-wise editing scales across output lengths and formats because each step updates only one chunk rather than the entire knowledge sequence.
  • Implementation: AnyEdit splits outputs with sliding-window or semantic-segmentation strategies, automatically adjusting edited-token counts to knowledge length.
  • Implementation: It locates each chunk’s final token, edits its hidden state to maximize that chunk’s probability, and updates model parameters with least-squares optimization.
  • Implementation: The resulting plug-and-play process integrates with existing methods to broaden their ability to edit knowledge across arbitrary lengths and formats.

5. Experiments

The experiments evaluate AnyEdit against established editing methods on long-form and diverse-formatted knowledge, including its generalization, scalability, and plug-and-play integration.

  • 5.2. Long-Form Knowledge Editing: AnyEdit outperforms all baselines across datasets, language models, and evaluation metrics.On UnKEBench, it improves BERT Score by over 20%.
  • 5.2. Long-Form Knowledge Editing: On Llama3-8B-Instruct UnKEBench paraphrase questions, AnyEdit improves BERT Score by 32% and ROUGE-L by 56%.These results indicate robust performance under rephrased queries.
  • 5.3. Diverse-Formatted Knowledge Editing: AnyEdit achieves consistent improvements across mathematics, poetry, news, code, and chemistry in the EditEverything dataset.The largest reported ROUGE-L gains occur in Code and News, at 60.58% and 52.38%, respectively.
  • 5.3. Diverse-Formatted Knowledge Editing: AnyEdit maintains stable performance as target-token counts increase, while MEMIT and AlphaEdit decline significantly beyond 30 tokens.The comparison examines performance as the number of edited target tokens grows.
  • 5.4. Boosting Current Editing Methods: Integrating AnyEdit improves existing editing methods across all reported metrics, supporting its use as a plug-and-play enhancement.The evaluation replaces baseline locate stages with autoregressive multi-token identification and editing.
  • 5.4. Boosting Current Editing Methods: AnyEdit introduces an average relative editing-time increase of 24.7% when integrated with other methods.Editing-time comparisons cover Counterfact, MQUAKE, and UnKEBench.
  • 5.5. Chunk-Size Analysis: AnyEdit performance declines when the sliding-window chunk size exceeds a certain threshold.The study varies chunk size to assess its effect on long-form knowledge editing.

6. Related Work

Related work spans parameter-modifying and parameter-preserving knowledge editing, with recent studies extending evaluation to unstructured knowledge beyond structured triples.

  • Unstructured Knowledge Editing: Figure 5 compares original and AnyEdit-enhanced MEMIT, AlphaEdit, and UnKE using yellow and blue bars, respectively.The figure visualizes performance changes after incorporating the autoregressive editing paradigm.
  • Unstructured Knowledge Editing: Table 2 reports average editing time per sample for baseline methods and their AnyEdit-integrated versions across Counterfact, MQUAKE, and UnKEBench.The '+' symbol denotes integration with AnyEdit.
  • Model Editing for Knowledge Update: Parameter-modifying methods encode updates by changing model parameters, whereas parameter-preserving methods add external memory, parameters, or in-context mechanisms.Examples include MEND, ROME, MEMIT, ICE, DeCK, SERAC, and GRACE.
  • Unstructured Knowledge Editing: Unstructured knowledge editing research addresses free-text knowledge beyond structured subject-relation-object triples.AKEW and UnKEBench were introduced to evaluate this setting, while UnKE updates all parameters within a single layer.

7. Conclusion & Limitations

AnyEdit addresses long-form, diverse-format editing through sequential hidden-state updates grounded in mutual-information theory and can integrate with traditional methods.

  • Conclusion: AnyEdit sequentially processes and edits token hidden states over long-form text to produce consistent and accurate knowledge updates.Its design addresses the limitations of single-token editing for complex and unstructured knowledge.
  • Conclusion: The framework is theoretically validated using the Chain Rule of mutual information and integrates with traditional editing methods.This broadens its applicability across knowledge-editing tasks.
  • Limitations: AnyEdit is not explicitly optimized for lifelong editing scenarios requiring continuous and iterative knowledge updates.Adapting the framework to periodic refinement over time remains a future challenge.
  • Limitations: AnyEdit is currently confined to textual knowledge editing and does not support multimodal knowledge integration.The paper identifies cross-modal synchronization across text, images, and audio as a future direction.

Impact Statement

AnyEdit enables precise and efficient updates across diverse knowledge formats, while its flexibility introduces misuse risks requiring careful deployment and oversight.

  • AnyEdit supports precise and efficient updates for long-form and diverse knowledge formats.
  • Greater editing flexibility may enable unauthorized knowledge injection or model tampering.
  • Responsible use requires careful deployment and oversight.

Ackonwledgments

The paper describes benchmark construction, standard evaluation metrics, lexical and semantic assessment, baseline methods, and computing support for its experiments.

  • Benchmark construction: EditEverything combines long, diverse question-answering data from multiple domains, including mathematics and coding.The mathematics data comes from Orca-Math, while coding data comes from MBPP.
  • Evaluation metrics: Standard editing evaluation measures efficacy, generalization, and specificity for structured knowledge updates.These assess target modification, transfer to equivalent queries, and preservation of unrelated knowledge.
  • Evaluation metrics: Long-form and diverse-formatted knowledge requires more flexible evaluation because exact-match efficacy can miss substantively correct answers.The paper incorporates evaluation methods suited to long-form responses.
  • Evaluation metrics: Lexical metrics include BLEU and ROUGE, while BERTScore measures semantic similarity with all-MiniLM-L6-v2 embeddings.
  • Baseline methods: Baselines include FT-L, MEND, ROME, MEMIT, AlphaEdit, and UnKE.The methods span fine-tuning, hypernetwork-based editing, factual association editing, batch updates, interference mitigation, and unstructured-text editing.

A.4. Implementation Details

The implementation uses existing baseline configurations and locate-then-edit components while applying key-value optimization and parameter updates across selected model layers.

  • Implementation: AnyEdit and AnyEdit* primarily follow MEMIT and UnKE baseline configurations.
  • Implementation: AnyEdit on Llama3-8B-Instruct edits layers 4–8, uses the last token, and performs 25 key-value optimization steps at learning rate 0.5.The configuration also uses clamp norm 4, layer-31 loss, weight decay 0.001, and KL regularization factor 0.0625.
  • Implementation: Chunked editing uses 40-token chunks without overlap, while the Qwen2.5-7B-Instruct configuration uses 50-token chunks.
  • Locate-then-edit paradigm: The locate-then-edit paradigm identifies key tokens and influential layers, then edits hidden states to change model outputs.For structured knowledge, the target is typically the last token of the subject; UnKE instead uses the last token of the input.
  • Key-value editing: The MLP input-output pair serves as a key-value pair, with keys computed by forward propagation and values optimized by gradient descent.Parameter updates can use closed-form ROME/MEMIT/AlphaEdit solutions or UnKE’s gradient-based optimization.
  • Parameter updates: MEMIT maintains new and existing associations, whereas AlphaEdit projects updates into the null space of existing keys and UnKE updates an entire layer.

B.2. Proof of Optimization-Conditional Mutual Information Equivalence

The proof connects the negative log-likelihood editing objective to conditional mutual information and decomposes the information across sequential output tokens. Case studies report stronger textual consistency and paraphrase generalization for AnyEdit than most baselines.

  • Optimization–CMI equivalence: Theorem B.1 establishes that the optimization objective is equivalent to maximizing conditional mutual information between X and Y given perturbed hidden state h′.
  • Optimization–CMI equivalence: The proof derives this equivalence by expanding conditional mutual information into entropy terms and showing that the X-independent term does not affect optimization.
  • Mutual-information decomposition: The decomposition relies on the autoregressive property that later hidden states do not influence earlier token generation.
  • Case studies: Case studies use samples from UnKEBench and EditEverything to examine long-form editing performance.
  • Case studies: AnyEdit restores target text with textual consistency and paraphrase generalization, while other baselines often fail to edit effectively.UnKE also performs strongly but shows a noticeable drop on paraphrase questions.

C.2. Supplementary Experimental Results on RQ1 & RQ2

UnKE+ achieves stronger lexical and semantic similarity than other evaluated methods on original and paraphrase questions, while AnyEdit’s performance depends on chunk size and balances quality against editing time.

  • Performance comparison: UnKE+ achieves higher lexical and semantic similarity than other baselines across original and paraphrase evaluations on UnKEBench and AKEW.It improves BLEU, ROUGE-1/2/L, and BERT Score, while MEMIT and AlphaEdit show weaker paraphrase generalization.
  • Performance comparison: UnKE+ provides the best balance between precise knowledge modification and robust generalization among the compared methods.
  • Chunk-size analysis: As chunk size increases beyond a threshold, AnyEdit’s editing performance declines because larger chunks are harder to edit effectively in one iteration.Smaller chunks improve manageability but require more iterations and increase editing time.
  • Chunk-size analysis: A balanced chunk size of 40 is recommended for most AnyEdit editing scenarios.This recommendation reflects the trade-off between editing quality and the time cost of additional iterations.
Loading 2502.05628v3…