PaperlayerView exact revision on arXiv

Source-linked AI summary

DynEdit: Dynamic Entropy-Guided Sequential Editing for Large Language Models

Jinhu Fu, Yan Bai, Yihang Lou, Li Sun, Jiahong Wu, Xiangxiang Chu, Sen Su

arXiv:2610.34768v1cs.CLcs.AI

TL;DR

Long-sequence editing methods can provide insufficient coverage or allocate effort uniformly across tokens of differing difficulty and information value. DynEdit uses entropy to adapt editing windows and token-level optimization, and consistently improves long-sequence editing, including by up to +18.96 BLEU over AnyEdit∗ on UnKE-Paraphrase.

  • Problem

    Single-position editors provide limited supervision for extended answers, while fixed schedules and uniform weighting fail to distinguish difficult, information-dense tokens.

    Source: §1 · p. 2, §3 · p. 4

  • Method

    DynEdit uses token entropy to adapt sliding-window coverage and to give high-entropy tokens stronger optimization weights within each window.

    Source: §1 · pp. 2–3, §4.2 · p. 6

  • Results

    DynEdit consistently improves long-sequence editing, with up to +18.96 BLEU over AnyEdit∗ on UnKE-Paraphrase.

    Source: §1 · p. 3, §6 · p. 10

  • Takeaways & Limitations

    DynEdit provides an approach for adapting editing allocation to the difficulty structure of long sequences while preserving general model behavior.

    Source: §6 · p. 10

    Entropy may not perfectly reflect token importance, especially in highly structured or domain-specific generation tasks.

    Source: D Limitations · p. 21

Abstract

from arXiv · show

Knowledge editing aims to update outdated or incorrect knowledge in large language models without full retraining. Existing parameter-editing methods remain limited for long-sequence knowledge: single-point editors provide insufficient coverage for extended targets, while fixed chunk-based editors distribute editing effort uniformly across rigid predefined windows, failing to prioritize difficult, information-dense tokens. In this work, we show that token entropy is closely associated with both editing difficulty and semantic importance: high-entropy regions are harder to edit and often contain key factual content. Motivated by this observation, we propose DynEdit, a dynamic framework for long-sequence knowledge editing. DynEdit uses token-level entropy to guide non-uniform sliding, increasing the coverage of difficult high-entropy regions, and further applies entropy-guided token-level optimization to strengthen updates on information-dense tokens within each editing window. Experiments on three benchmarks and two backbone models show that DynEdit consistently outperforms competitive baselines across diverse long-form editing settings, achieving gains of up to +14.0 BLEU points on challenging paraphrase queries and near-saturated semantic scores on several diverse knowledge domains.

1 Introduction

Long-sequence editing is limited by localized updates and fixed allocation that under-serve difficult, information-dense regions. DynEdit uses token entropy to guide denser editing and stronger optimization where those regions occur.

1 Introduction

  • Single-position editors provide limited supervision for extended answers because dependencies are distributed across the sequence.This limitation is termed the single-token editing bottleneck.Source: §1 · p. 2
  • Fixed windows allocate effort without regard to difficulty, redundantly editing low-entropy regions while underserving high-entropy factual details.Uniform within-window optimization can also dilute supervision on entities, dates, and locations with function words.Source: §1 · p. 2
  • DynEdit estimates pre-edit token entropy and reduces sliding stride in high-entropy regions, increasing window overlap and editing exposure there.It uses larger strides through predictable, low-entropy content.Source: §1 · pp. 2–3
  • Within each editing window, DynEdit gives high-entropy tokens larger optimization weights so information-dense tokens receive stronger supervision.This weighting complements denser sliding, which increases how often difficult regions are edited.Source: §1 · p. 3
  • DynEdit consistently improves long-sequence editing, with up to +18.96 BLEU over AnyEdit∗ on UnKE-Paraphrase, while maintaining comparable general ability on MMLU and IFEval.On EditEverything, it gains +27.03 Rouge-L on math and +23.88 BLEU on the longest sequences.Source: §1 · p. 3
  • The method is motivated by the observation that high-entropy tokens are often harder to edit and more semantically informative.The paper evaluates DynEdit across UnKE, CounterFact, MQuAKE, and EditEverything, with multiple backbone models.Source: §1 · pp. 2–3

2 Motivation

The motivation analyses associate token entropy with both editing difficulty and semantic importance. This association supports using entropy to allocate editing capacity across long targets.

2 Motivation

  • Higher-entropy tokens are consistently harder to edit, showing that editability varies across long sequences.The analysis measures pre-edit Shannon entropy and local overlap with the gold answer; larger overlap values indicate harder-to-rewrite regions.Source: §2 · pp. 3–4
  • High-entropy tokens tend to be semantically informative content, whereas low-entropy tokens are mostly function words.The analysis identifies named entities, nouns, verbs, adjectives, and adverbs as higher-entropy content-bearing tokens.Source: §2 · p. 4
  • Together, these findings motivate allocating denser editing to high-entropy regions and focusing optimization on tokens carrying core target content.Fixed editing density and largely uniform optimization can spend capacity on predictable, low-information regions while underserving difficult, important ones.Source: §2 · p. 4

3 Preliminaries

Parameter-modifying locate-and-edit methods inject a learnable shift at a selected hidden state and optimize it against the target sequence. Sequential window editing extends this setup to long targets but retains fixed windows and uniform token weighting.

3 Preliminaries

  • A single learnable memory shift at one layer and position is optimized by minimizing target-sequence negative log-likelihood, but may bottleneck on long, complex targets.The formulation aims to generate the target while preserving unrelated behavior.Source: §3 · p. 4
  • Sequential window editing partitions the target into chunks and optimizes them in order, using the prompt and preceding target tokens as each window’s prefix context.Past perturbations are replayed and the current perturbation is injected at the current edit position.Source: §3 · p. 4
  • Existing sequential editors use fixed window schedules and uniform token weights, motivating dynamic partitioning and adaptive objective weighting.A single perturbation explains all tokens within each window, despite heterogeneous semantics and uncertainty.Source: §3 · p. 4

4 Method

DynEdit uses the pre-edit model’s predictive uncertainty to guide both local edit placement and token-level optimization. Its method estimates token entropy, schedules windows dynamically, and weights optimization within each window.

4.1 Token-Level Entropy Estimation

  • DynEdit computes token entropy once under the pre-edit model; higher entropy indicates lower prediction certainty and often marks tokens harder to edit and more semantically informative.The resulting uncertainty map guides both dynamic scheduling and token-level weighting.Source: §4.1 · p. 5

4.2 Entropy-Guided Dynamic Scheduling

  • Harder, higher-entropy regions receive smaller strides and denser window coverage, while easier regions use fewer editing steps.A constant scheduling function recovers the fixed-stride schedule used by prior methods.Source: §4.2 · p. 6

4.3 Entropy-Guided Token-Level Optimization

  • Entropy weighting addresses gradient dilution: in long sequences, many background tokens can weaken critical-token gradients under uniform aggregation.The paper describes importance-weighted aggregation as increasing alignment with critical update directions.Source: §4.3 · p. 6
  • Within each focus window, entropy-based coefficients increase the influence of difficult, information-dense tokens, while suffix evaluation helps preserve future continuation.All tokens remain in the objective, and weak suffix weights provide a future-aware constraint.Source: §4.3 · p. 7

5 Experiments

The experiments evaluate DynEdit’s editing performance, domain generalization, component contributions, general ability, and robustness across target lengths. Across these settings, results favor entropy-guided editing, including on paraphrased and structurally complex targets.

5.1 Editing performance

  • DynEdit improves long-sequence editing across backbones, with gains up to +12.5 BLEU on UnKE-Original and +3.5 BLEU over AnyEdit∗ under UnKE; paraphrase gains reach +14.0 BLEU on UnKE and +7.5 BLEU on CounterFact.The advantage is also reported on difficult compositional and paraphrased long-form settings, with gains up to +8.3 BLEU.Source: §5.1 · p. 8

5.2 Diverse-Formatted Knowledge Editing Evaluation

  • DynEdit improves over AnyEdit across five knowledge types on both Qwen2.5-7B and Llama-3-8B, with gains up to +27.0 Rouge-L and consistent BERTScore improvements.Gains are most pronounced for structurally complex math, code, and poetry, where critical tokens and sequence coherence matter.Source: §5.2 · p. 8, Table 2 · p. 8

5.3 Ablation Study

  • Removing either dynamic sliding or entropy-guided weighting reduces performance, and the sliding ablation causes the larger drop; ablated variants still outperform AnyEdit in most settings.The reported explanation is that weighting depends on adaptive local editing to focus on informative regions.Source: §5.3 · pp. 8–9, Figure 4 · p. 9

5.4 Analysis of General Ability

  • DynEdit matches or slightly outperforms AnyEdit on most MMLU and IFEval metrics, with consistent improvements under MEMIT and comparable or better results under UnKE.The reported comparison indicates no additional degradation of general ability.Source: §5.4 · p. 9, Table 3 · p. 9

5.5 Length-wise Robustness Analysis

  • As target length increases, both methods degrade, but DynEdit has a flatter performance drop across Q1–Q4 and lower, more stable position-wise errors than AnyEdit.The analysis groups targets into four length quantiles and examines errors along the target sequence.Source: Figure 5 · p. 9, §5.5 · p. 9

6 Conclusion

DynEdit adapts editing allocation to heterogeneous difficulty in long sequences, and the paper reports consistent gains on long and paraphrased targets while preserving general model behavior. This addresses limitations of localized edits and of sequence-level methods that allocate effort without accounting for token-level differences.

6 Conclusion

  • DynEdit dynamically adjusts editing windows and token-level optimization weights using entropy, and experiments report gains on long and paraphrased targets while preserving general behavior.The conclusion characterizes the framework as a simple and effective approach within the evaluated scope.Source: §6 · p. 10

A Related Works

  • Localized edits can suit short triplet facts but fail to cover dependencies across long answers; multi-position editors extend coverage, yet still allocate effort without accounting for difficulty or token-level information.UnKE optimizes over long target sequences, while AnyEdit uses autoregressive sliding windows; the paper identifies their remaining allocation imbalance as motivation for DynEdit.Source: A Related Works · p. 14

B Proof of Proposition 1

The proof compares uniform and weighted gradient aggregation through alignment with the content gradient. It argues that weighting improves alignment when background gradients contain substantial orthogonal components.

B Proof of Proposition 1

  • Uniform aggregation combines the content and background gradients, and its cosine alignment with the content gradient decreases as orthogonal background components grow.The proof attributes the decrease to orthogonal components enlarging the denominator without contributing to the numerator.Source: B Proof of Proposition 1 · pp. 14–15
  • Weighted aggregation improves alignment with the content gradient by amplifying its contribution and reducing the impact of orthogonal background components.For fixed gradients, the proof also states that alignment increases monotonically with α/β, yielding a tighter lower bound than uniform aggregation.Source: B Proof of Proposition 1 · p. 15

C Experimental Supplements

C.1.1 Datasets

The experimental suite spans structured, multi-hop, and long-form knowledge editing benchmarks, with comparisons against representative editing methods. DynEdit is tested using AnyEdit-matched benchmarks and protocols while retaining each backbone editor’s update rules.

C.1.1 Datasets

  • UnKEBench contains 1,000 multi-sentence counterfactual texts, CounterFact uses structured counterfactual triplets, and MQuAKE evaluates propagation through multi-hop reasoning chains.Source: §C.1.1 · p. 15
  • EditEverything covers mathematics, poetry, news, code, and chemistry, with target sequences up to 458 tokens.Source: §C.1.1 · p. 15

C.1.2 Baselines

C.2 Experimental setup

  • Evaluation follows AnyEdit’s benchmarks and protocol, using original and paraphrased queries, locality questions when available, lexical metrics, and BERTScore.Experiments use Llama3-8B-Instruct and Qwen-series models, with generation capped at 512 tokens and near-deterministic decoding.Source: §C.2 · p. 16
  • DynEdit retains backbone-specific update rules but changes target-position scheduling and token-loss weighting; windows use denser strides when their mean entropy exceeds the target-wide mean.At each step, the focus window receives entropy-guided weights while the remaining suffix provides regularization signals.Source: §C.2 · pp. 16–17

C.3 Results of DynEdit in more Editors and Qwen3

DynEdit improves performance across the evaluated editors and base models, with particularly strong gains in paraphrased settings and for editors less effective on long sequences.

C.3 Results of DynEdit in more Editors and Qwen3

  • DynEdit consistently outperforms AnyEdit counterparts across MEMIT, UnKE, and AlphaEdit on lexical and semantic metrics.The table covers additional AlphaEdit and Qwen3-8B results; the left and right values denote original and paraphrase settings, respectively.Source: Table 4 · p. 17, §C.3 · p. 17
  • Gains are especially pronounced on paraphrased queries, indicating stronger robustness to input variation.Source: §C.3 · p. 17
  • DynEdit improves results on Llama3-8B and Qwen2.5-7B and remains effective on the stronger Qwen3-8B model.Source: §C.3 · p. 17
  • Gains are larger for editors less effective on long sequences, including MEMIT and AlphaEdit in long-target scenarios.The authors suggest entropy-guided allocation may compensate for limited long-sequence editing capacity by focusing optimization on critical tokens.Source: §C.3 · p. 17
  • DynEdit is presented as a plug-and-play enhancement that improves different editing frameworks across models and evaluation settings.Source: §C.3 · p. 17

C.4 Time complexity analysis

DynEdit retains its backbone editor’s update modules and memory profile, while entropy estimation and adaptive scheduling add moderate runtime overhead.

C.4 Time complexity analysis

  • Entropy estimation and adaptive sequential scheduling are the main added costs; storing token-level entropy and weights requires negligible extra memory.The entropy map is computed once before editing.Source: §C.4 · pp. 17–18
  • Runtime is mainly proportional to the number of dynamically selected windows, O(NDyn · Cedit).Cedit is the cost of optimizing one local edit.Source: §C.4 · p. 18
  • DynEdit keeps the same edited layers and update modules as AnyEdit, so its memory complexity remains essentially unchanged.Source: §C.4 · p. 18
  • DynEdit matches AnyEdit’s peak memory usage while adding moderate runtime overhead from dynamic sliding and weighted suffix optimization.It remains more memory-efficient than MEND and UnKE, with runtime described as practical for long-sequence editing.Source: §C.4 · p. 18, Figure 6 · p. 18

C.5 Batch Editing performance

In sequential batch editing, DynEdit maintains higher performance and paraphrase robustness as the number of edits grows, although all methods degrade.

C.5 Batch Editing performance

  • DynEdit maintains higher performance on original and paraphrased queries as edit counts rise, with a larger advantage at 16–32 edits.Competing methods show sharp drops in this high-edit regime.Source: Figure 7 · p. 18, §C.5 · pp. 18–19
  • DynEdit achieves higher paraphrase-to-original BLEU ratios than baselines, indicating stronger generalization beyond original surface forms.Figure 7’s right panel compares the ratio as a measure of robustness under linguistic variation.Source: Figure 7 · p. 18, §C.5 · p. 19
  • The authors attribute batch stability to dynamic scheduling that reduces low-information updates and entropy weighting that mitigates gradient conflict and knowledge dilution.Source: §C.5 · p. 19

C.6 Hyperparameter analysis

DynEdit performs best with balanced window and stride settings, preserves difficult factual content in case studies, and allocates more editing coverage to high-entropy regions.

C.6 Hyperparameter analysis

  • Moderate window sizes and dense but not excessive strides yield the best performance; overly large strides reduce difficult-region coverage, while overly small strides add redundant optimization.Very small windows lack sufficient context, whereas very large windows weaken the locality of entropy-guided allocation; performance remains robust across a broad range.Source: §C.6 · p. 19, Figure 8 · p. 19

C.7 Case Studies

  • In the case studies, DynEdit better preserves edited knowledge under original and paraphrased prompts, producing responses more aligned with targets and less prone to verbose hallucinated continuations.The examples include position-sensitive comic titles and a birthplace token that AnyEdit mishandles.Source: §C.7 · pp. 19–20

C.8 Editing Allocation Visualization.

  • Figure 9 shows DynEdit increasing coverage in high-entropy regions, unlike AnyEdit’s nearly uniform window allocation.The entropy curve represents pre-edit token uncertainty, and the lower bands show each method’s optimization coverage.Source: Figure 9 · p. 20, §C.8 · p. 21

D Limitations

The authors identify entropy-proxy accuracy, evaluation scope, and editing cost as limitations, and discuss potential benefits and misuse risks. The checklist also reports reproducibility and experimental-setting information.

D Limitations

  • Entropy may not perfectly reflect token importance, especially in highly structured or domain-specific generation tasks.Source: D Limitations · p. 21
  • DynEdit targets sequential text generation and is evaluated mainly on long-form factual editing; multimodal, interactive, and reasoning-intensive settings remain underexplored.Source: D Limitations · p. 21
  • Additional scheduling and weighting operations raise editing cost relative to fixed-window methods, although the reported overhead is moderate.Source: D Limitations · p. 21

E Broader Impact

  • Long-form editing may improve deployed models’ factual consistency and controllability, while misuse could enable misleading edits or safety-constraint bypasses.Source: E Broader Impact · p. 21

NeurIPS Paper Checklist

  • The checklist reports supplemental code and data, experimental details, error bars defined as one standard deviation across runs, and compute-resource information.Source: §2 · pp. 24–25

Related papers

Loading 2610.34768v1…