Source-linked AI summary
DOW-KE: Anchor-Free Multi-Layer Knowledge Editing via Direct End-to-End Weight Optimization
Ran Chen, Junbo Zhang, Qianli Zhou, Xinyang Deng, Wen Jiang
TL;DR
Multi-layer knowledge editing can optimize intermediate activation targets without directly optimizing their realized joint effect. DOW-KE instead jointly optimizes deployed weight updates through full-model backpropagation, achieving the highest composite Score in five of six evaluated settings.
Problem
Multi-layer locate-then-edit methods leave a gap because locally realized layer updates can fail to faithfully deliver their prescribed intermediate targets.
Method
DOW-KE removes intermediate anchors and jointly optimizes all edited-layer weight updates through full-model backpropagation with preservation constraints embedded in the computation graph.
Results
DOW-KE achieves the highest composite Score in five of six evaluated model-dataset settings across three models and two datasets.
Takeaways & Limitations
DOW-KE directly aligns the optimized update with the deployed update while retaining fact-specific response scope in multi-layer knowledge editing.
Takeaways & Limitations
The paper does not separately validate whether joint optimization absorbs unfavorable transport at one layer through the others.
Abstract
from arXiv · showhide
Multi-layer locate-then-edit methods for knowledge editing first optimize target residual-stream activations (anchors) at selected layers, then realize them layer by layer as weight updates. This pipeline optimizes an intermediate representation but deploys multi-layer weight updates whose joint effect through the true forward pass is never itself optimized: regardless of how anchors are set or propagated, each update comes from a local solve, so propagation-induced attenuation and distortion go uncorrected, leaving a closure gap between anchor targets and realized edits. We propose DOW-KE, an anchor-free method built on a single principle: what is optimized must be exactly what is deployed. DOW-KE backpropagates the final editing objective through the complete model, jointly optimizing the updates of all edited layers so cross-layer propagation and coupling enter every gradient step. The same principle dictates where preservation resides: embedding the preservation projection in the update parameterization, inside the computation graph, makes every gradient act on the deployed update; post-hoc constraints would reopen the gap, and the constrained search keeps edits clear of protected knowledge. In large-scale sequential editing on two datasets and three models, DOW-KE achieves the highest overall Score and neighborhood Specificity in five of six model-dataset settings among the evaluated baselines.
Introduction
LLM knowledge is distributed implicitly across parameters, making individual facts difficult to modify. DOW-KE addresses structural misalignment in multi-layer locate-then-edit methods by directly optimizing deployed weight updates through full-model backpropagation.
- LLM knowledge is implicitly distributed across parameters, making individual facts difficult to modify directly.Corpus-level updates commonly use continued pretraining or fine-tuning.
- Locate-then-edit methods identify target-associated parameters and optimize intermediate activation targets called anchors.ROME is described as a representative single-layer method, while multi-layer methods distribute updates across preceding shallower layers.
- Multi-layer editing assumes layerwise increments propagate approximately losslessly and combine linearly, but this leaves a gap between delivered and intended edits.The implicit linear-superposition assumption does not hold.
- Propagation attenuation, distortion, and interactions remain invisible to local objectives, while unobserved delivery can cause linked adverse effects.Because terminal effects are excluded from the objective, these losses cannot be detected or corrected.
- DOW-KE uses the final editing objective and jointly optimizes all edited-layer weight updates through full-model backpropagation.This anchor-free design reflects attenuation, distortion, cross-layer coupling, and subspace constraints directly in the gradients, eliminating delivery assumptions.
Related Work
Knowledge editing includes direct parameter modification methods that locate and distribute edits across model layers, as well as approaches using auxiliary memories or meta-learned parameter updates.
- Direct parameter modification: ROME pioneered locate-then-edit methods by optimizing an anchor target before converting it into a closed-form weight update.Its procedure uses causal tracing to identify fact-carrying layers and gradient descent to optimize the anchor.
- Direct parameter modification: MEMIT distributes residual updates across several layers, while AlphaEdit and EvoEdit constrain updates using protected-knowledge and prior-edit projections.AlphaEdit projects onto the null space of protected-knowledge keys; EvoEdit extends this projection to prior edits.
- Auxiliary parameters and external memory: Auxiliary-parameter methods leave the original model unchanged and store new knowledge in routed side memories or dynamic LoRA modules.WISE and MELO exemplify this external-memory paradigm.
- Meta-learning: Meta-learning methods train hypernetworks to transform naive fine-tuning gradients into parameter updates satisfying locality constraints.KE and MEND use this paradigm to generate edits through an auxiliary network.
Preliminaries
The preliminaries model FFN down-projection weights as associative memories that map key patterns to residual-stream values. They then formalize locate-then-edit methods as optimizing a deepest-layer anchor before converting its residual into sequential, preservation-aware weight updates.
- Associative-memory formulation: FFN down-projection matrices store key-value associations, so knowledge editing rewrites mappings from input patterns to residual-stream increments.The feature map is represented as ϕl(x) = σ(W l x), with W l k = v under the associative-memory view.
- Anchor optimization: Locate-then-edit methods optimize a deepest-layer anchor z using target likelihood and KL preservation objectives, averaging likelihood across random prefixes for robustness.The anchor specifies a desired residual-stream state but does not directly modify weights.
- Weight realization: The second stage converts anchor residuals into per-layer target values and solves regularized least squares using edit keys alongside preserved knowledge keys and values.The residual R specifies the desired correction, while joint key statistics constrain the solution directions.
- Sequential multi-layer editing: MEMIT distributes requests across a layer window, recomputes states after preceding updates, and evenly allocates remaining residuals, but each closed-form solve fits only its local target.Consequently, the layerwise procedure does not directly enforce attainment of the final anchor target.
Empirical Study
Multi-layer editing retains a closure gap because independently realized updates lose fidelity during preservation-constrained fitting and cross-layer transport. DOW-KE removes intermediate anchors and directly optimizes deployed layerwise updates through the full forward pass.
- Closure gap: The terminal residual gap is rgap ≈ 0.33, showing that multi-layer anchor targets remain partly unrealized.The gap jointly reflects failures in local target realization and lossless transport through the residual stream.
- Failure mechanisms: Preservation terms discount edit-target realization, while subsequent layers dilute and distort increments through attention and FFN responses.These effects accumulate because each layer independently solves a lossy realization problem.
- Related remedies: BLUE and FE improve anchor targets through actual forward computation, but their closed-form weight updates still lack feedback from realization errors.Those errors therefore propagate and accumulate through the residual stream.
- DOW-KE: DOW-KE directly optimizes layerwise weight updates against the final editing objective, placing transport, cross-layer coupling, and batch interference inside the forward-pass computation.Its update parameterization retains edit-key and preservation statistics while removing the anchor as an intermediate target.
- Limits of target remedies: Better target construction does not resolve realization: standard AlphaEdit and FE remain near a realization ratio of 0.5.Post-hoc scaling also cannot close the gap; the composite score peaks at 86.89 when α=0.9.
Method
DOW-KE replaces anchor residual allocation with directly optimized per-layer variables and jointly optimizes them through the deployed model’s final editing objective. Its update parameterization embeds input/output preservation constraints, while positional gradient routing prevents pre-subject writing without changing forward computation.
- Direct update parameterization: DOW-KE fixes the solution operator B_l and directly optimizes residual variables A_l, yielding each layer update Δ_l = A_lB_l.The residual allocated from the anchor target is replaced by a directly optimized variable at every edited layer.
- Preservation constraints: Input and output preservation are embedded in the deployed update: B_l suppresses selected key directions, while P_l^out excludes unrelated-knowledge output directions.The threshold τ controls the protected input subspace’s spectral-energy coverage and shifts the empirical Specificity–Generalization trade-off.
- Positional restriction: The positional rule changes only backward gradient routing, stopping direct gradients from pre-subject positions while leaving forward logits, loss, and deployed updates unchanged.The standard gradient path remains active through the subject’s final position, allowing backpropagation through the network.
- Cross-layer coordination: Removing residual allocation lets the end-to-end objective determine cross-layer division of labor, so other layers can absorb unfavorable transport at one layer.This benefit is described as a by-product of joint optimization and is not separately validated.
- End-to-end optimization: The variables {A_l} are jointly optimized using editing and KL losses evaluated on the model containing all layer updates.Unlike anchor optimization, both losses are evaluated after applying the updates at all edited layers.
Experiments
Experiments evaluate DOW-KE across three language models and two datasets under sequential editing, using efficacy, generalization, specificity, and overall Score. Results show strong performance, while ablations examine direct weight optimization and preservation constraints.
- Experimental setup: The sequential protocol applies 2,000 requests from each dataset to the same model in 20 successive batches of 100 requests.Efficacy, Generalization, and Specificity are measured on edit, rephrased, and neighborhood prompts, while their harmonic mean is reported as overall Score.
- Main results: Table 1 reports results averaged over five runs, with DOW-KE best in most settings and second-best in the remaining settings across three models and two datasets.The comparison includes locate-then-edit methods and fine-tuning-based baselines; DOW-KE has a clear advantage over BLUE in five of six model–dataset combinations.
- Ablations: Table 2 ablates DOW-KE’s anchor removal, direct weight optimization, and differentiable preservation-subspace constraint using AlphaEdit with the energy-threshold rule as baseline.The table also matches operating points through the weight-scaling coefficient α, which controls editing performance and Specificity.
- Ablations: Increasing the preservation threshold τ improves Specificity but reduces Generalization, while output-side preservation complements key-side preservation by strengthening protection of unrelated knowledge.The passage attributes the trade-off to fewer editable directions and narrower fitting, but identifies this explanation as requiring future verification.
Conclusion
DOW-KE is an anchor-free knowledge-editing method that closes the gap between intermediate targets and their realization by optimizing deployed multi-layer updates in one differentiable graph. Across three models and two datasets, it achieves the highest composite Score in five of six evaluated settings.
- Method: DOW-KE removes intermediate activation anchors and layerwise residual allocation, incorporating preservation constraints, transport attenuation, directional distortion, and cross-layer coupling into one differentiable computation graph.The optimized update is therefore the update that is deployed.
- Method: Edit keys and preservation statistics predetermine an update’s response scope, keeping each edit tied to a specific fact.
- Results: Highest composite Score: five of six evaluated settings across three models and two datasets.