Source-linked AI summary
EtCon: Edit-then-Consolidate for Reliable Knowledge Editing
Ruilin Li, Yibin Wang, Wenhong Zhu, Chenglin Li, Jinghao Zhang, Chenliang Li, Junchi Yan, Jiaqi Wang
TL;DR
Knowledge editing methods can struggle to preserve pretrained capabilities and to express edited facts reliably during autoregressive generation. EtCon combines targeted proximal fine-tuning with policy optimization to consolidate edits along generation trajectories. Experiments report improved reliability and generalization while preserving pretrained capabilities in realistic editing scenarios.
Problem
Existing knowledge editing methods can degrade pretrained capabilities and may fail to express parametrically stored edits consistently during autoregressive generation.
Source: §1 · p. 1Method
EtCon uses TPSFT for constrained targeted edits and GRPO to consolidate them by aligning autoregressive generation with updated facts.
Source: §1 · pp. 1–2Results
Takeaways & Limitations
EtCon's edit-then-consolidate framework addresses knowledge editing through both parametric updates and alignment of generation behavior with updated facts.
Source: §1 · p. 2, §7 · p. 8The consolidation stage cannot recover pretrained capabilities lost during editing.
Source: §6.1 · p. 7
Abstract
from arXiv · showhide
Knowledge editing aims to update specific facts in large language models (LLMs) without full retraining. Prior efforts sought to tune the knowledge layers of LLMs, achieving improved performance in controlled, teacher-forced evaluations. However, they still encounter challenges in real-world autoregressive generation scenarios, which greatly limit their practical applicability. Our empirical analysis reveals two issues: (1) Most methods degrade pre-trained capabilities after injecting new knowledge; (2) They may exhibit a discrepancy between stored parametric knowledge and inference-time autoregressive generation behavior. To this end, we propose EtCon, an edit-then-consolidate paradigm that couples targeted edits with post-edit consolidation. Specifically, our framework comprises two stages: (1) Targeted Proximal Supervised Fine-Tuning (TPSFT) performs a constrained targeted edit to update parametric knowledge while controlling policy drift. (2) Group Relative Policy Optimization (GRPO) consolidates the edit by aligning autoregressive trajectories with the intended fact. Extensive experiments demonstrate that our EtCon improves editing reliability and real-world generalization, while better preserving pre-trained capabilities.
1. Introduction
Knowledge editing remains unreliable in autoregressive generation: existing approaches can degrade pretrained capabilities and fail to express edited facts consistently. EtCon pairs constrained targeted edits with post-edit consolidation, improving editing reliability and generalization while preserving capabilities.
1. Introduction
- Existing editing methods can degrade pretrained capabilities through over-specialization and may accumulate drift across lifelong edits.The paper reports that such drift may even lead to model collapse.Source: §1 · p. 1
- Edited facts may be stored parametrically yet fail to persist in autoregressive generation because training uses ground-truth prefixes while inference uses self-generated prefixes.The resulting prefix distribution shift can make edited behavior fragile and prone to reversion.Source: §1 · p. 1
- EtCon uses TPSFT for constrained targeted edits and GRPO for trajectory-level consolidation that aligns parametric knowledge with generation behavior.TPSFT updates knowledge-relevant FFN layers under trust-region constraints using self-generated CoT-based targets.Source: §1 · pp. 1–2
- EtCon improves editing reliability and generalization by 35%-50% over strong baselines under autoregressive generation with natural stopping and an LLM-as-a-judge protocol.Experiments also report enhanced locality and preservation of critical pretrained capabilities.Source: §1 · p. 2
2. Related Work
Related work distinguishes in-place parameter edits from external-assisted methods and describes limitations of common teacher-forced evaluation protocols. The paper positions its approach within in-place editing and adopts a more realistic evaluation setting.
2.1. Overview of Knowledge editing methods
- In-place methods modify model parameters, while external-assisted approaches use auxiliary modules; the latter add deployment complexity despite balancing reliability and locality.In-place methods preserve the vanilla architecture, while external approaches include hypernetworks and external memory.Source: §2.1 · p. 2
- Locate-then-edit methods identify knowledge locations for targeted parameter changes, while PEFT methods update parameters through regularized gradient descent.Both are described as in-place approaches that integrate with existing deployment infrastructure without additional inference latency.Source: §2.1 · p. 2
2.2. Evaluation of Knowledge editing methods
- Standard evaluation measures reliability, generalization, and locality but often uses templated queries, constrained answers, and teacher-forced decoding.The paper therefore adopts a realistic evaluation approach for knowledge editing methods.Source: §2.2 · p. 3
3. Preliminary
EtCon treats parametric editing and behavioral alignment as complementary requirements because teacher-forced success may not transfer to autoregressive generation. It motivates post-edit GRPO consolidation, which improves reliability and generalization across edited baselines.
3.1. Group Relative Policy Optimization
- GRPO optimizes language-model policies through relative reward comparisons within sampled groups while constraining deviation from a reference policy.Its optimization uses a clipped surrogate objective.Source: §3.1 · p. 3
- GRPO computes advantages from multiple trajectories sampled for each query, avoiding a separate critic network and enabling stable policy updates.Source: §3.1 · p. 3
- Trajectory-level optimization suits aligning parametric knowledge that is not reliably applied during autoregressive generation.Source: §3.1 · p. 3
3.2. Motivation for Knowledge Consolidation
- Adding GRPO consolidation to edited methods dramatically improves reliability and generalization across multiple baselines, while applying it to unedited models yields minimal gains.Source: §3.2 · p. 3
4. EtCon
EtCon pairs a constrained, localized parametric edit with post-edit GRPO consolidation so edited knowledge is more reliably generated autoregressively. TPSFT updates selected FFN layers, while GRPO optimizes generation trajectories against a multi-part reward.
4. EtCon
- EtCon uses two stages: localized TPSFT edits selected FFN layers, then GRPO aligns autoregressive trajectories with edited knowledge.GRPO starts from the TPSFT-edited policy and uses it as the reference, refining generation behavior while preserving the parametric edits.Source: §4 · p. 3, Figure 3 · p. 4, §4.2 · p. 5
4.1. Knowledge Editing via Targeted Proximal Fine-Tuning
- TPSFT updates only selected FFN parameters and uses ratio clipping as a trust-region constraint on target-token likelihood updates.When the ratio exceeds 1+ϵ, the gradient is nullified, reducing optimization pressure on well-learned samples.Source: §4.1 · pp. 4–5
- TPSFT trains on self-generated CoT paths paired with target answers, filtering chains logically incompatible with the target.This supervision is intended to associate new knowledge with valid reasoning patterns rather than an isolated input-output mapping.Source: §4.1 · p. 4
4.2. Knowledge Consolidation via GRPO
- Because TPSFT is teacher-forced, its edited knowledge may not surface when inference depends on the model’s own generated history.GRPO addresses this mismatch by optimizing complete autoregressive trajectories scored for accuracy, format, cleanliness, and reasoning consistency.Source: §4.1 · p. 5, §4.2 · p. 5
5. Experiments
EtCon is evaluated for lifelong editing and retained general capabilities across multiple benchmarks and two base models. It achieves the top average score in all six main settings and remains stable through 3,000 sequential ZsRE edits.
5.1. Experiment Settings
- The evaluation uses 1,000 samples each from four editing benchmarks, two 7B–8B instruction models, and sequential lifelong edits with batch size 1.Baselines include four in-place parametric editing methods and two external-assisted methods.Source: §5.1 · p. 5
5.2. Main Results
- EtCon attains top Avg. across all six settings; on Qwen2.5-7B-Instruct it scores 51.5/44.2/56.8 on ZsRE/COUNTERFACT/QAEdit, +39.0/+30.9/+35.7 over runner-up.The six settings span three benchmarks and two base models; the reported gains are driven mainly by higher Reliability and Generalization while maintaining 24.2%–33.6% Locality.Source: Table 2 · p. 6, §5.2 · p. 6
5.1. Experiment Settings
- Editing is assessed with Reliability, Generalization, and Locality on three benchmarks, and edit-wise and multi-hop accuracy on MQuAKE-CF-v2.General ability is measured using C-Eval, LogiQA, and Exact Match and F1 on CoQA, DROP, and SQuAD 2.0.Source: §5.1 · p. 6
5.2. Main Results
- At 3,000 sequential ZsRE edits, EtCon’s Reliability falls from 73.5% to 58.2% and Generalization from 63.1% to 48.7%, while Locality stays within 28%–32%.FT-M’s Reliability drops from 16.6% to near-zero over the same longer-edit evaluation.Source: §5.2 · p. 6
6. Analysis
EtCon combines reliable editing with preservation of general capabilities, while consolidation also improves multi-hop performance. Ablations show that both the constrained edit and carefully designed consolidation rewards matter, and that consolidation cannot undo capability loss caused during editing.
6.1. Effectiveness of Consolidation Stage
- 75.1% Reliability and 63.0% Generalization, with C-Eval at 78.5 vs 79.5 and CoQA F1 at 69.4 vs 70.1, show EtCon balances editing effectiveness and capability preservation.Table 3 presents the comprehensive comparison on Qwen2.5-7B-Instruct.Source: §6.1 · p. 6, Table 3 · p. 7
- 24.2% accuracy on 2-hop and 33.3% on 4-hop queries exceed FT-M’s 2.7% and 4.2%, respectively, on MQuAKE-CF-v2.The authors suggest explicit multi-hop rewards during consolidation could strengthen performance further.Source: §6.1 · pp. 6–7, Table 4 · p. 7
- Adding Consolidation to FT-M and MMKE improves QAEdit Reliability by 25% to 28% and Generalization by about 20%.The authors interpret this as evidence that consolidation bridges edited parametric knowledge and autoregressive behavior.Source: §6.1 · p. 7
- Consolidation preserves general capabilities but cannot restore those lost during editing, so reliable lifelong editing requires lossless knowledge injection as well as consolidation.The authors identify lossless editing as a priority for future work.Source: §6.1 · p. 7
6.2. Ablation Study
- Neither SFT nor TPSFT alone yields reliably editable behavior, but TPSFT substantially reduces the capability degradation observed with SFT.The comparison is reported in Table 5.Source: §6.2 · p. 7, Table 5 · p. 8
- Removing Rcleanliness drops Reliability from 67.1 to 56.1 and Generalization from 53.4 to 22.4; removing Rconsistency lowers Reliability from 67.1 to 51.6.The authors associate these removals with reward hacking and self-contradictions, respectively.Source: §6.2 · p. 7, Table 5 · p. 8
- Clipping threshold ϵ = 0.6 offers the best reported trade-off, while reference update interval 1 performs best for Reliability and Generalization.Larger ϵ can increase policy drift, and larger reference intervals reduce Reliability and Generalization while leaving Locality nearly unchanged.Source: Figure 4 · p. 7, §6.2 · p. 8
- CoT augmentation better preserves Locality and general capabilities despite slightly lower Reliability, while targeted FFN edits outperform full-parameter updates on editing metrics with only a slight GC-Avg drop.Without CoT augmentation, supervising only target tokens leads to overfitting; under the same TPSFT setup, full-parameter updates have weaker editing performance.Source: §6.2 · p. 8, Figure 6 · p. 8, Table 6 · p. 8
7. Conclusion
EtCon uses constrained edits followed by consolidation to improve reliable knowledge editing while preserving pretrained capabilities.
7. Conclusion
- EtCon uses TPSFT for constrained knowledge injection and GRPO to align autoregressive generation with updated facts, improving editing reliability and generalization while preserving pretrained capabilities.The conclusion frames this as an edit-then-consolidate framework for realistic editing scenarios.Source: §7 · p. 8
A.1. Additional Implementation Details
The implementation specifies targeted FFN updates for editing and a weighted GRPO reward for consolidation.
Baseline Configuration
- Baseline experiments use EasyEdit with each comparison method’s default hyperparameter configuration.Further baseline details are attributed to Yang et al. (2025b) and Qi et al. (2025).Source: §A.1 · p. 12
Editing Stage
- EtCon updates only FFN down-projection layers 7–11 in Llama-3-8B-Instruct and 5–9 in Qwen2.5-7B-Instruct, using sequential batch-size-1 edits.TPSFT uses AdamW at 1 × 10^-5, ϵ = 0.6, up to six update steps per edit, and five epochs with early stopping.Source: §A.1 · p. 12
Consolidation Stage
- GRPO consolidation weights Raccuracy at 0.7, Rformat at 0.05, Rcleanliness at 0.15, and Rconsistency at 0.1.All reward components are binary; Raccuracy uses normalized exact-string matching against the ground truth.Source: §A.1 · p. 12, Table 7 · p. 12
A.2.1. EFFECT OF EDITED LAYERS ON ETCON
The appendix examines layer choice, portability, reasoning-oriented models, efficiency, training labels, evaluation, and reward hacking. It reports stronger results from shallow-layer edits and identifies scope limits and reward-design failure modes.
A.2.1. EFFECT OF EDITED LAYERS ON ETCON
- Under identical settings, editing Llama-3 layers 7–11 outperforms layers 17–21 in Locality and Generalization; deeper-layer edits are more prone to reward hacking.The authors propose that conflicts between retained shallow-layer knowledge and deeper injected information may contribute to the behavior.Source: §A.2.1 · p. 13, Table 8 · p. 13, Figure 7 · p. 13
A.2.2. PORTABILITY EVALUATION
- EtCon scores 26.2% on Llama-3 and 24.3% on Qwen2.5, exceeding FT-M’s 18.8% and 21.8%; gains remain limited because training uses single-hop QA only.The authors identify explicit supervision for multi-hop compositionality as a next step.Source: Figure 8 · p. 13, §A.2.2 · p. 14
A.3. Evaluation on Reasoning-Oriented Architectures
- On DeepSeek-R1-Distill-Qwen-7B, shallow layers 5–9 yield 88.6% Reliability, 53.5% Generalization, and 17.0% Locality.The reported comparison identifies this as the best trade-off among tested layer configurations.Source: §A.3 · p. 14, Table 9 · p. 14
A.4. Time Efficiency Analysis
- TPSFT averages 6.01 seconds per QAE edit, while TPSFT+Con approaches convergence by step 15 and converges faster than the other tested consolidation baselines.The editing latency comes from 100 single-instance edits on Qwen2.5-7B-Instruct; consolidation training lasted 15 steps, approximately one hour.Source: §A.4 · p. 15, Table 10 · p. 15, Figure 10 · p. 15
A.5. CoT Generation and Processing
- TPSFT labels combine natural reasoning paths with a verified target answer, and inconsistent generated samples are discarded and regenerated.The reasoning is enclosed in <think> tags and the answer is placed in <answer> tags.Source: §A.5 · p. 16
A.6. Real-world evaluation Details
- Real-world evaluation uses factual and multi-step inputs, evaluates complete autoregressive outputs, and obtains binary judgments from an LLM considering the question, target, and full answer.The evaluation process is described as three stages.Source: §A.6 · p. 17
A.7. Metrics for MQuAKE-CF-v2
- MQuAKE-CF-v2 evaluation samples 1,000 instances across 2-, 3-, and 4-hop questions, measuring direct edit-wise accuracy and multi-hop accuracy.The multi-hop metric tests reasoning that involves edited knowledge.Source: §A.7 · p. 17
A.8. Analysis of Reward Hacking Cases
- Without Rconsistency, outputs may self-correct after stating the target; without Rcleanliness, they may hedge with multiple answers, with removal drops of 11.0% and 15.5%.The authors report these cases as distinct reward-hacking patterns and use the performance drops to quantify the components’ role in consolidation.Source: §A.8 · p. 17, Figure 13 · p. 20, Figure 14 · p. 21