Hybrid-Policy Self-Editing for Composable Unstructured Knowledge Editing

Tianci Liu; Zihan Dong; Tianchun Li; Yi-Chung Chen; Qiming Cao; Xingchen Wang; Shiyang Wang; Zichen Miao; Linjun Zhang; Haoyu Wang; Jing Gao

Source-linked AI summary

Hybrid-Policy Self-Editing for Composable Unstructured Knowledge Editing

Tianci Liu, Zihan Dong, Tianchun Li, Yi-Chung Chen, Qiming Cao, Xingchen Wang, Shiyang Wang, Zichen Miao, Linjun Zhang, Haoyu Wang, Jing Gao

arXiv:2608.11660v1Aug 12, 2026cs.CLcs.AIcs.LG

TL;DR

Existing unstructured knowledge editors struggle to make injected passages composable into atomic facts and multi-hop reasoning. HPSE uses privileged-state self-distillation with hybrid rollouts to repair coverage failures, improving composability over pure on-policy distillation.

Problem
Existing unstructured knowledge editors largely fail to support both decomposition into individual facts and composition into multi-hop reasoning because they passively learn from fixed editing passages.
Method
HPSE performs proactive self-distillation from a privileged in-context model state, inserting missing facts into hybrid rollouts while remaining on-policy elsewhere.
Results
Replacing hybrid with on-policy rollouts lowered joint recall by 2.5 points and composition by 3.4 points under the same NLL anchor.
Takeaways & Limitations
HPSE offers a plug-and-play editing-signal improvement applicable across a wide range of gradient-based knowledge editors.
Takeaways & Limitations
Pure on-policy self-distillation can suffer coverage failure because pre-edited rollouts may go off-topic and provide little corrective signal for novel knowledge.

Abstract

from arXiv · show

Large language models (LLMs) achieve remarkable performance across natural language tasks, yet they are trained on static corpora and their knowledge quickly becomes outdated in a fast-changing world. This motivates knowledge editing (KE), which updates specific knowledge in an LLM without changing unrelated others. Recent works move from structured knowledge triples toward unstructured KE (UKE), where the edit is a free-form passage that may state multiple facts at once. Nonetheless, existing editors inject such a passage yet fail to use it: the edited model can recall the passage, but can neither answer atomic questions about its facts nor compose them into multi-hop reasoning. We attribute this missing property, which we term composability, to editors' passive reliance on the fixed passage as the sole learning source. In response, we cast editing as a proactive self-distillation from a privileged in-context state of the same model, which requires no external supervision. We further reveal that due to the novelty of the injected knowledge, the pre-edited model's own rollouts rarely cover it, which limits the effectiveness of pure on-policy distillation. To close this gap, we propose HPSE, which builds a hybrid rollout that steps in to place missing facts onto the student's own trajectory precisely where its coverage fails, while staying on-policy elsewhere. We theoretically analyze HPSE's advantage over pure on-policy distillation, and empirically establish its plug-and-play improvements across four LLM backbones and two KE editors under various scenarios.

1. Introduction

Knowledge editing aims to refresh outdated LLM knowledge without costly retraining, but unstructured edits remain non-composable: models may recall passages without decomposing their facts or composing them for multi-hop reasoning. HPSE addresses this coverage failure with hybrid self-distillation rollouts that improve supervision while remaining plug-and-play across gradient-based editors.

Motivation: Knowledge Editing updates an LLM with new knowledge while avoiding costly retraining and preserving unrelated knowledge and general capabilities.Static knowledge can become outdated in fast-changing settings, causing factual errors or unsafe generations.
Problem: Unstructured edits lose decomposition and composition: models recall whole passages but fail to answer atomic factual questions or perform multi-hop reasoning.These failures make edited knowledge largely less useful than knowledge acquired through pretraining.
Problem: The paper evaluates an untargeted UKE regime in which a free-form statement is paired with a generic prompt that does not identify which facts should be updated.Two representative benchmarks are recast as decomposition and composition probes.
Method: On-policy self-distillation fails under UKE because novel injected knowledge causes the model’s rollouts to go off-topic, creating a coverage failure and limited corrective supervision.A privileged in-context state of the same model supplies token-level distillation targets, but pure on-policy trajectories often miss the new knowledge.
Method: HPSE creates hybrid rollouts by having the privileged model intervene when the edited model deviates too far, yielding better coverage and stronger supervision for UKE.It changes only the training signal, assumes neither a particular parameter update nor parameterization, and is plug-and-play across gradient-based KE editors.

2. Composability Challenge in Unstructured Knowledge Editing

This section defines composability as the ability of unstructured knowledge edits to support both atomic fact retrieval and multi-hop reasoning, rather than mere passage memorization. A benchmark study shows that existing editors substantially fail these capabilities, leaving a gap between injecting knowledge and proactively using it.

Composability Definition: Composability requires decomposition of jointly injected facts and composition of related new knowledge for multi-hop reasoning.Decomposition means answering atomic questions directly; composition means proactively combining distinct facts, closely related to portability.
Untargeted Regime: The untargeted regime pairs a free-form passage with a generic instruction such as “introduce X,” without specifying which contents are updated.This reflects practical settings where human labelers may summarize passages rather than enumerate their contents.
Benchmarking Composability: The decomposition probe tests passage-level joint recall and targeted recall of each of five atomic facts in a free-form passage.The transformed UnKEBench evaluation replaces fact-revealing editing prompts with a more generic summary-style prompt.
Benchmarking Results: COIN⋆ repeats the injected passage instead of directly answering an atomic question, while AnyEdit fails to chain two edits and falls back on prior knowledge.These examples illustrate failures in decomposition and composition, respectively.
Conclusion: Composability remains an open UKE challenge, exposing a gap between passively injecting knowledge and proactively using it.The section positions this unresolved problem as the motivation for the proposed solution in the next section.

3. Proposed Method

HPSE addresses UKE composability by combining proactive self-distillation with targeted privileged step-ins when the edited model’s rollout misses novel facts. Its hybrid rollout and training objective are editor-agnostic, while theory shows a supervision advantage over OPSD that grows with edit length.

Motivation: Passive passage-level supervision limits generalization because it teaches new facts only under the passage’s own prefixes.Data augmentation enlarges the context set but still leaves the model learning from externally fixed targets.
Coverage Failure of OPSD: OPSD enables proactive self-distillation but fails on UKE because novel knowledge causes off-topic rollouts that improve decomposed recall while degrading joint and individual recall.The pre-edited model does not yet contain the new knowledge, so its generated trajectory may diverge from the editing passage.
Hybrid-Policy Self-Editing: HPSE constructs hybrid rollouts that mix edited-model and privileged-state tokens, stepping in when the policies disagree and the privileged model is confident.This places missing facts onto the student’s trajectory precisely where it would otherwise stray from the passage.
Training Objective: HPSE combines a hybrid rollout loss with an NLL anchor, supervising new facts along student trajectories while preserving passage-level likelihood training.The method uses λ = 1 and one greedy hybrid rollout per round for efficiency.
Applicability: HPSE changes the training signal without prescribing edit parameterization, enabling plug-and-play use with different gradient-based KE editors.The paper evaluates this applicability across editors in Section 4.
Theoretical Analysis: The hybrid rollout visits every fact prefix, yielding Ω(ℓ) supervision versus O(1) for OPSD on an edit with ℓnovel novel tokens, so the ratio grows at least linearly with ℓ.The advantage becomes more pronounced as the new knowledge spans more tokens, which is relevant to free-form UKE passages.

4. Experiments

Experiments across four LLMs, two composability benchmarks, and single or continual editing show that HPSE consistently improves gradient-based editors. Ablations and case studies attribute these gains to hybrid rollouts and demonstrate better decomposition and composition without sacrificing locality.

Single Editing: HPSE improved FT-M and LoRA across all 16 single-edit editor–LLM–benchmark combinations, with only occasional metric dips within two points.FT-M gained +6.8 points on MQuAKE-uns and +5.0 points on UnKEBench on average.
Single Editing: HPSE improved decomposition by raising joint and decomposed recall while largely preserving diversity and maintaining stable MMLU locality across four LLM backbones.Baselines either regurgitated lengthy joint answers or achieved diversity alongside poor decomposed and joint performance.
Single Editing: HPSE improved composition, addressing baselines’ frequent gap between recalling edited knowledge and combining it for multi-hop answers.COIN⋆, for example, reached 75.7 Ind. but only 12.0 Cmp. on Llama3.1, whereas HPSE improved composition performance.
Continual Editing: In continual editing, HPSE improved both editors across sequence lengths and LLMs, uniformly for LoRA and with only three FT-M exceptions within one point on average.LoRA’s MQuAKE-uns average relative improvement ranged from +55% to +149% across sequence lengths on both LLMs.
Ablation: Removing both HPSE components reduced performance below base LoRA on MQuAKE-uns, while replacing hybrid rollouts with on-policy rollouts lowered Jnt. by 2.5 and Cmp. by 3.4 points.Under continual editing at T = 10, the corresponding gaps widened to 6.3 and 6.0 points; the NLL anchor improved MQuAKE-uns by 1.8 points but reduced UnKEBench by 1.0 point.
Case Studies: Case studies show HPSE chaining separate edits for two-hop composition and decomposing a five-fact passage to answer an atomic question directly.Baselines instead failed to recall, hallucinated, repeated passages, or reverted to pre-edited knowledge.

5. Related Works

Knowledge editing methods broadly divide into internal storage, which updates model parameters, and external storage, which preserves base parameters while using auxiliary memory. Related work also establishes on-policy self-distillation as an efficient post-training paradigm, while HPSE is presented as augmentation-free and self-contained.

Knowledge Editing: Knowledge editing methods broadly follow two storage paradigms: internal storage and external storage.Internal storage writes new knowledge into model parameters, whereas external storage holds edits in auxiliary memory and freezes the base parameters.
Knowledge Editing: Internal-storage methods localize and update fact-relevant weights or train a small set of additional parameters through LTE, LoRA, and ReFT.Recent unstructured KE methods chunk document-level passages or edit larger parameter sets, but commonly focus on updating a small knowledge-relevant subset of weights.
Knowledge Editing: External-storage methods retain edits through meta-learning, retrieval-augmented generation, or routing over separately learned weights or module copies.These approaches store edits outside the frozen base parameters using auxiliary memory or separately learned components.
HPSE: HPSE is augmentation-free, improving composability without external data or an auxiliary model as a self-contained and affordable path to practical KE.This contrasts with external-storage approaches that rely on auxiliary memory or separately learned components.
On-policy Distillation: On-policy distillation and self-distillation supervise students on their own rollouts, providing dense token-level feedback from a teacher for efficient LLM post-training.Follow-up work strengthens the reliability of the teacher’s signal.

6. Conclusion · A. Omitted Theoretical Analysis · A.1. Notations

The paper concludes that composable unstructured knowledge editing requires both fact decomposition and multi-hop composition, motivating HPSE after existing editors fall short. The omitted appendix develops HPSE’s theoretical analysis by defining its notation and connecting the method to on-policy imitation learning.

6. Conclusion: Composable UKE requires injected edits to be decomposed into individual facts and composed into multi-hop reasoning.The paper studies this requirement in an untargeted regime and finds existing editors largely fall short because they rely passively on a fixed passage.
A. Omitted Theoretical Analysis: The appendix supplies the analysis behind Theorem 3.1, covering notation, theoretical results, method properties, and the connection to on-policy imitation learning.This analysis was omitted from the main body because of page limits.
A.1. Notations: The notation section defines the forward Kullback–Leibler divergence for discrete vocabulary distributions p and q.This divergence appears in the distillation loss of Eq. (1).
A.1. Notations: It also defines binary KL for Bernoulli distributions and notation for distribution modes, vocabulary size, and the uniform per-token KL bound Dmax.The bound is associated with Assumption A.3.
A.1. Notations: The analysis distinguishes the student πθ from the privileged teacher and defines the teacher’s greedy token at each step.These definitions establish the roles used in HPSE’s rollout construction.
A.1. Notations: HPSE emits a teacher token when both step-in-gate conditions hold; otherwise, it emits the student’s token.The gate includes the condition represented by κ in the supplied notation.
A.1. Notations: A fact span is a run of ℓ consecutive positions carrying injected knowledge, with Fj denoting prefixes preceding successive fact tokens y⋆j+1.The fact-span set is F = {F0, . . . , Fℓ−1}.
A.1. Notations: The analysis defines deep coverage and fact-signal for rollout policy µ, with Fj visited when the first j fact tokens are reproduced with probability cj(µ).The fact-signal consequently decomposes over these prefix-visit probabilities.

A.2. Main Result: Signal Separation · A.3. Self-Termination, Consistency, and Locality · A.4. Connection to Imitation Learning

HPSE theoretically separates hybrid from student-only rollout signals by restoring coverage of novel facts, while self-terminating and preserving off-fact updates. Its intervention is a confidence-gated coverage correction analogous to DAgger.

A.2. Main Result: Signal Separation: Under Assumptions A.1–A.3, student coverage decays as c_j(πθ) ≤ e^−τj, whereas hybrid coverage is c_j(πρ) = 1.The hybrid-to-student coverage ratio satisfies c_j(πρ) / c_j(πθ) ≥ e^τj.
A.2. Main Result: Signal Separation: Fact-signal separation scales with span length: Sπθ ≤ Dmax(1 − e^−τ) = Θ(1), while Sπρ ≥ ℓd(κ, τ) = Θ(ℓ).The separation is asymptotic in ℓ with τ, κ, and Dmax fixed, not an absolute claim that Sπθ is always numerically small.
A.2. Main Result: Signal Separation: Under greedy decoding, OPSD visits only the span entrance, whereas the hybrid injects the entire fact span; under sampling, reaching depth j requires Ω(e^τj) draws in expectation.The greedy result concerns per-round signal rather than convergence rate, and the sampled waiting-time result assumes temperature-1 sampling.
A.3. Self-Termination, Consistency, and Locality: The intervention self-terminates once the per-token gap falls to at most τ, because the gate then fails throughout the span.Under the off-fact condition, both policies coincide along the rollout and JHPSE reduces to OPSD’s KL objective plus the NLL anchor.
A.3. Self-Termination, Consistency, and Locality: In the realizable case, hybrid zero-loss minimizers match the teacher target at every fact prefix, while OPSD supervision there is exponentially weak or absent after greedy divergence.An ε-suboptimal OPSD objective can leave the KL at F_j essentially unconstrained because its visitation weight decays exponentially.
A.3. Self-Termination, Consistency, and Locality: For local edits, the gate never fires at off-fact states, so HPSE applies exactly the OPSD update there, although parameter sharing can still propagate in-span changes globally.The locality claim is a per-state consistency check, not a complete guarantee of unchanged parameters or trajectories.
A.4. Connection to Imitation Learning: HPSE resembles confidence-gated DAgger: it inserts the teacher trajectory when student rollouts miss states, correcting occupancy collapse rather than introducing a new distillation target.Its coverage interpretation contrasts OPSD’s multiplicative e^−τj decay with classical behavior-cloning’s additive O(ϵT2) gap.

B. Benchmark Construction Details · C. Implementation Details

The benchmark transformations add only an untargeted editing prompt while preserving original evaluation questions and gold answers for comparability. They convert structured MQuAKE edits into contextual passages and retain evaluation-only fields, including the multi-hop question.

B. Benchmark Construction Details: The benchmark-construction section documents the transformation process omitted from the main body.This section provides additional construction details beyond the main presentation.
B. Benchmark Construction Details: The transformed benchmarks add the untargeted editing prompt “Introduce {s}.” while keeping all original evaluation questions and gold answers unchanged.This preserves comparability with the original evaluation protocols.
B. Benchmark Construction Details: For UnKEBench, Gemini summarizes each edit passage into a short topic phrase used to construct the prompt “Introduce {s}.”.The editing prompt is the only added field; all other fields remain evaluation inputs.
B. Benchmark Construction Details: Each MQuAKE structured counterfactual statement is expanded into a short free-form passage asserting the new fact in natural-language context.The passage-generation prompt from AKEW is used verbatim, with Gemini as generator.
B. Benchmark Construction Details: Each generated MQuAKE passage is installed using “Introduce {s}.”, with {s} taken from the edit’s original subject field.Original questions and gold answers remain evaluation fields, and the multi-hop question is never used during editing.
B. Benchmark Construction Details: The transformed-benchmark examples are presented in Figures 8 and 9.These figures illustrate the resulting benchmark constructions.
C. Implementation Details: The implementation-details section provides further information omitted from the main body.No additional implementation specifics are given in the supplied passage.

C.1. Metric Computation … D. More Experiment Results

The paper specifies composability metrics and evaluation protocols for MQuAKE-uns and UnKEBench, details HPSE’s shared training setup and privileged prompt, and reports additional results showing consistent gains across both KE editors.

C.1. Metric Computation: UnKEBench evaluates joint recall (Jnt.) over all atomic facts and decomposition recall (Dmp.) with one question per atomic fact.The edited model generates up to 200 new tokens for both metrics under greedy decoding.
C.1. Metric Computation: UnKEBench scores editing correctness with FActScore for Jnt. and binary LLM judging for Dmp., while diversity uses SelfBLEU and locality uses MMLU.The UnKEBench judge is gemini-2.5-flash, and SelfBLEU is computed up to 4-grams.
C.1. Metric Computation: MQuAKE-uns evaluates individual recall (Ind.) on single-hop questions and composition recall (Comp.) on paraphrased multi-hop questions requiring all edited facts.Responses generate up to 128 new tokens, with correctness checked by official keyword-based substring matching.
C.2. Training details and hyper-parameters: HPSE experiments use four instruction-tuned backbones spanning Qwen2.5, Qwen3, Llama3.1, and Gemma2, with shared HPSE hyper-parameters.Each outer round constructs one fresh hybrid rollout via greedy decoding, followed by M inner distillation steps; training runs for R outer rounds.
C.2. Training details and hyper-parameters: HPSE uses the edit passage as its sole new-knowledge source, without extra training data or auxiliary models, and is applied on top of LoRA and FT-M.The editor hyper-parameters were tuned for the editors alone rather than for HPSE, placing HPSE at a disadvantage.
C.3. Privileged Model Prompt: The privileged policy is the same frozen base model as the student, conditioned on privileged edit-context information without external knowledge about composition or decomposition goals.Its prompt instructs Qwen2.5, Qwen3, and Gemma models to answer only from established facts and omit unstated details.
D. More Experiment Results: Additional results again show that HPSE improves both KE editors across settings, consistent with the main-body findings.These results were omitted from the main body because of the page limit.

D.1. Full continual editing results

Across continual-editing sequences of lengths T ∈{10, 20, 50, 100}, HPSE outperformed both KE editors in nearly all evaluated settings. Its only three exceptions occurred with FT-M and remained within one point on average.

Evaluation scope: The full continual-editing evaluation covered sequence lengths T ∈{10, 20, 50, 100}.These results underlie Figure 4 and are reported in Tables 6 and 7.
Comparative results: 29 out of 32 settings favored HPSE over both KE editors, with all three exceptions on FT-M within one point on average.This matches the continual-editing pattern described for the main-body results.

D.2. Hyper-parameter sensitivity

HPSE remains robust across broad sweeps of its gap and confidence step-in gates, with no abrupt degradation and only small changes in average score. Although further tuning may improve particular metrics, HPSE does not hinge on careful gate selection.

Gate sensitivity: HPSE’s gap gate τ and confidence gate κ jointly determine when the privileged model steps into the hybrid rollout.Each gate was swept over an 8× range around its default while other hyperparameters remained fixed, using UnKEBench (Qwen2.5) and MQuAKE-uns (Qwen3).
Robustness: Further gate tuning could improve particular metrics, such as direct recall (Jnt.) for decomposition (Dmp.) on UnKEBench with a larger τ.The authors did not tune either gate beyond the minimal calibration described in Appendix C.2.

D.3. Additional ablation studies

Additional continual-editing ablations show that HPSE’s advantage over removing hybrid placement widens substantially as edits accumulate, across compositionality and average scores on both benchmarks.

Continual editing: After T = 10 edits, HPSE’s gaps over “w/o HP” widen to 6.3 points on Jnt. and 6.0 points on Cmp., from 2.5 and 3.4 at T = 1.The comparison uses the identical configuration as Table 2 under continual editing.
Continual editing: Average-score gaps grow from 0.4 to 5.3 points on UnKEBench and from 2.4 to 4.9 points on MQuAKE-uns as edits increase from T = 1 to T = 10.These results confirm the benefit of the hybrid rollout under continual editing.

D.4. Additional case studies

Additional case studies illustrate OPSD’s coverage failure and composability failure modes across composition and decomposition tasks. The examples use Qwen2.5 and Llama with injected multi-fact edits and passages.

Coverage failure: Figure 12 shows OPSD’s coverage failure.The passage refers readers to Section 3.1 for background on this failure.
Failure modes: Figures 13 to 16 provide additional examples of composability failure modes with error highlighting.Figures 13 and 14 cover composition, while Figures 15 and 16 cover decomposition.
Composition: Composition case studies on Qwen2.5 use edits chaining employers, sports, creation locations, continents, makers, and founders.The injected chains include Uli Hoene → FC Groningen → cricket → Australia → South America, and Ford Territory → Colt’s Manufacturing Company → Lech Wałęsa.
Decomposition: Decomposition case studies on Llama include an injected passage describing Samuel Wells Williams as an award-winning motion picture director.The passage also states that Williams directed acclaimed films, was praised for developing actors and innovative filmmaking, and achieved box-office success.