StruEdit: Structured Outputs Enable the Fast and Accurate Knowledge Editing for Large Language Models

1 Introduction

• Knowledge editing seeks to update LLM knowledge because pretraining cannot guarantee timely answers as the world changes.The paper frames reliable answers using up-to-date knowledge as crucial for question answering.Source: §1 · p. 1
• Locate-then-edit methods struggle to identify which reasoning tokens to change and to preserve coherence when new and parametric knowledge conflict.The paper says incorrect localization or inaccurate knowledge can cause editing failure, while conflicts can lead to hallucinations or stubborn knowledge.Source: §1 · pp. 1–2
• StruEdit structures outputs, removes potentially affected information, and refills reasoning triplets with updated knowledge in one step.It extracts a source entity and sequential relations, then matches and selects relations in a knowledge structure to infer an answer.Source: §1 · p. 2
• STRUEDIT consistently achieves the highest editing accuracy and fastest speed compared with existing knowledge-editing methods.Figure 1 compares model editing, in-context editing, and STRUEDIT on multi-hop editing tasks.Source: Figure 1 · p. 1, §1 · p. 2

2 Knowledge Editing on Multi-Hop Editing Tasks

• Multi-hop editing must account for edited facts and related updates during reasoning to achieve thorough knowledge revision.The section contrasts this with single-hop fact editing and notes that changing one fact may trigger a ripple effect requiring additional updates.Source: §2 · pp. 2–3, §2.1 · p. 3

2.2 Evaluation and Analysis

• The evaluation uses MQUAKE multi-hop fact chains with LLAMA2-7B-CHAT and GPT-3.5-TURBO-INSTRUCT, testing edit-memory batch sizes of 1 and full.A full batch simulates providing all new knowledge, even when it is not directly relevant to the current task.Source: §2.2 · p. 3
• Structural editing consistently performs best on MQUAKE-2002, leading ME and ICE in the full-batch setting across open- and closed-source models.Table 1 reports accuracy (%) for 2-, 3-, and 4-hop tasks.Source: §2.2 · p. 3, Table 1 · p. 3
• Structural editing has the smallest accuracy drop from batch_size = 1 to batch_size = full because it does not require locating specific knowledge.Full batch provides all new knowledge, including facts unrelated to the current reasoning task.Source: §2.2 · p. 3
• As reasoning hops increase, structural editing maintains 100% accuracy at batch_size=1 and remains stable with full batch.The authors attribute this stability to standardized structured knowledge, which they say is more reliable for reasoning than text and reduces factual hallucinations.Source: §2.2 · pp. 3–4

3 STRUEDIT: An Improved Baseline of Knowledge Editing

• STRUEDIT updates related knowledge rather than retaining parametric knowledge, reducing the burden on LLM reasoning over new facts.The method uses the extracted question logic to reason over an up-to-date knowledge structure.Source: §3 · p. 4

3.1 Structrual Editing on Parametric Output

• STRUEDIT extracts a source entity and sequential relations from an LLM-generated chain, keeps invariant relations, and discards other entities before refilling affected information.The authors describe entities as unstable connections and invariant relations as stable over time.Source: §3.1 · pp. 4–5

3.2 Multi-hop Reasoning with LLMs

• A reasoning path chains the source entity through sequential relations to the final answer in the knowledge structure.The target path contains one entity-relation-entity triplet for each hop.Source: §3.2 · p. 5
• Discrepancies between extracted and structured entities or relations can send reasoning down an alternate path and yield an incorrect outcome.For example, “spouse of” may not match “married to” in the knowledge structure.Source: §3.2 · p. 5
• STRUEDIT queries LLMs to match the source entity against all entities and each extracted relation against relations linked to the previous entity.These alignments define the path within the knowledge structure that leads to the final answer.Source: §3.2 · p. 5

4.1 Datasets and Tasks

• The evaluation uses MQUAKE-3K and its MQUAKE-2002 and MQUAKE-HARD derivatives for multi-hop question answering on counterfactual edits.The authors construct knowledge graphs from MQUAKE triples for STRUEDIT’s knowledge structure.Source: §4.1 · p. 5

4.2 Models and Baselines

• The evaluated models are LLAMA2-7B-CHAT, LLAMA2-13B-CHAT, GPT-3.5-TURBO-INSTRUCT, and GPT-4O-MINI.Source: §4.2 · p. 5
• Baselines span parameter-editing methods ROME, MEND, and MEMIT, plus in-context methods IKE, MeLLo, and DEEPEDIT.Source: §4.2 · pp. 5–6

4.3 Overall Performance

• With full-batch edit memory, STRUEDIT leads ME and ICE on open-source models and averages 58.8% higher accuracy than DEEPEDIT on closed-source models.Its lead is greater on smaller open-source models than on more powerful closed-source models.Source: Table 2 · p. 6, §4.3 · p. 6, Table 3 · p. 7

4.4 Robustness of Knowledge Editing

• Per added hop, ME accuracy decreases 57% on average and ICE 56%, while STRUEDIT declines 38% on average across 2-, 3-, and 4-hop QA.The authors attribute the declines primarily to hallucinations from multi-hop reasoning and knowledge conflicts.Source: Figure 5 · p. 7, §4.4 · p. 7
• As edited instances increase, all methods decline, but STRUEDIT remains best across instance settings and has the smallest average decline.The authors link the broader robustness challenge to localization errors and hallucinations from knowledge conflicts.Source: §4.4 · p. 7, Figure 6 · p. 8

4.5 Editing Latency

• STRUEDIT has the best editing efficiency, even compared with IKE, without relying on lengthy chain-of-thought generation.ME latency includes model editing and reasoning; ICE latency includes knowledge retrieval and reasoning.Source: §4.5 · pp. 7–8, Table 4 · p. 8

• Errors or outdated information in pretraining data can make LLM parametric knowledge inaccurate and lead to hallucinations.Source: §5 · p. 8
• External tools and retrieval-augmented methods provide knowledge evidence, but that evidence can conflict with parametric knowledge and become unreliable.The passage notes this risk particularly when LLMs are overly confident in their parametric knowledge.Source: §5 · p. 8
• Knowledge editing updates outdated information through parameter or structure changes (ME) or prompts containing updated facts (ICE), with both approaches affected by localization or knowledge conflicts.Source: §5 · p. 8

• STRUEDIT removes all parametric knowledge, whether or not it conflicts with new knowledge, unlike ME and ICE.Source: §6 · p. 8
• It extracts entities and relations from questions and uses LLMs to reason over up-to-date knowledge structures to derive answers.Source: §6 · p. 8
• The conclusion characterizes STRUEDIT as offering higher editing accuracy, faster performance, and greater robustness.Source: §6 · p. 8

• STRUEDIT requires a more structured knowledge format than the text-based editing information used by ME and ICE.Source: Limitations · p. 9
• The decline from editing tasks to QA tasks reflects loss in entity and relation extraction for STRUEDIT.Source: Limitations · p. 9
• Performance still declines as reasoning hops and edited-instance counts increase, indicating potential errors in LLM reasoning.Source: Limitations · p. 9

• The study uses only open-source datasets and open-source or scientifically well-established models.Source: Ethical Considerations · p. 9
• Counterfactual public datasets are used to evaluate knowledge updates.Source: Ethical Considerations · p. 9
• The researchers emphasize transparency and responsible technology use for societal benefit.Source: Ethical Considerations · p. 9