Decoding by Contrasting Knowledge: Enhancing LLMs' Confidence on Edited Facts

• As real-world information changes, LLM knowledge can become outdated and potentially produce factual inaccuracies or false information.Source: §1 · p. 1
• Knowledge editing updates facts without the computational burden of retraining, while ICE guides frozen LLMs through contextual prompts without changing model parameters.Source: §1 · pp. 1–2
• ICE struggles with stubborn knowledge: strongly pretrained facts that external prompts may not alter, even with retrieval, conflict checking, or reasoning guidance.Source: §1 · p. 2
• ICE raises the predicted probability of edited facts, but a residual gap between new and original parametric knowledge can still impede editing.Source: §1 · p. 2
• DeCK enhances attention to new knowledge and contrasts post-editing token distributions with original parametric distributions to predict the next token.Source: §1 · p. 2
• Extensive MQUAKE experiments report that DeCK enhances ICE without changing model internals or external prompts, and the authors present it as a step toward effective, accountable knowledge editing.Source: §1 · p. 2

• LLM decoding computes a next-token distribution conditioned on preceding tokens, then samples from it to continue generation.Source: §2 · pp. 2–3
• Figure 2 tracks the first output tokens representing new and parametric knowledge by their logits and ranks in the vocabulary.Source: Figure 2 · p. 3
• Knowledge editing changes model behavior so an edited fact replaces the original, including knowledge needed to answer affected multi-hop questions.Source: §2 · p. 3

3.1 How ICE Effectively Edits Knowledge in LLMs?

• ICE’s effectiveness remains difficult to explain because existing methods rely on black-box modeling of LLMs.The section frames its analysis as an investigation of ICE’s intrinsic mechanisms.Source: §3.1 · pp. 3–4
• ICE substantially raises edited-knowledge logits and top ranks, while parametric-knowledge logits remain largely unchanged or decrease.The shift increases the model’s confidence in edited facts.Source: §3.1 · p. 4

3.2 Inherent Challenges of Stubborn Knowledge

• Stubborn knowledge describes edits that fail because existing parametric knowledge is too confident or new knowledge is not confident enough.Even after editing, the new answer can remain just below the parametric answer; one example has English trailing French by 0.516.Source: §3.2 · p. 4

4.1 Editing Signal Enhancement

• Editing Signal Enhancement increases attention to edited facts by adjusting original logits with semantic relevance scores and targeting higher probabilities for related tokens.It defines Knowledge Enhancement Divergence (KED) between the enhanced next-token distribution and a target distribution; token weights reflect semantic relevance.Source: §4.1 · p. 5

4.2 Decoding by Contrasting Knowledge

• DeCK subtracts parametric-knowledge log probabilities from ICE-guided log probabilities, with γ controlling the subtraction’s proportion.Adaptive plausibility constraint filters out tokens with low probabilities under the editing output.Source: §4.2 · p. 6
• Without adaptive plausibility filtering, subtraction can amplify extremely low-probability tokens and produce implausible words.Editing Signal Enhancement raises the new-knowledge signal before contrastive processing so it is not filtered out.Source: §4.2 · p. 6

4 DeCK: Enhancing LLMs’ Confidence on Edited Facts

• DeCK contrasts the enhanced ICE distribution with the original parametric distribution to predict the next token, while maintaining both token sequences during generation.The two sequences may have different lengths, so DeCK synchronizes their updates by appending each new token to both.Source: §4 · p. 4, §4.2 · p. 6

5.1 Experimental setup

• Experiments covered MQUAKE-3K and its MQUAKE-2002 and MQUAKE-HARD derivatives, three LLAMA-CHAT models, and MISTRAL-7B-INSTRUCT.Baselines included IKE, MeLLo, and ROME; IKE used multi-hop demonstrations and chain-of-thought prompting, and DeCK was deployed on IKE and MeLLo.Source: §5.1 · pp. 6–7

5.2 Main Results

• 63% and 219%: on HARD, DeCK increased ICE’s editing success rate for LLAMA2-13B-CHAT and LLAMA3-8B-INSTRUCT, respectively.In the batch-size-1 evaluation, IKE with DeCK consistently achieved the best performance across models and datasets.Source: §5.2 · p. 7, Table 1 · p. 7
• In full-batch experiments, DeCK improved MeLLo consistently, while IKE did not improve consistently.Table 2 evaluates accuracy on LLAMA2-CHAT models using the full edit-memory batch size.Source: §5.2 · p. 7, Table 2 · p. 8

5.3 Metamorphosis of Stubborn Knowledge

• DeCK improved the ranking of new knowledge that IKE had failed to edit, including cases where its original probability rank was between 2 and 100.Table 3 reports average new-knowledge ranking and its improvement after DeCK.Source: §5.3 · pp. 7–8, Table 3 · p. 8
• On STUBBORN datasets, IKE’s performance declined and fell below ROME on LLAMA2-13B-CHAT; DeCK improved IKE by up to 80% on that model.The datasets group examples by how often IKE failed to edit the same knowledge across different questions.Source: §5.3 · p. 8, Table 4 · p. 8

5.4 Ablation Study

• A too-large γ can reduce DeCK’s performance below original ICE, while editing-signal enhancement consistently helps keep edited knowledge from being filtered out.The ablations examine the contrasting coefficient and editing-signal enhancement components.Source: §5.4 · p. 9, Table 5 · p. 9, Table 6 · p. 9

Hallucinations and Misinformation

• Hallucination-mitigation work includes training-stage data curation and knowledge grounding, and inference-stage confidence estimation, retrieval, and knowledge editing.The passage identifies hallucination as a source of LLM-generated misinformation.Source: §6 · p. 9

Contrast Decoding

• Contrast-decoding methods compare distributions from expert and amateur models, mature and premature layers, or models injected with hallucinations.The cited approaches are CD, DoLa, and ICD, respectively.Source: §6 · p. 9

Model Editing and In-Context Editing

• Model editing changes internal model structure, whereas ICE uses edited facts and retrieved demonstrations in prompts.The passage describes these as approaches to knowledge editing.Source: §6 · p. 9

• DeCK significantly improves editing accuracy, according to the reported experimental results.Source: §7 · p. 9
• DeCK contrasts logits from new edited knowledge with parametric knowledge to amplify the changes introduced by in-context editing.Source: §7 · p. 9
• DeCK requires two token sequences during generation, resulting in approximately 1.6X higher latency than original decoding.Source: §7 · p. 10