PaperlayerView exact revision on arXiv

Source-linked AI summary

Taming Knowledge Conflicts in Language Models

Gaotang Li, Yuzhong Chen, Hanghang Tong

arXiv:2503.10996v2cs.CLcs.LG

TL;DR

Knowledge conflicts occur when parametric memory contradicts contextual information, and prior accounts leave the mechanisms and flexible mitigation of these conflicts incompletely understood. The paper identifies attention heads that encode both sources and proposes JUICE, a test-time dual-run intervention. Across 11 datasets and 6 architectures, it reports state-of-the-art performance and consistent improvements.

  • Problem

    Language models can face contradictory parametric and contextual knowledge, while prior work leaves conflict mechanisms and flexible steering toward either source relatively unexplored.

    Source: §1 · pp. 1–2

  • Method

    JUICE identifies reliable attention heads and uses a dual-run test-time intervention to steer models toward parametric or contextual knowledge without fine-tuning.

    Source: §1 · p. 2

  • Results

    Across 11 datasets and 6 architectures, JUICE achieves state-of-the-art performance, robust generalization, and significant, consistent improvements.

    Source: §1 · p. 2, §6 · p. 9

  • Takeaways & Limitations

    The analysis supports a view of knowledge conflict involving attention heads that encode both knowledge sources and offers a practical intervention for steering models between them.

    Source: §1 · p. 2, §6 · p. 9

    The theoretical training analysis assumes a perfectly learned first layer, linear attention, and strictly orthogonal embeddings.

    Source: §5.1 · p. 8

Abstract

from arXiv · show

Language Models (LMs) often encounter knowledge conflicts when parametric memory contradicts contextual knowledge. Previous works attribute this conflict to the interplay between "memory heads" and "context heads", attention heads assumed to promote either memory or context exclusively. In this study, we go beyond this fundamental assumption by uncovering a critical phenomenon we term the superposition of contextual information and parametric memory, where highly influential attention heads simultaneously contribute to both memory and context. Building upon this insight, we propose Just Run Twice (JuICE), a test-time attention intervention method that steers LMs toward either parametric beliefs or contextual knowledge without requiring fine-tuning. JuICE identifies a set of reliable attention heads and leverages a dual-run approach to mitigate the superposition effects. Extensive experiments across 11 datasets and 6 model architectures demonstrate that JuICE sets the new state-of-the-art performance and robust generalization, achieving significant and consistent improvement across different domains under various conflict types. Finally, we theoretically analyze knowledge conflict and the superposition of contextual information and parametric memory in attention heads, which further elucidates the effectiveness of JuICE in these settings. Our code is available at https://github.com/GaotangLi/JUICE.

1. Introduction

The introduction frames knowledge conflict as a choice between parametric beliefs and contextual information, and argues for a unified way to steer models toward either source. It presents JUICE, motivated by attention heads that encode both sources, and reports broad empirical and theoretical support for the approach.

1. Introduction

  • Knowledge conflicts arise when parametric memory contradicts contextual information, and models may need to prioritize either source depending on the application.Prior work leaves underlying mechanisms and broad mitigation strategies relatively unexplored, while often focusing on contextual reliance alone.Source: §1 · pp. 1–2
  • Attention components can encode contextual information and parametric memory in superposition rather than contributing exclusively to one source.The study also reports inconsistent component behavior across conflict degrees and counteracting effects from multiple individually effective interventions.Source: §1 · p. 2
  • JUICE steers models toward parametric or contextual knowledge without fine-tuning by identifying reliable attention heads and intervening through a dual-run approach.The first run saves identified-head outputs; scaled versions are used to intervene during the second run.Source: §1 · p. 2
  • Theoretical analysis treats knowledge conflict as conflicting inference tasks arising from distinct training tasks and supports the effectiveness of JUICE in this setting.It also analyzes how attention heads can contribute to both parametric and contextual knowledge and how standard training encourages such heads.Source: §1 · p. 2
  • Across 11 datasets and 6 architectures, extensive experiments show JUICE sets new state-of-the-art performance and robust generalization, achieving significant and consistent improvements.Source: §1 · p. 2

2. Problem Setup

The paper studies both preserving parametric beliefs when contexts mislead and prioritizing context when model knowledge is unreliable. It evaluates these settings using factual-recall conflicts, context-oriented benchmarks, and six open-source base models.

2. Problem Setup

  • The setup treats parametric and contextual knowledge as complementary targets, depending on whether the provided context is misleading or accurate.When context is misleading, the goal is to enhance reliance on parametric beliefs; when context is accurate and prior knowledge may be outdated or incorrect, the goal is to increase contextual reliance.Source: §2 · pp. 2–3

2.1. Parametric Datasets

  • Six parametric datasets cover factual domains and sentence-level substitution and paragraph-level coherent-counterfactual conflicts.The study also includes conflict-free cases, ensures conflicting contextual answers are factually incorrect, and primarily evaluates exact-match accuracy against the factually correct answer.Source: §2.1 · p. 3

2.2. Contextual Datasets

  • The contextual setting treats the input context as the target and model prior knowledge as unreliable, using Memo-Trap and NQ-Swap benchmarks.Performance is evaluated by exact-match accuracy with respect to the contextual answer.Source: §2.2 · p. 3

2.3. Models

  • The benchmark uses six open-source base models, with Gemma used mainly for analysis and all six used to evaluate intervention methods.The models are Gemma-2b, Llama2-7B, Llama3-8B, Phi2-2.7b, StableLm2-1.6b, and Olmo-7b.Source: §2.3 · p. 3

3. Interpreting and Resolving Knowledge Conflicts

Component effects change with conflict type and can counteract one another, challenging fixed memory-head/context-head roles. JuICE addresses these effects through consistent head selection and a dual-run intervention.

3.1. Analysis

  • 0 whole layers, 1 MLP, and 6 attention modules (of 26) yielded consistent parametric gains across conflict types; knockout effects varied across clean, substitution, and coherent prompts.Removing nearly all components lowered parametric-answer probability on clean prompts but raised it on coherent-conflict prompts.Source: Figure 3 · p. 4, §3.1 · p. 4
  • Half of the four most influential memory heads under substitution conflicts had the opposite effect under coherent conflicts.The table ranks heads by contextual-probability increase under substitution-conflict knockouts.Source: Table 1 · p. 4, §3.1 · p. 4

3.2. Our Approach: Just Run Twice (JUICE)

  • JuICE runs the model twice, saving identified-head outputs first and adding scaled versions to corresponding activations in the second run.The first-run activations are intended as more reliable steering directions to mitigate counteracting effects.Source: Figure 4 · p. 5, §3.2 · p. 5

3.1. Analysis

  • Combining individually helpful interventions could reduce performance, as interventions may counteract one another.The authors link this to modified earlier-layer activations changing downstream component behavior.Source: §3.1 · pp. 4–5, Table 2 · p. 5
  • CP Superposition names component roles that depend on their inputs, so a component may contribute to both contextual information and parametric memory.The analysis ties the changing roles to the residual streams received from prior layers.Source: §3.1 · pp. 4–5

3.2. Our Approach: Just Run Twice (JUICE)

  • JuICE selects attention heads whose individual scaling consistently improves the desired probability across conflict types, retaining the top K by aggregated scores.Head scores are computed on a small selection dataset; selected heads must have non-negative scores across all conflict types.Source: §3.2 · p. 5
  • Primary experiments use four head-selection samples; the number of intervened heads is fixed, and inference scaling factors are set using validation data.The authors also test identification in one domain followed by evaluation across other domains.Source: §3.2 · p. 5

4. Intervention Experiment

Across parametric-belief interventions, JuICE outperforms baselines in most cases and can nearly reverse contextual answering even under coherent conflicts. The experiments distinguish three conflict types and also report contextual-knowledge interventions.

4.1. Enhancing Parametric Beliefs

  • Parametric-belief experiments distinguish no-conflict, substitution-conflict, and coherent-conflict settings.The results are presented across different models.Source: §4.1 · pp. 5–6
  • JUICE consistently and significantly outperforms all baselines in most cases, nearly reversing contextual-answer tendencies even under coherent conflicts.Table 3 reports accuracy (%) for parametric-memory interventions; additional model results are in Appendix D.2.Source: Table 3 · p. 6, §4.1 · p. 6
  • The section also reports contextual-knowledge intervention results, following the same convention as the parametric-memory results.Table 3 reports accuracy (%) and marks the best result in bold.Source: Table 4 · p. 6, Table 3 · p. 6

2. JUICE achieves improvements on zero-shot clean

JUICE improves contextual knowledge promotion and remains effective across hyperparameter and prompt variations. The dual-run design avoids the instability of combining individually effective interventions in one pass.

2. JUICE achieves improvements on zero-shot clean

  • PH3 and Prompt fail to address coherent conflicts effectively; JUICE performs better with smaller identification sets and transfers across domains from world-capital-selected heads.PH3's transfer is limited to closely related datasets.Source: §2 · p. 6

4.2. Enhancing Contextual Reliance

  • JUICE consistently outperforms all baselines on average and promotes contextual knowledge across models.The reported comparison is for contextual-reliance interventions.Source: §4.2 · pp. 6–7

4.3. Robustness of JUICE

  • JUICE maintains consistently high performance across varied identification-set sizes, intervention counts, and scaling factors.The hyperparameter analysis uses Gemma and varies one parameter at a time.Source: §4.3 · p. 7, Figure 5 · p. 7
  • JUICE remains effective across paraphrased prompt templates, shifting reliance from context to parametric memory.The evaluation randomly selects among curated templates for each conflict type.Source: §4.3 · p. 7

4.4. JUNE vs. JUICE: Effect of Running Twice

  • Single-pass interventions are unstable and can degrade performance, whereas JUICE's dual-run design delivers consistent interventions.The comparison combines interventions that are individually effective.Source: §4.4 · p. 7, Figure 6 · p. 7

5. Theoretical Analysis

The theoretical analysis models knowledge conflict as distinct training tasks overlapping at inference. It shows how attention-head superposition can arise and why JUICE can outperform single-pass interventions in the simplified setting.

5. Theoretical Analysis

  • The analysis models conflicts as distinct training tasks that overlap at inference through attention weight matrices shaped by gradient descent.The authors focus on the attention module's weight matrices to formalize the empirical observations.Source: §5 · pp. 7–8

5.1. CP Superposition

  • A two-layer transformer can perfectly solve factual recall and induction, with its second layer forming a superposition after the first layer copies prior embeddings.The construction's second layer learns critical tokens and associated memory for both tasks.Source: §5.1 · p. 8
  • In a simplified setup, gradient descent from zero initialization can train the constructed superposition head; the authors conclude standard language-model objectives encourage superposition.The analysis assumes a linear attention model and strictly orthogonal embeddings.Source: §5.1 · pp. 8–9

5.2. Knowledge Conflict

  • At inference, a model capable of both tasks may output either answer depending on relative coefficients; fact frequency correlates with a stronger tendency toward parametric answers.The authors note that the coefficients should be sample-dependent in practice.Source: §5.2 · p. 9
  • When induction dominates, deleting the two heads with JUNE or PH3 does not yield the parametric answer, whereas JUICE does.This result is stated for the model and condition in Proposition 5.2.Source: §5.2 · p. 9

6. Conclusion

The paper presents knowledge conflicts as involving superposed contextual information and parametric memory, and proposes training-free JUICE to steer between them. It reports empirical and theoretical support while identifying conflict types outside its scope.

6. Conclusion

  • The study presents JUICE as a training-free test-time intervention that steers models toward parametric beliefs or contextual information and reports effective performance across datasets and conflict types.The authors connect these findings to model understanding and practical controllability.Source: §6 · p. 9

A. Related Works

  • Compared with Yu et al. (2023) and Jin et al. (2024b), the study covers broader conflict types and datasets and identifies superposition beyond an exclusive memory-head/context-head account.It also connects attention-based associative memorization with knowledge conflicts.Source: A. Related Works · pp. 15–16
  • The paper leaves intercontext conflicts and intra-memory conflicts outside its scope as directions for future research.These include contradictions within context and inconsistent responses to semantically identical but syntactically different queries.Source: A. Related Works · p. 15
  • The paper treats reliance on context or parametric knowledge as a choice for users and applications, rather than prescribing one universally correct behavior.It describes both perspectives as valuable.Source: A. Related Works · p. 15

C. Conflict Examples

The examples illustrate clean factual prompts and conflicting contextual claims at sentence and paragraph scale across capital-city, sports, and company-headquarters questions.

C. Conflict Examples

  • The LeBron James examples pair a clean sport question with tennis substituted in a sentence or reinforced through a coherent paragraph.The prompt asks which sport James plays.Source: C. Conflict Examples · p. 17
  • The Amazon examples contrast a clean headquarters-location prompt with Tokyo supplied as either a sentence-level substitution or a coherent paragraph claim.The question asks where Amazon's headquarters are located.Source: C. Conflict Examples · p. 17
  • The France examples contrast a clean capital-city prompt with a sentence-level Beijing substitution and a coherent paragraph elaborating the false association.The paragraph asks for France's capital after presenting Beijing as the answer.Source: C. Conflict Examples · p. 17
  • The authors expect high accuracy on clean inputs, moderate-to-low accuracy on substitution conflicts, and near-zero performance on coherent conflicts.The coherent-conflict format is attributed to Xie et al. (2024).Source: C. Conflict Examples · p. 17

D. Expanded Experiment Section

This appendix expands experimental reporting and documents the data, evaluation, and hyperparameter choices behind those experiments.

D. Expanded Experiment Section

  • The expanded experiment section reports intervention results with three models, while noting that many details and results are omitted for space.The appendix describes itself as complementary to the main paper’s experiment section.Source: D. Expanded Experiment Section · p. 17

D.1. Detailed Setups and Hyperparameters

  • Parametric examples come from verified Wikidata triplets and are turned into clean, substitution-conflict, and coherent-conflict prompts.Dataset sizes are around 200 examples for world capital, official language, and company founder, and around 500 for athlete sport, company headquarters, and book author.Source: §D.1 · pp. 17–18
  • NQ-Swap tests answers against reliable gold contexts, while MemoTrap covers four context-instruction tasks that deliberately deviate from common knowledge.MemoTrap includes Hate Speech Ending, History of Science QA, Proverb Ending, and Proverb Translation.Source: §D.1 · p. 18
  • The appendix details head-selection evaluations, including measuring first-correct-token probability and testing selected heads on held-out samples.The described setups use filtered World Capital data and evaluate effects under substitution and coherent conflicts.Source: §D.1 · p. 18
  • Hyperparameters include K=5 or 10 for JUICE by model size, validation-selected scaling factors, and task-specific prompts that instruct reliance on memory or context.The appendix also gives comparison-method settings, including PH3 head counts tuned from {1, 3, 5, 7, 9, 15}.Source: §D.1 · p. 18

D.2. Comprehensive Model Experiments

The appendix adds model results for parametric-memory intervention using the main paper’s experimental setup.

D.2. Comprehensive Model Experiments

  • Additional model experiments report that the main paper’s conclusions still hold.The appendix says these results follow Section 4’s setup.Source: §D.2 · p. 19
  • Table 5 reports full parametric-memory intervention results in accuracy (%), with the best result marked in bold.Source: Table 5 · p. 19

D.3. Details on Robustness Study

The appendix reports prompt and hyperparameter robustness, then details JUICE’s head-selection and dual-run intervention procedure.

D.3. Details on Robustness Study

  • JUICE maintains high performance across varied head-set sizes, numbers of intervened heads, and scaling-factor combinations in the Gemma World Capital study.The study varies one hyperparameter at a time and measures average accuracy across three conflict types for two plots.Source: §D.3 · p. 19
  • JUICE remains effective across randomly sampled paraphrased prompts and outperforms alternatives when shifting reliance from context to parametric memory.Table 7 reports robustness across different prompt templates using the same intervention procedure.Source: §D.3 · pp. 20–21, Table 7 · p. 21

E. Algorithm Details

  • Stage 1 selects top-ranked heads that score positively across clean, substitution, and coherent conflicts, using a small dataset and summed target-token probability changes.Multiple scaling factors are tested per head to ensure quasi-monotonicity.Source: E. Algorithm Details · p. 21
  • Stage 2 saves selected-head activations in a first run, then adds scaled saved outputs to those heads during a second run.Validation determines the scaling factors.Source: E. Algorithm Details · p. 21
  • JUNE is a single-run ablation that shares JUICE’s head identification stage but directly scales selected head outputs during inference.The authors use it to examine the significance of JUICE’s dual-run design.Source: E. Algorithm Details · p. 21

F. Limitations and Future Works

The appendix identifies efficiency and conflict-setting boundaries for JUICE and lays out assumptions and setups for the theoretical analysis.

F. Limitations and Future Works

  • JUICE’s activation caching may slightly slow inference and increase memory use; the study focuses on irrelevant contexts and leaves more complex conflict settings for future work.The authors note that parametric and contextual answers may not always be distinct in more abstract domains.Source: F. Limitations and Future Works · p. 21

G. Theoretical Analysis

G.1. Setups

  • The analysis uses a two-layer attention-only Transformer and considers factual recall mappings alongside induction sequence-completion tasks.Factual recall maps factual tokens to answer tokens; induction predicts the token following an earlier trigger occurrence.Source: §G.1 · pp. 23–24

G.2. Additional Notations

  • The notation distinguishes vocabulary size from dataset size, with separate sizes for factual and induction datasets.Source: §G.2 · p. 24

G.3. General Assumptions

  • The general assumption treats embeddings, unembeddings, and positional vectors as random unit vectors in a high-dimensional space, yielding near-orthogonality.Source: §G.3 · p. 24

G.3.1. ADDITIONAL ASSUMPTIONS IN TRAINING DYNAMICS

  • Additional training-dynamics assumptions require orthogonal vectors and no duplicate samples, including each noisy induction answer appearing once.The paper says these assumptions ease the analysis.Source: §G.3.1 · pp. 24–25

G.4. Proofs

The proofs establish a two-layer solver for factual recall and induction, derive a gradient-descent construction for a superposition layer, and formalize conflict outcomes and JUICE’s advantage in the stated setting.

G.4. Proofs

  • A two-layer transformer can solve both factual recall and induction perfectly with appropriate scaling.For factual recall, the correct-answer logit dominates; for induction, the construction produces the correct answer.Source: §G.4 · p. 25
  • Under the stated assumptions, gradient descent can learn the superposition-layer weight construction from zero initialization to perfect training-distribution accuracy in expectation.The setup ignores positional encodings and freezes Wlin as a random matrix; convergence is described with an appropriate learning rate.Source: §G.4 · pp. 25, 27
  • The first gradient step builds factual associations in WOV; subsequent updates can focus WKQ on critical tokens, with the desired induction term having the largest expected signal.The factual-recall construction also includes a benign spurious correlation in WKQ.Source: §G.4 · pp. 29–30
  • The constructed model outputs the factual recall answer when exp(C1)C3 < exp(C2)C4, and the induction answer otherwise.Source: §G.4 · p. 30
  • When induction dominates, JUICE reaches the parametric answer, whereas single-pass JUNE/PH3 deletion does not and can yield random guessing.The comparison is made for the constructed model when exp(C1)C3 >> exp(C2)C4.Source: §G.4 · p. 31
Loading 2503.10996v2…