PaperlayerView exact revision on arXiv

Source-linked AI summary

Large Language Models in Resolving Contextual Knowledge Conflicts

Xinye Yang, Zhenyang Liu, Ruisi Li, Yuanyuan Lei

arXiv:2609.03148v1cs.CL

TL;DR

The paper studies conflicts within contextual evidence, where prior datasets provide limited coverage of implicit conflicts, domains, and conflict types. It builds a multi-domain dataset and analyzes conflict processing, finding an earlier-evidence bias and reporting that activation steering improves reasoning accuracy and summary balance.

  • Problem

    Prior contextual-conflict datasets rely on template-based construction, emphasize explicit factual contradictions, and offer limited domain and conflict-type coverage with possible class imbalance, despite the importance of conflict handling for reliable multi-source applications.

    Source: §1 · p. 1

  • Method

    The paper builds a 5,781-sample dataset spanning six conflict types and two task families, analyzes conflict representations and evidence attribution, then steers activations without training or labels.

    Source: §1 · p. 2

  • Results

    Models often detect conflicts internally yet favor earlier evidence; steering improves reasoning accuracy and produces more balanced summaries.

    Source: §1 · p. 2, §6 · p. 9

  • Takeaways & Limitations

    Earlier-position bias is a key obstacle to comprehensive evidence integration, and activation steering is presented as a way to mitigate it.

    Source: §1 · p. 2, §6 · p. 9

    The steering method applies only to open-weight models and adds inference-time overhead; experiments cover GPT and LLaMA at 8B and 20B.

    Source: Limitations and Future Work · p. 9

Abstract

from arXiv · show

Most prior works focused on conflicts between an LLM's internal parametric knowledge and externally provided context. In contrast, we investigate how LLMs handle conflicts that arise within contextual knowledge itself. We introduce a taxonomy of six types of contextual conflicts (factual, inferential, temporal, granularity, perspective, and ambiguity) and contribute a comprehensive dataset ContextConflict for this setting. The dataset contains 5,781 samples, covers both reasoning and summarization tasks, and includes both explicit contradictions and implicit conflicts that require multi-step reasoning. Experiments on nine LLMs show that current models still fall short in resolving contextual knowledge conflicts. We further provide mechanistic interpretability insights into how LLMs process such conflicts, revealing their latent awareness of conflicts and the representational geometry underlying conflict processing. In addition, our analysis uncovers a consistent model bias towards earlier evidence, and this positional preference serves as a key obstacle to effective conflict resolution. Motivated by these findings, we further propose a simple training-free, label-free steering method that steers activations to encourage a more comprehensive incorporation of evidences for better conflict resolution. On our dataset, the method consistently improves accuracy on reasoning tasks and generates higher-quality, more balanced summaries for summarization tasks.

1 Introduction

The paper studies conflicts within contextual evidence, which can undermine downstream reliability when unresolved. It introduces a multi-domain dataset, analyzes how models process conflicts, and proposes activation steering to improve resolution.

1 Introduction

  • Prior contextual-conflict datasets rely on template-based construction, emphasize explicit factual contradictions, and have limited domain coverage and possible class imbalance.These limitations leave implicit conflicts requiring multi-step reasoning underrepresented.Source: §1 · p. 1
  • CONTEXTCONFLICT contains 5,781 samples across six conflict types, covering reasoning and summarization tasks with both explicit and implicit cases.Explicit cases permit direct comparison; implicit cases require multi-step cross-evidence reasoning.Source: §1 · p. 2
  • The paper's mechanistic analysis examines conflict awareness, representational geometry, and evidence attribution at both representation and output levels.It uses concept activation vectors, spectral energy decomposition, and Shapley-based attribution scores.Source: §1 · p. 2
  • Models show internal awareness of conflicts, with conflict types appearing at different layer depths and in distinct latent-space geometries.The analysis also finds a consistent preference for earlier evidence, which is identified as an obstacle to comprehensive integration.Source: §1 · p. 2
  • Steering improves accuracy on reasoning tasks and produces more balanced summaries on summarization tasks.The method is training-free and label-free, and nudges activations to encourage more comprehensive evidence integration.Source: §1 · p. 2

2 CONTEXTCONFLICT: Contextual Knowledge Conflict Dataset

CONTEXTCONFLICT organizes contextual conflicts into six types and distinguishes reasoning from summarization tasks, with explicit and implicit cases in each. Its construction combines original and semi-synthetic data, with labeling and quality checks tailored to the risk of each construction step.

2.1 Overview

  • The six conflict types probe distinct capabilities and divide into reasoning tasks (inferential, misinformation, temporal) and summarization tasks (granularity, perspective, ambiguity).Explicit cases require at most one inferential step; implicit cases require at least two, and Table 1 reports per-type implicit proportions.Source: §2.1 · p. 2, Table 1 · p. 3
  • Reasoning instances have verifiable answers evaluated by accuracy, while summarization instances call for balanced integration and use a Shapley-based Balance score.Summaries should contrast perspectives, cover each entity in ambiguity cases, or reconcile compatible levels of granularity.Source: §2.1 · p. 3

2.2 Dataset Composition

  • The 5,781 instances comprise 1,734 semi-synthetic samples and 4,047 preserved-original samples drawn from ten base datasets.Semi-synthetic samples use GPT-5 only to generate conflicting evidence or timestamps; gold labels come from source datasets or human re-annotation.Source: §2.2 · p. 3
  • Quality assurance scales with label-corruption risk: some subsets receive full double re-annotation, others 30% audits, and preserved-original data 10% spot checks.For sources used without synthetic generation, 10% is also randomly sampled for quality review and conflict-category validation.Source: §2.2 · pp. 3–4
  • NEJM-MedQA cases distinguish compatible diagnoses at different specificity levels (granularity) from wrong conclusions caused by flawed reasoning (inferential).The original question and reasoning chains are preserved as evidence.Source: §2.2 · p. 3
  • FOLIO and ENTAILMENTBANK receive one or two extra statements and double re-annotation; SciFact is augmented with style-matched conflicting evidence and 30% audits.Disagreements in re-annotation are resolved through discussion.Source: §2.2 · pp. 3–4
  • ConflictBank-temporal instances gain explicit timestamps without changes to the original evidence, and each is checked by two annotators with discussion-based adjudication.Source: §2.2 · p. 4

3 Evaluation

The evaluation defines task-specific measures, validates Evidence Balance through human agreement, causal intervention, and scorer robustness, and finds persistent reasoning difficulty, earlier-evidence bias, and a distinction between evidence grounding and balance.

3.1 Evaluation Metrics

  • Reasoning uses gold-label Accuracy; summarization uses Shapley-based Evidence Balance, while Faithfulness measures evidence support rather than factual correctness.Evidence Balance measures the inequality of evidence contributions; lower scores indicate more balanced integration.Source: §3.1 · p. 4

3.2 Validating the Balance Metric

  • On 108 samples, Balance agrees with human annotators at κ = 0.50–0.52 (inter-human κ = 0.52); in 72 samples, removing highest-attributed evidence causes larger likelihood drops than removing lowest.The scorer-size check found absolute Balance values shifted, but model rankings remained essentially unchanged across alternative scorers.Source: §3.2 · pp. 4–5, Table 2 · p. 5, Table 3 · p. 5

3.3 Evaluation Results

  • 68.0%, 61.5%, and 44.3% are the highest reported accuracies for Temporal, Misinformation, and Inferential tasks, respectively; Inferential remains below 50% even for the strongest model.Inferential accuracy scales from 24.8% on llama-3.1-8b-instruct to 44.3% on claude-4.5-sonnet.Source: §3.3 · p. 5
  • 69.0% of evidence contribution comes from the first evidence in Ambiguity for Llama-3.1-8B-Instruct, and Balance is not monotonic with model capacity.The authors conclude that scale alone cannot eliminate position bias; even top-performing models remain far from perfectly balanced integration.Source: §3.3 · p. 5
  • 92.5% Faithfulness on Perspective coexists with Balance of 34.5 for GPT-OSS-20B, indicating that grounding and balanced evidence integration are not tightly coupled.The authors interpret this pattern as evidence that the bias operates primarily at the evidence-selection stage.Source: §3.3 · p. 5

4 Analysis

The analysis uses conflict awareness, representational geometry, and evidence-position bias to examine how models encode and resolve contextual conflicts. It finds layer- and conflict-type-dependent representations alongside a consistent preference for earlier evidence.

4 Analysis

  • The analysis combines hidden-state separability, spectral energy, and Shapley attribution to trace conflict detection, representation structure, and evidence-order effects.Mechanistic analyses focus on Llama-3.1-8B-Instruct, with key trends replicated on GPT-OSS-20B.Source: §4 · p. 5

4.1 Conflict Awareness via Concept Vectors

  • Conflict awareness emerges differently by type: most types reach AUC > 0.85 in mid-to-late layers, while temporal and ambiguity conflicts saturate earliest.Inferential and misinformation conflicts emerge later; granularity follows a U-shape, while perspective peaks in middle layers before declining.Source: §4.1 · p. 6

4.2 Spectral Energy Analysis Reveals Conflict Dimensionality

  • Conflict types show distinct geometric patterns: temporal representations compress, granularity and ambiguity disperse, inferential and misinformation hover near zero, and perspective varies by layer.The patterns are qualitatively consistent across models and indicate that conflict processing is not a one-size-fits-all geometric process.Source: §4.2 · p. 7

4.3 Evidence Position Bias in Internal Representations

  • Across seven tested models, Evidence 1 disproportionately drives outputs, and the combined representation shifts toward the first evidence across layers.Shapley attribution reports 69.0% of contributions for Evidence 1 in Ambiguity, 38.4% in Granularity, and 35.0% in Perspective; these exceed uniform baselines.Source: §4.3 · p. 8

5 Activation Steering for Effective Conflict Resolution

The paper proposes training-free, label-free activation steering that shifts representations toward a neutral integration center. The method wins five of six model-by-reasoning comparisons and remains robust on summarization balance, with one reported temporal-task exception.

5 Activation Steering for Effective Conflict Resolution

  • The training-free, label-free method steers residual-stream activations toward a neutral center to counter the first-evidence bias.It collects single- and combined-evidence activations, computes a direction, then injects it during generation.Source: §5 · p. 8
  • All-Tokens injection produces significant gains for both models, including a +19.2-point Temporal accuracy gain for Llama-3.1-8B-Instruct.The two schedules evaluated are First-Generated and All-Tokens.Source: §5 · pp. 8–9
  • Our direction wins five of six model×reasoning comparisons and remains robust on summarization balance.The reported exception is GPT-OSS-20B Temporal, where CAS scores 65.5 accuracy versus 64.1 for u(l).Source: §5 · p. 9
  • The GPT-OSS-20B Temporal comparison favors CAS over u(l), with accuracy of 65.5 versus 64.1.The authors attribute this exception to temporal reasoning's use of every timestamped piece uniformly, which favors blanket context amplification.Source: §5 · p. 9
  • The All-Tokens schedule yields the largest accuracy gains, while reasoning faithfulness decreases slightly as reasoning shifts toward multiple evidence pieces.The authors characterize the shift as exceeding verbatim grounding.Source: §5 · p. 9

6 Conclusion

The paper releases a 5,781-sample dataset spanning six conflict types and reasoning and summarization tasks. Its analysis identifies earlier-evidence bias and proposes activation steering to improve evidence integration.

6 Conclusion

  • The paper releases a 5,781-sample multi-domain dataset spanning six conflict types across reasoning and summarization tasks.Source: §6 · p. 9
  • Training-free, label-free activation steering mitigates earlier-evidence bias and improves evidence integration under conflict.The analysis finds that models often detect conflicts internally yet still allocate disproportionate trust to earlier evidence.Source: §6 · p. 9
  • Mechanistic analysis shows that models often detect conflicts internally but still give earlier evidence disproportionate trust.Source: §6 · p. 9

Limitations and Future Work

The paper identifies access, scale, data realism, and attribution assumptions as limitations. It leaves larger-scale steering experiments and evaluation on noisy, heterogeneous real-world retrieval for future work.

Limitations and Future Work

  • Residual-stream access limits the steering method to open-weight models and adds inference-time overhead.Source: Limitations and Future Work · p. 9
  • Compute constraints restrict steering experiments to GPT and LLaMA models at the 8B and 20B scale, leaving 70B or 120B scaling for future work.Source: Limitations and Future Work · p. 9
  • Synthetic samples support controlled manipulation, but evaluation on noisy real-world retrieval with heterogeneous evidence remains open.Source: Limitations and Future Work · p. 9
  • The Shapley-based attribution assumes equal expected contributions from evidence pieces, which may not hold when real-world evidence quality varies.The authors identify weighted Shapley attribution with source-reliability priors, alongside Faithfulness, as a promising extension.Source: Limitations and Future Work · p. 9

Ethical considerations

The paper describes safeguards for its synthetic misinformation data and reviews related work on contextual conflicts, attribution, interpretability, and positional bias. It distinguishes its focus on internal conflict processing and evidence integration from prior approaches.

Ethical considerations

  • Misinformation samples are research-only, labeled for factual correctness, and accompanied by planned usage guidelines to mitigate misuse.The dataset excludes personally identifying information and offensive content; its base datasets are publicly available under open-access licenses permitting research reuse.Source: Ethical considerations · p. 9

A Extended Related Work

  • Prior contextual-conflict datasets are described as limited in real-world complexity, implicit multi-step conflicts, domain diversity, and class balance.The paper positions its six-category dataset as jointly covering two task paradigms across multiple domains.Source: A Extended Related Work · p. 12
  • Unlike context-aware decoding, which treats all evidence as one block, the proposed steering targets asymmetry among individual evidence pieces without external models or training.The authors describe the methods as addressing orthogonal problems and not directly comparable as alternatives.Source: A Extended Related Work · p. 12
  • The paper adapts Shapley attribution to conflict fairness analysis and introduces Balance, a normalized-Gini metric for evidence-integration bias.Source: A Extended Related Work · p. 12
  • The paper addresses a gap in dataset-based mechanistic analysis of conflict awareness, representational geometry, feature organization, and evidence-integration bias.Source: A Extended Related Work · p. 12
  • In conflict settings, positional bias concerns selective integration among relevant but competing evidence, rather than retrieval of relevant items from noisy context.The paper reports that its training-free steering improves reasoning and maintains strong summarization evidence-integration balance on tested models.Source: A Extended Related Work · p. 13

B Dataset Details

The appendix describes prompt-based dataset construction, operational criteria for distinguishing overlapping conflict types, and representative examples across the taxonomy.

B.1 Construction Process

  • The construction appendix lists ten base datasets and presents GPT-5 prompts for generating misinformation, inferential, temporal, and factual conflicts.It also describes a manual re-annotation interface for FOLIO instances.Source: §B.1 · p. 13

B.2 Decision Rules for Borderline Conflict-Type Overlaps

  • Temporal conflicts contain evidence true at different time points, whereas misinformation conflicts include factually incorrect evidence labeled for accuracy.Source: §B.2 · p. 13
  • Ambiguity conflicts integrate distinct real entities sharing a name without cross-evidence reasoning; inferential conflicts require combining evidence to derive one correct conclusion.Source: §B.2 · p. 13
  • Perspective conflicts involve incompatible stances on the same question, while granularity conflicts give compatible answers at different specificity levels.Source: §B.2 · p. 13

B.3 Data Samples

  • The appendix provides examples of all six conflict types, including an inferential case with an uncertain conclusion and a temporal case requiring reasoning across events.Other examples cover accuracy-labeled scientific claims, multiple entities sharing a name, varying specificity, and diverse policy viewpoints.Source: §B.3 · p. 13

C.1 Evaluation via Different Scorer Models

The Shapley-based Balance metric gives consistent model rankings across two scorer models and aligns broadly with human judgments. The appendix also details how conflict-consistent probe pairs are constructed.

C.1 Evaluation via Different Scorer Models

  • Model rankings by Balance remain nearly identical under Llama-3.1-8B and Gemma-2B scoring, supporting robustness to scorer choice.Absolute Balance scores differ between scorers; Gemini-2.5-Pro ranks best for Ambiguity, while Llama-3.1-8B-Instruct has the highest scores across conflict types.Source: §C.1 · p. 14

C.2 Human Annotation and Metric Validation

  • Human annotators show moderate agreement (κ = 0.5199), while the automatic metric agrees with them at κ = 0.5208 and 0.5014.Two independent annotators rated evidence contributions for a subset of summarization samples; 108 samples were annotated.Source: §C.2 · p. 14

D.1 Conflict-Consistent Pair Construction and Cross-Validation Statistics

  • Counterpart evidence is filtered by conflict type: misinformation keeps factually correct evidence, while temporal pairs retain evidence consistent with one timestamp.For inferential conflicts, the original entailment chain is retained and the GPT-5-generated conflicting branch removed; granularity pairs keep a shared specificity level, and perspective pairs keep a shared stance.Source: §D.1 · p. 15
  • Each conflict sample is paired with one consistent counterpart, and linear logistic-regression probes classify residual-stream activations using 5-fold stratified cross-validation.Pairs preserve the question and surrounding context while replacing or filtering evidence; the probe uses C=1.0 and a maximum of 1000 iterations.Source: §D.1 · pp. 15–16

D.2 Concept Vector Analysis on Additional Models and Implicit Conflicts

The appendix finds strong conflict awareness across tested models but weaker detection of conflicts requiring multi-step cross-evidence inference. Earlier evidence retains an advantage across models and after input shuffling, while steering performance is stable across tested strengths.

D.2 Concept Vector Analysis on Additional Models and Implicit Conflicts

  • GPT-OSS-20B shows strong conflict awareness across types, with temporal and ambiguity conflicts saturating quickly and other types emerging gradually.The observed awareness trends resemble those in Llama models despite GPT-OSS-20B having fewer layers.Source: §D.2 · p. 16, Figure 19 · p. 22
  • AllSides perspective conflicts fluctuate near chance, unlike Perspectrum’s stable high AUC; FOLIO inferential conflicts also show lower AUC than EntailmentBank.The results suggest reduced representation sensitivity when conflicts require multi-step cross-document synthesis, whether through logical inference or selective framing.Source: §D.2 · pp. 18–19, Figure 21 · p. 23, Figure 20 · p. 23

D.3 Spectral Energy Analysis Implementation Notes

  • Spectral analysis uses the top 10 singular values of centered activations, computed with PyTorch’s svd_lowrank and ε_ER = 10^-12.Activations are taken at the last non-padding token of each sample.Source: §D.3 · pp. 19–20

D.4 Spectral Energy Analysis in Other Models

  • Figure 22 shows type-specific delta energy-ratio geometric patterns in openai/gpt-oss-20b that remain consistent with Llama-3.1-8B, suggesting similar geometric trends across the tested models.Source: §D.4 · p. 20

D.5 Evidence Position Bias Across Tested Models

  • Earlier evidence receives larger contributions across seven tested models, including small and large, proprietary and open-source systems.The experiments found that model scale and training approach did not remove this bias.Source: §D.5 · p. 20, Figure 23 · p. 24

D.6 Bias Measurement Implementation Notes

  • The layer-wise directional projection tracks alignment between combined representations and evidence directions; larger gaps indicate stronger positional asymmetry.The projection is an alignment strength, not probability mass; its values can be negative and do not sum to one.Source: §D.6 · p. 21

D.7 Representation-Level Position Bias in GPT-OSS-20B

  • GPT-OSS-20B’s combined-evidence representations align toward earlier evidence from low to high layers, mirroring the reported Llama-3.1-8B-Instruct pattern.This cross-model consistency is reported for the tested models and dataset.Source: §D.7 · pp. 21–22

D.8 Position Shuffling Experiment

  • Evidence 1 continues to dominate after input-order shuffling on both tested models, so shuffling does not fully resolve position bias.The authors interpret this as evidence that the bias is mechanistically stable in their experiments.Source: §D.8 · p. 23, Table 9 · p. 23, Table 10 · p. 23

E Mitigation Method Details

The appendix uses distinct task-appropriate prompts for summarization and reasoning. Summarization has three prompt variants, while reasoning uses prompts for single- and combined-evidence settings.

E.1 Prompts

  • Each task prompt consists of a system message and a user message and is designed to be concise and task-appropriate.Summarization and reasoning use different prompts.Source: §E.1 · p. 24

E.1.1 Summarization Tasks

E.1.2 Reasoning Tasks

Loading 2609.03148v1…