ECon: On the Detection and Resolution of Evidence Conflicts

1 Introduction

• Earlier studies focus on conflicts between model knowledge and retrieved evidence or on distracting evidence, rather than conflicts within a set of context evidence.The paper also identifies high-quality benchmark creation without labor-intensive human labeling as an outstanding challenge.Source: §1 · p. 1
• The method generates diverse, validated answer and factoid conflict pairs, with human annotations demonstrating high-quality labels.The evaluation covers detection on both conflict types and LLM answer resolution.Source: §1 · pp. 1–2
• NLI and LLMs show high precision on answer conflicts, but weaker models have low recall; FC models struggle with lexically similar conflicts that NLI and LLMs find easier.Stronger models, including GPT-4 and NLI-xxlarge, are more robust, especially on nuanced conflicts.Source: §1 · p. 2
• LLMs often favor one conflicting evidence piece without giving reasons, may rationalize conflicts through hallucination, and more often rely on internal knowledge when they hold prior beliefs.Models’ tendency to refrain from answering when given conflicting evidence increases with conflict intensity.Source: §1 · p. 2

2.1 Answer and factoid conflicts

• Answer conflicts occur when two evidence pieces support conflicting answers, while factoid conflicts model conflict between evidence represented as sets of factoids.Answer labels do not capture conflicts that leave the answer unchanged or the composition of the evidence.Source: §2.1 · p. 3

2.2 Conflict detection

• Detection classifies evidence pairs as conflicting or non-conflicting; evaluated detectors include NLI, factual-consistency models, and LLMs.NLI classification uses two threshold-agnostic formulas, and predictions are averaged across both evidence input orders.Source: §2.2 · p. 3
• Figure 1’s pipeline generates alternative answers for questions and supporting evidence for each answer, then uses a checker to validate quality.The paper’s evidence-pair generation includes answer conflicts.Source: Figure 1 · p. 3

2.3 Conflict resolution

• For resolution, models generate rationales and answers from a question and conflicting evidence, with question-only inputs also used to assess internal knowledge.The setup uses chain-of-thought prompting.Source: §2.3 · p. 3

3 Conflict detection

• The section evaluates detectors on generated answer conflicts and factoid conflicts, including answer-centric pollution attacks intended to simulate malicious modifications.The pollution setting modifies existing evidence to support a conflicting answer while retaining much of its detail.Source: §3 · p. 3, §3.1 · p. 3, §3.1.3 · p. 5

3.1 Answer conflicts detection

• For answer conflicts, evidence is generated for alternative answers and paired when it supports conflicting answers to the same question; NQ and CWQ provide the source data.Evidence is checked with NLI and LLM reasoning, and failed examples are filtered out.Source: §3.1 · p. 3, §3.1.1 · p. 4

3.1.1 Evaluation setup

• Human annotation of 200 sampled answer-conflict pairs found 92% pseudo-label accuracy and substantial annotator agreement (Fleiss’ κ = 71.2%).Question ambiguity was the major reason for incorrect generations.Source: §3.1.1 · p. 4

3.1.2 Main results and analysis

• NLI and LLM detectors generally have high precision but low recall; NLI performs best overall, though some NLI models weaken on longer contexts.Even weaker LLMs can exceed 90% precision; most LLMs and factual-consistency models are relatively robust to context length.Source: §3.1.2 · pp. 4–5, Table 2 · p. 5

3.1.3 Detection under pollution attacks

• Under answer-pollution attacks, NLI and LLM models tend to detect close polluted conflicts best, with similar performance on direct and far-polluted conflicts.Factual-consistency models do not show the same trend; AlignScore performs worst on close polluted conflicts.Source: Table 4 · p. 5, §3.1.3 · pp. 5–6

3.2.1 Evaluation setup

• The factoid-conflict pipeline uses StrategyQA’s human-verified factoids, semantically perturbs selected factoids, and generates evidence from original or perturbed sets.An NLI model checks that generated evidence entails the factoids used to produce it.Source: §3.2.1 · p. 6

3.2.2 Analysis on data

• The conflict approximation correlated positively with two annotators’ perceived intensity (Pearson ρ = 0.622, p-value 1.4 × 10−6); factoid conflict types showed higher diversity, with Negation and Degree considerable.Source: §3.2.2 · p. 6

3.2.3 Results and analysis

• Models detect higher-intensity factoid conflicts more reliably, while stronger models are more robust on low-intensity conflicts; corroboration matters little for most.AlignScore is an exception, with performance influenced by conflict intensity and likely by its sentence-wise score computation.Source: Table 5 · p. 7, §3.2.3 · p. 7

4.1 Evaluation setup

• The evaluation prompts LLMs to answer 120 conflicting-answer instances with zero-shot chain-of-thought, then prompts them using questions alone to probe parametric knowledge.The questions-only answers are treated as reflecting the models’ parametric knowledge.Source: §4.1 · p. 7

4.2 Analysis on conflict resolution behaviors

• Responses are categorized as refraining, resolving by content reliability, resolving by internal knowledge, resolving by chance, or integrating both answers.The authors describe refraining and integration as relatively objective, while chance-based resolution is least desired.Source: §4.2 · p. 8
• As conflict intensity increases, models are more likely to refrain, while weak conflicts are more often integrated with corroborating evidence.Figure 5 stratifies resolution behavior by human-labeled conflict intensity.Source: §4.2 · p. 8, Figure 5 · p. 8
• 24% of Sonnet and 38% of Haiku responses resolve by chance, choosing an answer without reasonable explanation.The authors describe this behavior as potentially harmful; stronger models tend toward more objective responses.Source: §4.2 · p. 8
• When models hold an internal belief for one of the answers, they more often resolve implicitly by chance or explicitly using internal knowledge, and less often choose objective responses.Table 6 compares behavior percentages with and without an internal belief over the instance.Source: §4.2 · p. 8, Table 6 · p. 8

5 Related work

• Earlier work addresses belief-versus-retrieved-knowledge conflicts and LLM robustness to irrelevant or distracting evidence.The cited areas include conflicts with parametric knowledge and distracting evidence.Source: §5 · p. 8
• This work focuses on conflicts among multiple context evidence, or inter-evidence conflicts, rather than conflicts between parametric beliefs and retrieved knowledge.Related work also includes studies of distracting evidence.Source: §5 · pp. 8–9
• Unlike factual-consistency and fact-checking work, this study evaluates conflicts; strong consistency evaluators and LLM checkers do not necessarily detect nuanced inter-evidence conflicts well.The paper also evaluates NLI models, factual-consistency models, and LLMs.Source: §5 · p. 9

6 Conclusion

• The paper introduces a method to generate high-quality evidence conflicts and evaluates NLI, factual-consistency, and LLM detection methods.Source: §6 · p. 9
• Advanced models such as GPT-4 perform robustly, while weaker models struggle especially with nuanced conflicts.Source: §6 · p. 9
• LLMs often resolve conflicts by favoring one piece of evidence without sufficient justification.Source: §6 · p. 9

Limitations

• The evaluation focuses on textual evidence, leaving AI-generated images, audio, tables, and topological graphs outside its scope.Source: Limitations · p. 9
• The work does not evaluate domain-specific adaptations for conflict detection or resolution.Source: Limitations · p. 9
• The authors identify domain-specific concerns, such as those in health conflict detection, as requiring attention.Source: Limitations · p. 9

• The study uses StrategyQA, NaturalQuestions, and ComplexWebQuestions, which come from public sources.Source: Ethics Statement · p. 9
• The authors cannot guarantee that these sources are free of harmful or toxic content.Source: Ethics Statement · p. 9

A.1.1 Datasets

• Answer conflicts are generated from the NaturalQuestions and ComplexWebQuestions validation sets.Source: §A.1.1 · p. 13
• Factoid conflicts are generated from the StrategyQA train set.Source: §A.1.1 · p. 13
• To mitigate variation in factoid counts, the dataset retains only question-factoid pairs with three or four factoids.Source: §A.1.1 · p. 13

A.1.2 Annotations / A.3.1 Models / A.3.3 LLMs prompting details

• Three team domain experts evaluated evidence-pair data quality, while the detection study averaged model performance across both evidence input orders.The evaluated detector families included NLI models, factual-consistency models, and zero-shot LLMs; llama3-70b-instruct generated alternative answers, modified factoids, and evidence pairs, and performed some quality checks.Source: §A.1.2 · p. 13, §A.3.1 · p. 14, §A.3.3 · p. 14

A.1.3 Quality check

• Among 100 excluded instances, incomplete evidence accounted for 45%, contradicting evidence 31%, and distracting evidence 24%.The checker uses entailment checks and an LLM-based consistency check for answer conflicts; for factoid conflicts, it checks entailment of seed factoids and contradiction of their opposites.Source: §A.1.3 · p. 13

A.2 Supplementary related work

• The paper frames its work as orthogonal to prior LLM task evaluations and proposes extending its database-free evidence-conflict paradigm to information extraction.It focuses on generating and validating evidence conflicts to simulate misinformation scenarios.Source: §A.2 · p. 14

A.3.6 Sensitivity to prompt wording

• Prompt wording affected LLM recall, whereas precision remained relatively stable, with deviations generally below 0.5%.The authors interpret this as prompt wording influencing how LLMs define conflicts more than their precision in detecting them.Source: §A.3.6 · p. 15, Table 8 · p. 15

A.3.9 Intensity of conflicts / corroboration

• The study controls factoid-conflict intensity by varying conflicting factoid pairs and corroboration intensity by varying corroborative pairs.Its examples contrast one versus three conflicting pairs and one versus three corroborative pairs.Source: §A.3.9 · pp. 15–16

A.3.13 Prompting LLMs to predict scores

• LLM conflict-detection precision typically exceeded 94–95% across decision thresholds, while recall was notably affected by the threshold.The study elicited conflict scores from 0 to 5 and normalized them to [0, 1] for analysis.Source: §A.3.13 · p. 16

A.3.11 Does the data generation model have an advantage in conflict detection? / A.3.12 Does the quality filter have an advantage in conflict detection?

• Neither the data generator nor the quality-filter model showed a significant detection advantage on data generated or filtered by itself.The reported comparison found no significant advantage for generator models in classification, and NLI-xlarge models generally did not outperform on data they filtered versus xxlarge-filtered data, or vice versa.Source: §A.3.11 · p. 16, §A.3.12 · p. 16, Table 9 · p. 17

A.3.14 Effect of combining conflict detectors

• Ensembling NLI/FC signals improved weaker models’ recall by up to +26%, with a slight precision decrease, but had minor, inconsistent negative effects on stronger models.The ensemble sometimes outperformed stronger models, but did not consistently benefit them.Source: Table 13 · p. 18, §A.3.14 · p. 18

• The appendix shows interfaces for evaluating conflict and factoid conflict resolution and provides prompts for language models on answer and factoid conflicts.Source: Figure 8 · p. 20, Figure 9 · p. 20, Table 14 · p. 21
• Supplementary answer-conflict detection results cover original versus reversed evidence order, evidence length, and performance on conflicting and non-conflicting labels.Source: Figure 10 · p. 21, Figure 11 · p. 21, Table 16 · p. 22
• Additional evaluations examine answer pollution, modified conflicting samples with similar textual similarity, varying conflict intensity, and breakdown accuracy by evidence-pair type.Source: Figure 12 · p. 24, Figure 13 · p. 24, Figure 14 · p. 25
• Other materials assess pairs generated from three or four factoids, the impact of models’ internal beliefs, corroboration intensity, and examples of answer and factoid conflicts.Source: Figure 15 · p. 27, Figure 16 · p. 27, Figure 17 · p. 27