Do We Need an LLM to Know Why Sources Conflict? Decision Models for Knowledge-Conflict Typing in RAG

1 Introduction

• 10 of 11 decision models match or exceed the original paper’s LLM, while current Gemini 3.8 Flash is significantly more accurate than 8 of them.The evaluation uses CONFLICTS and the original paper’s protocol, with Gemini rerun on identical inputs.Source: §1 · p. 2
• Decomposing the taxonomy into yes/no questions lowers accuracy for all 11 models, and a repair fitted on one model helps none.The direct question remains the reliable format.Source: §1 · p. 2
• With a quarter of queries escalated to Gemini 3.8 Flash, 8 of 11 cascades are statistically tied with it at 25–37% of its cost.The cascade sends each model’s least certain queries to the LLM.Source: §1 · p. 2
• Dates play no measurable role, source-order sensitivity varies by model, and a planted false source is more apparent in probabilities than in top answers.These are pre-specified diagnostics of decision-model behaviour.Source: §1 · p. 2
• The types from 10 of 11 decision models yield answers that follow expected behaviour as often as answers using the LLM’s types.The comparison holds under both specified judges.Source: §1 · p. 2

2 Related Work

• Cattan et al. (2025) unified conflict causes into a taxonomy with expected behaviours and released CONFLICTS, the first benchmark annotated with conflict type.Earlier work on retrieved-context conflicts often treated one cause at a time.Source: §2 · p. 2
• Prior decision-model evaluations cover knowledge, classification, NLI, and judging, but no prior work compares these models on conflict typing across many retrieved sources.Huang et al. (2026) instead use a decision model to assess whether one report supports or contradicts another.Source: §2 · p. 3
• Cascades route easy inputs to a cheap model and hard inputs to an expensive one; effectiveness depends on confidence ranking the cheap model’s errors.Selective classification studies this confidence-ranking property, which the paper measures against random escalation.Source: §2 · p. 3

3 Task and Data

• The five labels are no conflict, complementary information, conflicting opinions or research outcomes, outdated information, and misinformation.They distinguish agreement, compatible answers, contested views, time-related disagreement, and sources that are wrong.Source: §3 · p. 3
• CONFLICTS contains 458 queries with an average of 9.2 Google Search results; a stratified 52-query development split leaves 406 test queries.The development split is used only to choose settings; reported results use the test split unless stated otherwise.Source: §3 · p. 3, Table 1 · p. 3
• The primary metric is five-class accuracy; the study also reports macro-F1, per-class F1, and ECE for systems returning probabilities.Macro-F1 accounts for imbalanced classes, and ECE measures calibration of the top-class probability.Source: §3 · p. 3

4 Conflict Typing with Decision Models

• A decision model receives the query and numbered search results as its state and returns probabilities for answers to typed questions.Several questions over the same state can be answered in one request.Source: §4 · p. 3
• The flat format asks one choice question with the five categories, copied with their definitions and examples from Cattan et al. (2025).The predicted type is the most probable option, whose probability is treated as model confidence.Source: §4 · p. 4
• The tree format asks five yes/no questions and combines their probabilities into a distribution over conflict categories.Its two versions differ in question order and in how they assign probability mass to categories.Source: §4 · p. 4
• A cascade sends the fraction f of lowest-confidence queries to an LLM and replaces the decision model’s answers with the LLM’s answers.The threshold uses no labels, and the reported operating points were fixed at f = 25% and f = 50%.Source: §4 · p. 4

5 Experimental Setup

• Every system receives each search result as its URL, title, snippet, available publication date, and 512-token segment, with results kept in dataset order and numbered.The results form the decision models’ state and fill the paper’s prompt for the LLMs.Source: §5 · p. 4
• The census runs all gateway-listed decision models with pinned identifiers where available; Classify-1 receives the same numbered results in one text field.Two entries are excluded: Laya cannot accept the task question within its input limit, and a floating Jev alias duplicates a pinned version; gateway-unserved open-weight models are out of scope.Source: §5 · p. 4
• The LLM baselines rerun Gemini 2.5 Flash with the paper’s prompt verbatim and add Gemini 3.8 Flash at low and default thinking levels.Gemini 2.5 Flash is tested with thinking off and, as a sensitivity check, on; malformed category formats are parsed leniently, and the gateway reports billed cost and successful-request wall-clock latency.Source: §5 · p. 4
• Prompts, questions, inputs, and the development/test split were frozen before test calls, and every later addition was specified in writing before running.Tree v2 was selected using the development split; every run is reported.Source: §5 · pp. 4–5
• Comparisons use identical queries, 95% bootstrap intervals, and two-sided exact McNemar tests; with 406 test queries, accuracy intervals are about ±4.8 points wide.Differences under about five points are generally reported as ties; multiple comparisons are not corrected, so conclusions rely on patterns across models rather than isolated tests.Source: §5 · p. 5

6.1 Decision models against the LLMs

• 50.0–67.0% accuracy: 10 of 11 decision models match or exceed Gemini 2.5 Flash’s 61.3%, while billed models cost 4–39× less and all are 2.6–6.0× faster.Gemini 3.8 Flash reaches 70.0% and is significantly more accurate than 8 decision models; the comparison uses the same 406 test queries.Source: §6.1 · p. 5, Table 2 · p. 6

6.2 Flat or tree?

• Yes/no decomposition lowers accuracy for all 11 models, while a repair fitted on one model does not improve any other model.Tree v2 is tied with flat for 7 models and significantly worse for 4; it is better calibrated than flat only on the model used to fit it.Source: §6.2 · p. 5

6.3 Confidence-gated cascades

• 8 of 11 cascades are statistically tied with Gemini 3.8 Flash when their least-confident quarter of queries is escalated, at 25–37% of its cost.Confidence ranks decision models’ errors (AUROC 0.64–0.76); escalating by confidence beats random escalation for 10 of 11 models, but the small test set cannot establish equivalence.Source: §6.3 · p. 6, Table 5 · p. 8

6.4 Diagnostics

• Removing dates or adding recency ranks significantly changes accuracy in 0 of 22 tests, while source-order shuffling changes predictions on 5.4–42.1% of queries.Order sensitivity varies by model; only 2 of 22 shuffle tests significantly change accuracy, both for GPT-6 Luna Decisions.Source: §6.4 · p. 6, Table 6 · p. 9
• For planted false sources, decision models’ Misinformation probabilities separate planted from clean copies with AUROC 0.66–0.86, but their hard-label detection is less reliable than Gemini 3.8 Flash’s.The probabilities reach AUROC of at least 0.75 for 10 of 11 models; the study plants one plausible wrong answer in a single source.Source: §6.4 · p. 7, Table 7 · p. 9
• Jev 1.14’s accuracy differences from Jev 1.13 are not significant for the flat question or tree v2, so the reported conclusions do not depend on which version is served.The flat-question difference is −2.2 points (p = 0.12), and tree v2 differs by −0.7 points (p = 0.66).Source: §6.4 · p. 8

6.5 End to end: do cheaper types keep the gain?

• 10 of 11 decision-model predictors are tied with Gemini 2.5 Flash in the end-to-end test, and all 11 cascades are tied with it under the primary judge.With the primary judge, Vanilla is 87.9%, Pipeline 91.6% (+3.7 points), and Oracle 95.3% (+7.4); the second judge finds no significant predicted-type gain over Vanilla.Source: §6.5 · pp. 8–9

What decision models offer as conflict classifiers

• Most hosted decision models classify queries in under half a second at no more than $0.41 per thousand, and confidence-gated cascades recover much of the current LLM gap at a quarter to a third of its cost.The cheaper predicted types cost no measurable answer quality end to end.Source: §7 · p. 9

Choosing among decision models

• Accuracy spans 50.0–67.0%, while calibration (ECE 0.031–0.188) and confidence-based error ranking vary independently, so model selection should assess all three on deployment data.One of the most accurate models has confidence no better than random for escalation.Source: §7 · p. 9

How to ask

• The direct choice question was reliable for every model, whereas yes/no decomposition lowered accuracy and a repair fitted on one model helped none.The authors recommend starting with the flat question for a new decision model.Source: §7 · p. 9

What the diagnostics change

• Removing dates did not matter: outdated sources were recognized from their content, while sensitivity to source order varied so much across models that it has to be measured for each one.Source: §7 · p. 9

Limitations

• CONFLICTS has only five natural Misinformation test queries, and the planted variant tests one substituted entity or value in one source; the 406-query test also cannot establish formal equivalence for cascades.The end-to-end evaluation uses one generator and LLM judges validated on 100 answers, which do not see search results; factual grounding was not measured.Source: §7 · p. 10

8 Conclusion

• Most tested decision models matched the original study’s LLM at lower cost and latency, while confidence-gated cascades recovered much of the gap to the current LLM for most models.Types from 10 of 11 models yielded answers following expected behavior as often as the LLM’s types; future work should test judges that see sources and probability-guided generation.Source: §8 · p. 10

Ethics and Broader Impact

• Cheap conflict classification could be misused to suppress minority viewpoints, whereas the expected behavior for conflicting opinions is to present all viewpoints.The planted-misinformation variant uses public data and factual substitutions on non-sensitive questions; CONFLICTS queries are safe by construction.Source: Ethics and Broader Impact · pp. 10–11

Decision models, flat

• The flat classifier asks one choice question over five conflict categories, using the original prompt’s instruction, definitions, and examples word for word.The appendix specifies the query and search-results input and requests an explanation plus a category number from 1 to 5.Source: A Prompts and Questions · p. 13

Decision models, tree

• The tree formulation asks five yes/no questions about whether answers differ, are compatible, concern opposing opinions, reflect outdated information, or include misinformation.The probabilities are combined in code into a distribution over categories.Source: A Prompts and Questions · pp. 13–14

Planted wrong answers

• The planted-wrong-answer prompt requests a plausible incorrect answer of the same type and format as the correct answer.One result containing the gold answer is rewritten for eligible queries; yes/no-answer queries are excluded.Source: A Prompts and Questions · p. 14

End to end

• For end-to-end generation, the study uses Cattan et al.’s answer prompt and inserts the conflict type’s definition and expected behavior before the query.A behavior judge checks whether the answer handles sources as expected; a separate answer-recall judge checks for the correct answer.Source: A Prompts and Questions · p. 14

• The development/test split, prompts, question texts, and combination rules were frozen with recorded file hashes before any test-set call.Source: B Protocol Record · p. 14
• Subsequent additions and amendments—including extra models, diagnostics, cascades, and the end-to-end test—were logged with reasons before results were computed.Prompts were checked on development data for errors only, and none changed after scoring; all runs are reported.Source: B Protocol Record · p. 15
• The primary judge matched annotators’ majority on 92%, while the second judge matched on 87%; code, splits, prompts, cached responses, and per-query outputs are released.One annotator label was excluded because the sheet showed another item’s answer.Source: B Protocol Record · p. 15