Source-linked AI summary
Rich Knowledge Sources Bring Complex Knowledge Conflicts: Recalibrating Models to Reflect Conflicting Evidence
Hung-Ting Chen, Michael J. Q. Zhang, Eunsol Choi
TL;DR
The paper investigates how retrieval-based QA models combine parametric knowledge with multiple retrieved passages when those sources conflict. It analyzes source reliance and confidence, then tests calibration methods that encourage abstention from single answers. Models generally rely on retrieved evidence, yet their confidence is not sensitive to conflicts, and calibrator-based abstention remains limited.
Problem
Prior work largely assumes consistent knowledge sources, leaving unclear how QA models combine parametric knowledge with multiple retrieved passages when evidence conflicts.
Method
The study perturbs retrieved evidence, analyzes two QA datasets and models, and evaluates a calibrator trained to abstain when conflicting evidence suggests multiple answers.
Results
Models primarily rely on retrieved passages, use a few relevant passages alongside parametric knowledge when answers conflict, and have confidence scores that are not sensitive to those conflicts.
Takeaways & Limitations
A separately trained calibrator can learn to abstain on conflicting evidence, but it improves moderately and does not generalize across different conflict sets.
Takeaways & Limitations
The findings may not hold for substantially worse retrieval or models with substantially richer parametric knowledge, and most conflicts studied are simulated within extractive QA.
Abstract
from arXiv · showhide
Question answering models can use rich knowledge sources -- up to one hundred retrieved passages and parametric knowledge in the large-scale language model (LM). Prior work assumes information in such knowledge sources is consistent with each other, paying little attention to how models blend information stored in their LM parameters with that from retrieved evidence documents. In this paper, we simulate knowledge conflicts (i.e., where parametric knowledge suggests one answer and different passages suggest different answers) and examine model behaviors. We find retrieval performance heavily impacts which sources models rely on, and current models mostly rely on non-parametric knowledge in their best-performing settings. We discover a troubling trend that contradictions among knowledge sources affect model confidence only marginally. To address this issue, we present a new calibration study, where models are discouraged from presenting any single answer when presented with multiple conflicting answer candidates in retrieved evidences.
1 Introduction
Retrieval-based QA combines parametric knowledge with retrieved evidence, but these sources can conflict. This paper studies which source models use and whether they can recognize when conflicting evidence warrants abstention.
- Motivation: Retrieval-based generation combines pretrained parametric knowledge with retrieved evidence documents to produce free-form answers.This approach aims to combine the broad factual knowledge of language models with the provenance of retrieval.
- Research question: The paper asks which knowledge source models ground their answers in when given numerous evidence passages and a pretrained, finetuned language model.The question matters for interpreting and debugging systems in adversarial or complex real-world settings where sources may conflict.
- Study scope: The study expands prior single-document analysis to settings with up to 100 passages and mixed evidence suggesting different answers.Conflicts can arise from outdated passages, adversarial edits, or differing author opinions.
- Main findings: Across two datasets and two models, retrieval-based generation models primarily extract answers from a few highly relevant documents rather than aggregating across many passages.The study then examines sensitivity to semantic perturbations such as adding negation.
- Calibration: Models are encouraged to abstain when multiple answer candidates are plausible, but teaching abstention is challenging and a separately trained calibrator helps only moderately.The paper identifies calibration as an unresolved weakness when rich evidence contains conflicting answers.
- Main findings: With high-recall retrieval, models rely almost exclusively on evidence passages, but when passages conflict they prefer answers matching their parametric knowledge.The paper also examines ambiguous and context-dependent queries and conflicts between passages.
2 Background
The paper studies open-retrieval QA systems that use a corpus of passages and retrieval-based readers. It focuses mainly on FiD, while comparing its multi-passage architecture with RAG and measuring model confidence under evidence perturbations.
- Task setting: Open-retrieval QA seeks an answer to a question using a large corpus of passages, evaluated here on NQ-Open and TriviaQA with Wikipedia as the corpus.The passages are retrieved to help answer the question.
- Models: The study examines Fusion-in-Decoder and Retrieval Augmented Generation, which share a dense retriever that selects a fixed number of passages similar to the query.Their reader architectures and learning objectives differ.
- FiD: FiD encodes each retrieved passage with the question, concatenates the resulting representations, and lets the decoder use cross-attention across all passages.This allows the reader to model interactions among evidence documents during decoding.
- Experimental setup: The analysis mainly uses a trained FiD-large checkpoint and retrains FiD with 1, 5, 20, or 50 retrieved passages for passage-count comparisons.RAG is included in selected main analyses to check whether FiD trends generalize.
- RAG: RAG conditions on each retrieved document separately and marginalizes answer probabilities across documents, which lets it jointly train reader and retriever but ignores document interactions.FiD instead models interactions while keeping reader and retriever training disjoint.
- Confidence analysis: Model confidence is measured using answer-generation probability or a separately trained calibrator score, comparing confidence before and after perturbing knowledge sources.The calibrator estimates the probability that the model predicts the answer correctly.
3 When do retrieval-based generation models rely on parametric knowledge?
Retrieval-based generation models usually rely on retrieved evidence rather than parametric knowledge, especially when retrieval has high answer recall. Increasing the number of passages further reduces memorization, while poor retrieval increases reliance on parametric knowledge.
- 3.4%/6.2% of examples in NQ/TriviaQA for FiD and 20.2% for RAG in NQ contain spans not found in the evidence passages.FiD remains extractive even when retrieved documents omit the answer string, copying evidence-derived text in 82.9%/69.6% of such NQ/TriviaQA examples.
- Over 70% of examples differ from the closed-book model’s prediction, indicating hybrid models use passages even without an exact answer-string match.This pattern holds even when the generated answer is abstractive.
- Providing 100 passages raises correct-span availability to nearly 90%, compared with up to 50% for one passage, helping the model remain extractive.The study identifies the multi-passage setting as the key difference from earlier single-document conflict experiments.
- The study evaluates knowledge reliance by substituting answer entities in all retrieved passages after retrieval, using same-type random entities on correctly answered examples.It also measures whether models preserve the original answer or follow the substituted evidence.
- With multiple passages and high answer recall, FiD memorization is uncommon—below 3.6%/8.5% for NQ/TriviaQA—and evidence passages dominate parametric knowledge.When answer recall decreases, reliance on parametric knowledge increases; with comparable recall, FiD memorizes less than RAG.
4 Simulating Mixed Bag of Evidence Passages
The study simulates conflicting evidence by perturbing retrieved passages and tests how answer selection, passage relevance, and confidence respond. Models often preserve the original answer when conflicting passages remain, while changing answers more when highly ranked passages are perturbed.
- Entity Substitution: The study perturbs subsets of retrieved passages by replacing answer entities, using random, top-retrieval, or top-attention sampling.The experiments analyze the best-performing FiD model trained with 100 passages.
- Entity Substitution: 98% of predictions avoid the original answer after all passages are perturbed, but random perturbation of 50% leaves the original answer favored 52% versus 25% on NQ and 59% versus 15% on TriviaQA.These results show that parametric knowledge remains influential when retrieval contains multiple potential answers.
- Entity Substitution: 30% of answers change when the top 25% of passages are perturbed, compared with 8% after randomly perturbing passages.Retrieval- or attention-ranked perturbations affect predictions more than random perturbations.
- Entity Substitution: With access to up to three passages containing the gold answer, the model still generates the original answer nearly 70% of the time.On average, only 16.7 NQ-Open and 21.5 TriviaQA passages among 100 contain the gold answer entity.
- Confidence Study: Model confidence does not clearly track the percentage of perturbed passages, although a separately trained calibrator consistently outperforms the inherent confidence score.Confidence should decrease with mixed evidence, but the observed relationship is unclear.
- Adversarial Semantic Perturbation: Semantic perturbations include negation, future tense, modal verbs, and text infilling, but the rules cover only 67–86% of answer-containing sentences.Results distinguish partial coverage from full coverage because unperturbed answer sentences can preserve the original answer.
- Adversarial Semantic Perturbation: Calibration scores remain mostly steady after semantic perturbation, with decreases observed in only 30–40% of examples.The model is particularly less sensitive to future-tense perturbations.
5 Re-Calibrating Models Given a Mixed Bag of Evidence
The paper frames calibration as detecting conflicting evidence and abstaining when multiple answers are plausible. Augmented training improves calibrator performance moderately, but generalization across conflict types remains limited.
- Calibration Setting: A calibration system should detect conflicting answers and abstain unless the evidence is conflict-free and the predicted answer matches the annotation.The study evaluates binary calibration accuracy on examples where the original FiD model answers correctly.
- Conflict Sets: The evaluation includes ambiguity-based conflicts, time-based conflicts from different corpus snapshots, and substituted-answer conflicts.AmbigQA supplies alternative interpretations, while SituatedQA supplies answers from different time periods.
- Results: Augmented training improves calibrator performance, but the improvement does not easily generalize across methods for collecting conflicting answers and evidence.More realistic AmbigQA and SituatedQA evidence generalizes better than simulated partial substitution despite being smaller.
- Results: Joint training on all conflict types improves performance over baseline calibrators only modestly compared with training separately on each conflict type.The results motivate future work on calibrator generalization across knowledge-conflict types.
6 Related Work
Related work covers memorization, robustness to misinformation and perturbations, and conflicting-context evaluation. This paper differs by analyzing model behavior and confidence under multiple conflicting passages.
- Memorization and Evaluation Overlap: Prior work found that entity-answer memorization occurs with one passage but can be substantially reduced when models are trained with multiple passages.Other studies also showed that overlap between training and evaluation data can inflate QA performance.
- Conflicting Contexts: Concurrent work studies robustness to misinformation by generating contradicting contexts, whereas this paper examines model behavior and confidence in those settings.The distinction is between constructing conflicting passages and analyzing how models respond to them.
- Perturbation-Based Robustness: Prior robustness studies use minimal perturbations, counterfactual data, and knowledge poisoning to test invariance or robustness under changed inputs.These approaches include heuristic and learned methods for generating perturbed examples.
7 Conclusion
The study finds that QA models usually rely on a few retrieved passages, while their confidence remains largely insensitive to conflicting knowledge. Calibration can encourage abstention, but generalization remains limited.
- Current models ground answers mostly in retrieved passages when paired with a high-recall retriever.
- Models rely on a few highly relevant passages and use parametric knowledge to break ties between conflicting answers.
- Models largely ignore semantic perturbations and output a potential answer entity found in retrieved passages.
- Confidence scores are not sensitive to knowledge conflicts, although a separately trained calibrator provides some improvements.
- A calibrator trained on conflicting evidence can learn to abstain, but it does not generalize to different conflict types.
- The authors present calibrator-based abstention as a partial solution and identify limited aggregation of conflicting information as a continuing weakness.
Limitations
The study’s conclusions are bounded by its model, dataset, and task choices, and most conflicts are simulated rather than drawn from real-world evidence.
- The study uses current state-of-the-art models and popular benchmark datasets, so its findings may not hold for substantially different models or datasets.
- The experiments focus on extractive QA with short entity-span answers, leaving multisentence and conditional question answering for future work.
- Most knowledge conflicts are simulated, so identifying and evaluating real-world knowledge conflicts remains future work.
A.2 Model and Training Details
The appendix describes the retrieval-reader architecture, calibrator inputs, training setup, perturbation procedures, and supplementary conflict experiments.
- FiD combines a retriever that ranks passages by question-passage similarity with a reader that processes retrieved evidence.
- The reader uses T5-large, concatenates each passage with the question, and truncates each input to 250 word pieces.
- Training and evaluation include multiple evidence-passage counts, entity substitution sets, and perturbations targeting verbal tense, negation, modality, and future tense.
- On SituatedQA, valid alternative entities bias the model more toward the substitute answer, suggesting parametric knowledge affects which answer appears more likely.
A.7 Full Results on No Answer Overlap Set
The appendix provides the full results for substituting different numbers of passages across NQ-Open, AmbigQA, and SituatedQA no-answer-overlap sets.
- Table 12 reports exact-match results for substituting different numbers of passages across the NQ-Open, AmbigQA, and SituatedQA sets.
A.8 Confidence Study Full Results
This section reports full confidence-study results and examines calibration changes after perturbation, including an out-of-domain memorization setting. The perturbation ratio distributions are bell-shaped and shift slightly toward negative values.
- Table 14 contains the full confidence-study results for adversarial semantic perturbation.
- Figure 3 plots the probability density of calibration-score ratios after versus before perturbation on a logarithmic scale.The area under the distribution sums to 1.
- The calibration-score ratio distributions are bell-shaped but shift slightly toward the negative x-axis.
- The out-of-domain memorization study replaces answer entities after retrieval and evaluates FiD readers trained on NQ-Open or TriviaQA against the other dataset.Results are reported in Tables 15 and 16.
B Further Analysis
Further analysis improves entity substitution using valid alternatives, examines memorization across domains and passage counts, and finds that fine-tuning accounts for most of the memorization effect. Models also use parametric plausibility when selecting among answer candidates.
- Memorization Analysis: Domain-adapted FiD models retain low memorization when given multiple retrieved passages in out-of-domain evaluations.Tables 15 and 16 report Exact Match and Memorization Ratio for cross-domain evaluations.
- Improving Substitute Entities: Prior substitution methods used same-coarse-type entities, but these substitutes could be unreasonable for the question.The paper gives “Heartbreak Hotel” as an example of an implausible substitute answer.
- Improving Substitute Entities: The study injects valid alternative answers from AmbigQA and SituatedQA to represent different question interpretations and temporal contexts.Both datasets augment NQ-Open with annotated alternatives.
- Improving Substitute Entities: Models output original answers more often when replaced with better alternatives, contrary to the expectation that valid substitutes would reduce hesitation.Table 17 reports these perturbation results using alternatives from AmbigQA and SituatedQA.
- Memorization Analysis: The analysis attributes the memorization effect almost exclusively to fine-tuning and finds lower memorization in the AmbigQA/SituatedQA NAO set after accounting for answer-overlap proportions.
- Memorization Analysis: The model uses parametric knowledge about which answer candidate is more reasonable, even when it behaves like a copying model.The conclusion concerns answer selection in the analyzed perturbation settings.