Source-linked AI summary
Entity-Based Knowledge Conflicts in Question Answering
Shayne Longpre, Kartik Perisetla, Anthony Chen, Nikhil Ramesh, Chris DuBois, Sameer Singh
TL;DR
The paper asks how QA models combine learned parametric knowledge with contextual passages when the two conflict. It formalizes entity-based conflicts through automated substitutions, evaluates popular QA models and influencing factors, and proposes mitigation training. Training on substituted instances reduces hallucination to negligible levels and improves out-of-distribution F1 by 4% to 7%.
Problem
The paper investigates how QA models use parametric and contextual knowledge when contextual information contradicts knowledge learned during training.
Method
An automated framework identifies named-entity answers, substitutes them with alternate entities, and creates conflict instances for evaluating and training QA models.
Results
4% to 7% higher F1 on out-of-distribution examples and negligible hallucination result from training with substituted instances.
Takeaways & Limitations
Knowledge conflicts reveal whether models read contextual information or rely on memorized answers, and the proposed mitigation encourages generalization to evolving knowledge.
Takeaways & Limitations
The automated framework’s answer-type identification and substitution validity may need finer-grained NER models and better-defined substitutions to improve fluency and correctness.
Abstract
from arXiv · showhide
Knowledge-dependent tasks typically use two sources of knowledge: parametric, learned at training time, and contextual, given as a passage at inference time. To understand how models use these sources together, we formalize the problem of knowledge conflicts, where the contextual information contradicts the learned information. Analyzing the behaviour of popular models, we measure their over-reliance on memorized information (the cause of hallucinations), and uncover important factors that exacerbate this behaviour. Lastly, we propose a simple method to mitigate over-reliance on parametric knowledge, which minimizes hallucination, and improves out-of-distribution generalization by 4%-7%. Our findings demonstrate the importance for practitioners to evaluate model tendency to hallucinate rather than read, and show that our mitigation strategy encourages generalization to evolving information (i.e., time-dependent queries). To encourage these practices, we have released our framework for generating knowledge conflicts.
1 Introduction
Knowledge-dependent QA combines parametric knowledge learned during training with contextual knowledge supplied at inference, creating ambiguity when the two conflict. The paper formalizes entity-based conflicts, evaluates model reliance on memorized answers, and introduces a substitution-based mitigation strategy that improves out-of-distribution generalization by 4% to 7%.
- 1 Introduction: Knowledge-dependent QA systems combine parametric knowledge learned in model weights with contextual knowledge retrieved as passages.These sources have an ambiguous division of labour during inference.
- 1 Introduction: Memorization can make models hallucinate by repeating training answers while ignoring relevant retrieved documents.This behaviour also limits interpretability and generalization to evolving, time-dependent knowledge.
- 1 Introduction: The paper studies knowledge conflicts in which contextual information contradicts knowledge learned during pre-training or fine-tuning, focusing on named-entity substitutions.The framework replaces an answer entity in a gold document with an alternate entity, changing the answer.
- 1 Introduction: The framework generates substitution instances for Natural Questions and NewsQA to evaluate how QA models rely on parametric knowledge.It supports entities from datasets or knowledge graphs and allows custom substitution policies.
- 1 Introduction: 4% to 7% higher F1 on out-of-distribution examples follows training with substituted instances, while hallucination falls to negligible levels.The mitigation strategy improves generalization by encouraging models to prioritize contextual knowledge.
2 Substitution Framework
The substitution framework creates entity-based knowledge conflicts by replacing named-entity answers in QA contexts, using configurable policies and entity sources. Human evaluation finds generally reasonable quality, while identifying gaps against curated resources and scope boundaries.
- Framework: The framework maps a QA instance to a new instance by replacing the original answer and every occurrence of it in the context.It transforms x = (q, a, c) into x′ = (q, a′, c′), with a′ as the gold answer.
- Identifying Named Entity Answers: It identifies named-entity answers, links them to Wikidata, records entity types and auxiliary information, and filters unsupported instances.The framework focuses on person, date, numeric, organization, and location entities, replacing all answer spans according to policy.
- Types of Substitutions: Substitution policies draw entities from Wikidata or dataset answers and can control semantic, type, alias, and popularity properties.Corpus Substitution samples same-type in-dataset entities; Popularity Substitution samples same-type Wikidata entities within specified popularity bounds; Alias Substitution uses Wikidata aliases; Type Swap uses nonsensical in-domain entities.
- Substitution Quality: Human grading measures whether substitutions are grammatical in context and whether the resulting context contains the answer to the question.Correctness compares the substituted answer with the span selected by annotators.
- Substitution Quality: Just above 80% fluency and correctness were retained for automated substitutions on Natural Questions, slightly below original examples.Compared with human-curated resources, the gaps were 0-14% for fluency and 4-11% for correctness.
- Substitution Quality: The framework scales beyond human annotation but is most effective for entity-based tasks with answers that can be classified by named-entity recognition.The authors identify finer-grained entity types and valid substitutions as directions for improving fluency and correctness.
3 Experimental Setup
The experiments evaluate retrieve-and-read QA systems under knowledge conflicts across training, development, and out-of-distribution settings. They compare generative and extractive readers and quantify reliance on memorized answers using the Memorization Ratio.
- Data and evaluation: Knowledge conflicts are evaluated on training, development, and out-of-distribution sets, with development examples separated by answer overlap.The OOD set excludes training examples, while development data is divided into Answer Overlap and No Answer Overlap subsets.
- Models and retrieval: The primary system uses dense passage retrieval during training and provides one retrieved document to the reader.Some experiments also use TF-IDF, and single-passage training helps decouple reader–retriever interactions.
- Inference setup: During inference, readers receive either a gold document or its substituted version, while the retriever is ignored.This isolates the reader’s response to contextual contradictions.
- Models and retrieval: The generative reader is a T5 model that receives a query concatenated with contextual text and decodes an answer.T5 is used as a consistent component across high-performing retrieval-based QA implementations.
- Models and retrieval: An extractive reader based on RoBERTa predicts an answer span taken directly from the context.It is trained with gold passages that always contain a gold span.
- Metrics: The Memorization Ratio measures how often a correctly answered original example yields the Original rather than the Substitute answer after substitution.Predictions are categorized as Original, Substitute, or Other using SQuAD-based Exact Match; MR estimates brittleness to changing information.
4 Experiments
Experiments show that QA models often favor memorized answers over conflicting contextual evidence, with this behavior shaped by model size, retrieval quality, answer properties, and data familiarity. Substitution-based training reduces memorization to negligible levels and improves out-of-domain generalization.
- 4.1 Results: The model predicted the contextual Substitute answer rarely more than 50% for NQ and significantly less for NewsQA, while reverting to the memorized Original answer up to 20% and 75%, respectively.Knowledge conflicts also triggered many Other predictions, often incorrect, demonstrating weak reliance on retrieved evidence.
- 4.1 Results: Knowledge conflicts increased uncertainty, especially on in-domain examples (74%), with uncertainty highest when the model predicted Other.Abstention could prevent some erroneous answers but would exchange incorrect answers for no answers without fixing contextual neglect.
- 4.1 Results: Other answers occurred at least 15% of the time under alias substitution and extractive QA, showing instability even when substitutions were semantically equivalent or the model had previously answered correctly.The extractive model’s Other rate reached 27% on the NewsQA out-of-distribution set.
- 4.2 Factors Impacting Model Behaviour: Memorization increased with model size, rising from < 15% to ≥50% on Train and Dev (AO) within two orders of magnitude, without diminishing returns.Even the 11B model showed increased memorization on Dev (NAO).
- 4.2 Factors Impacting Model Behaviour: Higher retrieval quality reduced memorization, while less relevant training passages made models more likely to predict the Original answer and ignore context.Training with gold passages minimized memorization by conditioning the reader to expect the answer in the passage, but this is costly and not standard practice.
- 4.2 Factors Impacting Model Behaviour: Numeric originals were especially memorized: replacing a numeric entity with a textual one led models to predict the Original answer at least 83% of the time.Textual-to-textual substitutions more often produced the Substitute answer than textual-to-numeric substitutions, suggesting answer plausibility affects behavior.
- 4.4 Mitigating Memorization: Augmenting training with corpus-substituted passages reduced memorization to negligible levels across knowledge-conflict datasets and improved out-of-domain generalization by 7% on NQ Dev NAO and 4% on NewsQA.The targeted substitutions teach retrieve-and-generate models to rely on context rather than memorize answers.
5 Related Work
Prior work connects parametric overreliance in QA to overstability, altered-document sensitivity, and robustness evaluation, while this paper studies entity substitutions as knowledge conflicts. Related approaches also target context use through contrastive training, test-time learning, and neuro-symbolic methods.
- Prior studies show that replacing retrieved documents with random ones can leave long-form QA performance similar, indicating overreliance on parametric knowledge.
- The paper’s qualitative analysis groups Other predictions by fine-grained phenomena, while mixed training with substitutions reduces memorization and improves out-of-distribution generalization.
- Fact-checking models struggle with subtly changed documents, while contrastive training improves attention to context.
- Overstability describes outputs remaining constant despite semantic input changes and relates to minimal pairs, contrast sets, and counterfactually created data.
- Entity-name swapping has been used for robustness in coreference and named-entity resolution, and this work applies similar frameworks to contextual-parametric conflicts.
6 Conclusion
The paper formalizes contextual-parametric knowledge conflicts in QA, evaluates model behavior under entity substitutions, and proposes mitigation through substitution-based training. Its findings connect reduced memorization with improved out-of-distribution generalization and relevance to evolving world knowledge.
- The paper examines conflicts between contextual and parametric knowledge in QA and contributes a framework for creating such conflicts.
- It rigorously evaluates model behavior under the substitution framework and proposes a method to mitigate memorization.
- The proposed mitigation improves out-of-distribution generalization, offering insights into model interpretability and adaptation to evolving world knowledge.