Source-linked AI summary
ClashEval: Quantifying the tug-of-war between an LLM's internal prior and external evidence
Kevin Wu, Eric Wu, James Zou
TL;DR
LLMs may hallucinate, and retrieved documents may contain incorrect information, so models must arbitrate between external context and their own knowledge. ClashEval evaluates this challenge across questions, domains, models, and perturbed documents. The results show strong susceptibility to incorrect context, with lower adoption as the context deviates further from truth.
Problem
LLMs can give incorrect answers, and retrieval may supply erroneous or harmful information, raising the question of how models arbitrate when context conflicts with their prior knowledge.
Source: §1 · pp. 1–2Method
ClashEval pairs over 1,200 questions across six domains with relevant documents whose answers are perturbed, then benchmarks six LLMs.
Source: Our contributions · p. 2, §3.3 · p. 5Results
LLMs override their correct prior knowledge with incorrect retrieved information over 60% of the time, while greater contextual deviation negatively correlates with context preference.
Source: §5 · p. 9Takeaways & Limitations
The benchmark and analyses highlight the challenge of correctly discerning when to trust retrieved information and when to reject it.
Source: §5 · p. 9The dataset has an enriched rate of contextual errors, so its reported metrics are not meant to represent bias rates in the wild.
Source: §5 · p. 9
Abstract
from arXiv · showhide
Retrieval augmented generation (RAG) is frequently used to mitigate hallucinations and provide up-to-date knowledge for large language models (LLMs). However, given that document retrieval is an imprecise task and sometimes results in erroneous or even harmful content being presented in context, this raises the question of how LLMs handle retrieved information: If the provided content is incorrect, does the model know to ignore it, or does it recapitulate the error? Conversely, when the model's initial response is incorrect, does it always know to use the retrieved information to correct itself, or does it insist on its wrong prior response? To answer this, we curate a dataset of over 1200 questions across six domains (e.g., drug dosages, Olympic records, locations) along with content relevant to answering each question. We further apply precise perturbations to the answers in the content that range from subtle to blatant errors. We benchmark six top-performing LLMs, including GPT-4o, on this dataset and find that LLMs are susceptible to adopting incorrect retrieved content, overriding their own correct prior knowledge over 60% of the time. However, the more unrealistic the retrieved content is (i.e. more deviated from truth), the less likely the model is to adopt it. Also, the less confident a model is in its initial response (via measuring token probabilities), the more likely it is to adopt the information in the retrieved content. We exploit this finding and demonstrate simple methods for improving model accuracy where there is conflicting retrieved content. Our results highlight a difficult task and benchmark for LLMs -- namely, their ability to correctly discern when it is wrong in light of correct retrieved content and to reject cases when the provided content is incorrect.
1 Introduction
LLMs can hallucinate, while retrieved material can also be incorrect, creating a need to arbitrate between external context and internal knowledge. The introduction presents ClashEval as a benchmark for testing this arbitration in both directions and examining factors associated with context preference.
1 Introduction
- ClashEval tests whether models can choose between their priors and context when either source may be wrong, rather than succeeding by always ignoring retrieved information.It also examines how context preference relates to confidence in the prior response and the degree of contextual deviation from the reference answer.Source: §1 · p. 2
Our contributions
- The benchmark contains over 1,200 questions across six domains, each paired with relevant contextual material whose answer is perturbed from subtle to extreme errors.Source: Our contributions · pp. 2–3
- The authors benchmark six LLMs and report three metrics for evaluating their ability to arbitrate between prior knowledge and retrieved context.Source: Our contributions · p. 3
- The study analyzes context preference across perturbation levels and prior-response token probabilities, and proposes using token probabilities to improve performance on conflicting evidence.Source: Our contributions · p. 3
2 Related Works
Related work covers RAG evaluation and model confidence, but the authors identify a gap in systematic analysis of how log-probability confidence relates to preference for RAG-provided information.
2 Related Works
- Previous work explored automated RAG evaluation frameworks in various settings [Es et al., 2023a, Hoshi et al., 2023, Saad-Falcon et al., 2023a, Zhang et al., 2024], but systematic exploration of log-probability confidence and preference for RAG-provided information remains absent.This work considers LLMs available only via inference and uses only one document as context.Source: §2 · p. 3
3 Methods
The methods define prior-versus-context response metrics, assemble six-domain question-answer data, and systematically alter retrieved answers to study model responses.
3.1 Definitions and Metrics
- A QA instance pairs query q with context c; the prior response r(q) uses parametric knowledge alone, while r(q|c) is conditioned on context.Source: §3.1 · p. 3
- The metrics capture conditional correctness and whether responses follow the prior or context when only one is correct.Source: §3.1 · p. 3
3.2 Dataset
- The evaluation uses 1,294 questions across six domains and six models, with standard, strict, and loose prompts for contextual responses.Source: §3.1 · p. 3
- The datasets include UpToDate drug pages, Wikipedia Olympic records, and AP headlines, yielding filtered sets of 249, 191, and 238 question-answer pairs, respectively.Source: §3.2 · p. 4
- Dates, names, and cities questions came from 1,000 Wikipedia articles, were format-constrained and excerpt-checked, then sampled at 200 items per domain.Source: §3.2 · p. 5
3.3 Modifying the Retrieved Documents
- Retrieved answers receive ten numerical multipliers or year offsets, or three prompted categorical edits (slight, significant, comical), with excerpts regenerated to reflect each change.Source: §3.3 · p. 5
- Each modified fact replaces its original in retrieved text; GPT-4 answers from the question and context, and output-token log probabilities are collected.Source: §3.3 · p. 6
4 Results
Models vary in choosing between prior and context, and context preference shifts with document multiplicity, perturbation magnitude, and prior-answer probability. Token-probability correction improves accuracy while increasing prior bias.
4.1 Prior vs. Context Conflict Resolution
- 74.3% accuracy, 15.7% context bias, and 2.1% prior bias make Claude Opus the top model across all three reported metrics.The reported subset contains questions where either the prior or context is correct; the table also tracks responses using neither answer.Source: §4.1 · p. 6, Table 2 · p. 6
4.2 Multi-document Contextual Information
- Adding four embedding-similar documents lowered accuracy and increased neither-answer responses, while reducing context bias for GPT-4o and Claude Opus.The authors note that longer contexts can worsen model performance, while multiple documents can also protect against hallucination.Source: §4.2 · p. 6
4.3 Context Preference Rate vs. Degree of Context Modification
- 30% less adherence to incorrect context than GPT-4o at the same modification degree was observed for Claude Opus as context deviation increased.Stronger ClashEval models had lower intercepts and steeper negative slopes; the authors also report different truthfulness priors across domains.Source: §4.3 · p. 7, Figure 3 · p. 7
4.4 Context Preference Rate vs. Prior Token Probability
- Across all six QA datasets, prior-answer probability correlated negatively with context preference, with domain-specific slopes from -0.1 to -0.45.A -0.45 slope corresponds to an expected 4.5% decrease in contextual preference per 10% increase in prior-response probability.Source: §4.4 · p. 7, Figure 4 · p. 8
4.4.1 Initial Methods for Improving Prior vs. Context Conflict Resolution
- 14% higher overall accuracy and 20% lower context bias came with calibrated token probability correction, while prior bias rose from 2% to 8.5%.At the same 8.5% prior-bias rate, the method reached 75.4% accuracy versus 57.5% for random replacement.Source: §4.4.1 · p. 8, Table 3 · p. 9
5 Discussion
ClashEval reveals that models often favor retrieved context over correct prior knowledge, while their choices vary with the context’s truthfulness and their confidence. The authors frame resolving conflicts between priors and retrieved information as a challenge for robust, trustworthy language models.
5 Discussion
- The benchmark and evaluations provide insights into how LLMs arbitrate between internal knowledge and contextual information when they conflict.The authors propose a simple method for improving models under ClashEval and identify resolving this tension as a challenge for safe and trustworthy language models.Source: §5 · p. 9
- Models, including GPT-4o, override correct prior knowledge with incorrect retrieved information over 60% of the time, with preference decreasing as content becomes less truthful.Models differ in their truthfulness priors across domains; for a given deviation magnitude, Claude Opus follows incorrect context 30% less often than GPT-4o. Lower confidence in an initial response is associated with greater deference to context, with calibration varying by model.Source: §5 · p. 9
- The analyses cover a limited scope: fact-based questions without multi-step logic or document synthesis, and an enriched error rate that does not represent bias rates in the wild.The proposed token-probability method also applies only to models that provide probability outputs.Source: §5 · p. 9
A Appendix
The appendix reports model comparisons and visualizations of benchmark metrics, prior probabilities, and context preference. The provided captions identify comparison scopes but do not supply results for every table.
A Appendix
- Claude Opus performs best across all three metrics, with a context bias rate of 0.157.The metrics are context bias, prior bias, and accuracy.Source: Table 4 · p. 12
- Figure 5 plots each model’s performance across three metrics with 95% confidence intervals.It plots data from Table 4, using different colors for the metrics.Source: Figure 5 · p. 12
- The loose prompt has lower and steeper drops in context preference than the strict prompt as prior probability varies.The figure compares GPT-4 responses under prompts that encourage judgment versus literal adherence to retrieved context.Source: Figure 6 · p. 13
- Tables 5–7 cover accuracy and mean prior probability across datasets, GPT-4o and Claude Opus accuracy under different context conditions, and prior-versus-context choices across document counts.The captions identify these comparison topics but provide no outcome values.Source: Table 5 · p. 14, Table 6 · p. 15, Table 7 · p. 15