Same Question, Different Source, Different Answer: Auditing Source-Dependence in Medical Multi-Source RAG

Yubo Li; Rema Padman; Ramayya Krishnan

Source-linked AI summary

Same Question, Different Source, Different Answer: Auditing Source-Dependence in Medical Multi-Source RAG

Yubo Li, Rema Padman, Ramayya Krishnan

arXiv:2605.29084v1May 27, 2026cs.CLcs.AIcs.IR

TL;DR

Multi-source RAG can answer the same medical question differently depending on the retrieved institution, a problem that single-gold evaluation cannot diagnose. The paper introduces a benchmark, hierarchical retrieval and auditing strategy, and structured-output judge for measuring inter-source relationships, finding that better retrieval reveals more disagreement prevalence without increasing its intensity.

Problem
Single-gold-answer evaluation does not measure whether a multi-source RAG answer depends on the institutional document retrieved for the same question.
Method
The paper releases TransplantQA, HERO-QA, and a structured-output judge to ground patient questions in multiple handbooks and score inter-source relationships.
Results
13.6 pp lower handbook absence rates leave per-pair divergence essentially unchanged, indicating that prior estimates understated disagreement prevalence rather than intensity.
Takeaways & Limitations
The framework transfers conceptually to legal and educational RAG, where measuring source-dependence addresses the same multi-source evaluation blind spot.
Takeaways & Limitations
The empirical study is confined to English-language U.S. solid-organ transplant patient education, while transfer to legal and educational RAG remains conceptual.

Abstract

from arXiv · show

A retrieval-augmented generation (RAG) system deployed over a multi-author institutional corpus can give a different answer to the same question depending on which source it retrieves -- a failure mode the dominant single-gold-answer paradigm cannot diagnose. We argue that source-dependence is a missing axis of NLP evaluation, and that auditing it means shifting the unit of evaluation from answer correctness to the inter-source relationship. We make this concrete in transplant patient education, where institutional sources demonstrably disagree, releasing three artefacts: TransplantQA, a benchmark of real patient questions, each answered by grounding generation in multiple institutional handbooks as candidate sources; HERO-QA, a hierarchical retrieval strategy that grounds and audits each answer; and a structured-output judge that scores inter-source relationships on a validated 5-label taxonomy. At scale, better retrieval reveals far more disagreement than prior estimates suggested -- understating its prevalence, not its intensity. The framework is domain-agnostic and transfers to legal and educational RAG: measuring source-dependence is a responsibility for deployed multi-source NLP generally.

1 Introduction

The paper identifies source-dependence as a missing evaluation axis for multi-source RAG: the same patient question can receive institution-specific answers that single-gold benchmarks cannot diagnose. It operationalizes this shift with TransplantQA, HERO-QA, and structured inter-source judging, finding that prior estimates understated disagreement prevalence rather than intensity.

Institutional handbooks can recommend different waiting periods for the same post-transplant travel question while answers remain equally confident and fluent.The guidance may be institution-specific rather than universal.
Single-gold medical QA benchmarks cannot determine whether an answer depends on which institutional document the retriever returns.They score against curated gold answers rather than evaluating inter-source relationships.
The paper proposes evaluating the structured relationship between answers generated from different documents for the same question.This reframes the unit of analysis from single-answer correctness to source-dependence.
TransplantQA grounds 1,115 patient questions in 102 handbooks from 23 U.S. centers across five organ types for full-corpus and stratified comparison.The benchmark includes general and organ-specific question subsets.
13.6 pp lower handbook absence rates leave per-pair divergence essentially unchanged, showing that prior estimates understated disagreement prevalence, not intensity.The comparison uses a reference run versus an earlier 14B run with a lower-capacity retriever.
The framework's multi-source benchmark, inter-source taxonomy, and structured-output judge are designed to transfer to legal and educational RAG.The paper presents source-dependence measurement as relevant to deployed multi-source NLP beyond medicine.

2 Related Work

Prior medical QA benchmarks generally assume one curated gold answer, while related judge and inconsistency methods typically reduce comparisons to scalar, label, or binary signals. TransplantQA instead evaluates relationships between answers grounded in different institutional documents.

Medical QA benchmarks such as MedQA, MedMCQA, PubMedQA, and BioASQ evaluate single-best-answer prediction against curated gold answers.Patient-facing extensions retain the single-gold assumption.
TransplantQA makes inter-source answer relationships the unit of analysis and tests institutional heterogeneity at a scale not previously reported for medical QA.The paper states that, to its knowledge, no prior medical QA benchmark tests this heterogeneity at comparable scale.
Existing LLM-as-judge protocols typically return a scalar or label, whereas this judge co-emits divergence topics and clinical significance.The additional metadata supports taxonomy and severity analyses.
NLI, factuality decomposition, and hallucination evaluations target binary signals against a reference rather than relationships among alternative source-grounded answers.TransplantQA treats each answer as part of an inter-source comparison.
TransplantQA connects patient-facing educational material to institutional variation in clinical practice as an NLP-measurable phenomenon.The benchmark is presented as an instrument for measuring visible institutional variation.

3 The TransplantQA Benchmark

TransplantQA combines institutional transplant handbooks with real patient information-seeking questions so answers can be compared across plausible sources. Its corpus and question design support both broad and organ-stratified analyses while protecting user and document-identifying information.

TransplantQA pairs patient-education handbooks with questions drawn from real patient information-seeking behavior and evaluates relationships between answers grounded in different documents.This distinguishes the benchmark from single-gold medical QA.
102 handbooks come from 23 major U.S. solid-organ transplant centers and span five organ types.The centers include large academic and community-based transplant programs distributed across the United States.
The corpus contains 37 pre-transplant, 39 post-transplant, and 26 combined handbooks because centers organize education differently.Each phase-specific document is treated as a distinct unit.
1,115 questions were curated from public forums, social media, advocacy organizations, and institutional Q&A pages for cross-center comparison.Questions were harvested from genuine transplant information needs and filtered from more than 3,000 candidates.
Questions receive organ, 13-topic, and fine-grained sub-topic annotations, with multi-labeling for cross-cutting concerns.The annotation scheme includes 43 unique sub-topic tags.
The general subset contains 311 questions answered by every handbook, while 804 organ-specific questions are restricted to matching-organ handbooks.This design enables full-corpus and stratified inter-source analyses.
Released questions and handbook identifiers are anonymized to remove user-identifying content while retaining public center names for cross-institutional analysis.The benchmark uses anonymized handbook identifiers and rephrased questions.

4 Pipeline Architecture

The pipeline pairs real patient questions with multiple institutional handbooks, retrieves evidence for grounded answers, and compares answer pairs using a structured five-label judge. HERO-QA combines length-based routing, hierarchical retrieval, fusion, reranking, expansion, and fallback while preserving audit metadata.

Pipeline overview: The pipeline produces a structured matrix of pairwise inter-handbook relationships for every benchmark question.It takes the benchmark questions and handbook corpus as input and uses Qwen3-32B for generation and judging in the reference run.
Benchmark construction: TransplantQA contains 1,115 questions paired with 102 patient-education handbooks from 23 U.S. centers across five organ types.The questions include 311 general items answered by every handbook and 804 organ-specific items.
Audit and generation: HERO-QA retains retrieval metadata for audit and triggers full-document fallback when retrieved evidence is weak.Generation uses the top evidence to ground Qwen3-32B answers.
HERO-QA retrieval: HERO-QA routes short handbooks to full-document context and decomposes longer handbooks into document, section, and child-chunk levels.Short handbooks bypass retrieval, while longer documents preserve headings and pages during hierarchical decomposition.
HERO-QA retrieval: For long handbooks, four first-stage retrievers are fused, reranked, and expanded from child chunks to parent sections before generation.The system uses dense retrieval, child BM25, section-body navigation, title navigation, RRF fusion, cross-encoder reranking, and parent-section expansion.
Structured pairwise judgment: The judge classifies non-absent answer pairs as ABSENT, CONSISTENT, COMPLEMENTARY, DIVERGENT, or CONTRADICTORY along a coverage–agreement axis.Its structured JSON also records reasoning, divergence topics, and clinical significance where applicable.

5 Validating the Evaluation Instrument

The structured judge was validated against human annotators and against a label-only alternative. It showed strong overall categorical agreement, but its clinical-significance grades were only directionally reliable and its middle-category distinctions remained difficult.

Human–judge agreement: On 146 human-agreed pairs, the structured judge achieved 87.7% agreement, judge-vs-majority κ = 0.842, and weighted F1 = 0.876.Macro F1 was 0.841.
Human–judge agreement: Per-label F1 was 1.00 for ABSENT, 0.99 for CONTRADICTORY, 0.83 for CONSISTENT, 0.70 for COMPLEMENTARY, and 0.69 for DIVERGENT.Performance was strongest at the extremes and weaker for the intermediate relationship categories.
Failure modes: Of 18 judge errors, 14 (78%) clustered at the COMPLEMENTARY/DIVERGENT boundary.The judge called eight human-COMPLEMENTARY cases DIVERGENT and six CONSISTENT.
Structured versus label-only judging: The label-only plus post-hoc protocol agreed categorically at κ = 0.669 but downgraded 31 of the structured judge’s 40 DIVERGENT pairs to COMPLEMENTARY.It also returned high significance for all 44 evaluated DIVERGENT/CONTRADICTORY pairs, yielding κ = 0 against the structured judge’s mixed ratings, despite being approximately 5–6× faster per pair.

6 Benchmark Characterization

The benchmark reveals substantial inter-handbook heterogeneity, dominated by absence and complementary coverage rather than explicit contradiction. Stronger retrieval increases observed disagreement prevalence by exposing answers previously hidden by retrieval absence, while per-pair divergence remains comparatively stable.

Global label distribution: 78.9% of 5,730,465 pairwise comparisons pre-screen as ABSENT because at least one handbook returned NOT ADDRESSED.Among the remaining 1,211,220 pairs, LLM judging assigns inter-source relationship labels.
Global label distribution: 75.4% of LLM-judged pairs are COMPLEMENTARY, followed by DIVERGENT at 12.9%, CONSISTENT at 7.1%, and CONTRADICTORY below 0.1%.The dominant disagreements involve different aspects or substantively different recommendations, whereas explicit contradiction is rare.
Per-organ heterogeneity: Absence rates remain 60–78% across organs, while per-pair divergence rates cluster between 0.14 and 0.19.The proportion of questions with any divergence spans 30–56%, reflecting different concentrations of answers across questions.
Per-handbook heterogeneity: Per-handbook absence rates span 0.45 to 0.99, with a mean of 0.74 and systematic differences in editorial comprehensiveness.Some handbooks are broadly comprehensive, while others remain silent even within their own organ.
System-level comparison: A pipeline upgrade lowered absence by 13.6 percentage points on average while leaving per-pair divergence essentially unchanged.The proportion of questions showing any divergence rose by 15.9 percentage points because stronger retrieval exposed disagreements previously hidden by absence.
Downstream uses: Structured judge outputs enable a 991-theme taxonomy and severity-weighted analyses, with severity aggregation closely tracking unweighted disagreement frequency.Across question, topic, and handbook levels, the reported Spearman correlation exceeds 0.99.

7 Discussion

The framework is designed for multi-source RAG beyond medicine, although its empirical evidence is medical and its LLM judge retains important biases and costs.

Generalisation: The framework’s multi-source benchmark, inter-source taxonomy, and structured-output judge can transfer conceptually to legal and educational RAG.Legal and educational systems likewise retrieve across jurisdictional, state-stratified, or publisher-specific corpora where single-gold benchmarks cannot expose source-dependent answers.
Scope and limitations: Legal and educational transferability is conceptual, while the empirical instantiation is confined to U.S. solid-organ transplant patient education.The judge also inherits self-preference, length and citation artefacts, and cost limitations.

8 Conclusion

The paper releases a benchmark, retrieval system, and structured judge for auditing inter-source heterogeneity in medical RAG. Its central conclusion is that prior estimates understated disagreement prevalence because absence hid it, while structured judging is necessary to distinguish disagreement types and severity.

Contributions: TransplantQA, HERO-QA, and a structured-output LLM judge measure inter-source heterogeneity in deployed medical RAG.The released artefacts include 48,056 answers, 5.73M pairwise comparisons, and judge–majority agreement of κ = 0.842.
Conclusions: Prior estimates understated disagreement prevalence, not intensity, because retrieval absence concealed disagreements.The framework’s structured single-call judge preserves the distinction between DIVERGENT and COMPLEMENTARY relationships and supports severity gradation.

Limitations

The empirical study is limited to a U.S. English-language solid-organ transplant patient-education snapshot, while broader transferability remains conceptual. Evaluation also depends on an LLM judge and may conflate retrieval failures with genuine source disagreement.

The empirical instantiation covers U.S. solid-organ transplant patient education in English using a 2024–2025 snapshot; legal and educational transferability is conceptual.
The LLM judge’s 200-pair validation measures population-level agreement but cannot detect biases across institution, organ, or answer length.
Apparent inter-source divergence may be inflated by retrieval failures rather than true disagreement, although absence pre-screening partially mitigates this.
The benchmark contains 1,115 questions selected from more than 3,000 candidates collected across four families of public, patient-facing sources.
Questions receive one or more of 13 top-level topic labels and 43 fine-grained sub-topic tags, with multi-label category percentages allowed to exceed 100%.

C Data card (Datasheet for Datasets)

The data card describes TransplantQA as a benchmark for evaluating medical RAG under institutional heterogeneity, with documented composition, processing, intended uses, and judging procedures. It releases large-scale grounded answers and pairwise source comparisons while explicitly excluding direct clinical decision support and centre ranking.

The dataset was created to evaluate medical RAG systems over genuinely heterogeneous institutional sources and analyze that heterogeneity.
TransplantQA contains 1,115 patient-derived questions, 102 handbooks from 23 U.S. centres across five organ types, 48,056 grounded answers, and 5,730,465 pairwise comparisons.
Handbooks were downloaded from public institutional websites, extracted from PDFs, section-chunked with 512-token sub-chunking, and used to generate answers and judgments at temperature 0.
The benchmark is intended for multi-source medical RAG evaluation, institutional-heterogeneity measurement, and LLM-judge research, but not for ranking transplant centres or direct clinical decision support.
HERO-QA grounds each answer in evidence from a specific transplant centre and requires section-heading citations, while unanswered questions receive a standardized NOT ADDRESSED response.
The judge assigns exactly one of five inter-answer labels—ABSENT, CONSISTENT, COMPLEMENTARY, DIVERGENT, or CONTRADICTORY—and records reasoning, divergence topic, and clinical significance in JSON.
Across 1,211,220 LLM-judged pairs, JSON parsing succeeded on more than 99.5% of calls, with fallback extraction for unparseable outputs.

F Compute cost

The reference production run used GPU-based generation and comparison shards on PSC Bridges-2, with reported approximate cost and resumable execution. The released tables document compute cost and system-level retrieval-comparison deltas.

Production runs used NVIDIA H100 80 GB GPUs on PSC Bridges-2 via SLURM, aggregating 20 generation and 20 comparison shards.
The approximate reference-run compute cost was $1.3K–$1.8K at an indicative H100-80 GB cloud rate of $3–4/hour.
The pipeline is fully resumable, allowing stalled or pre-empted shards to restart without recomputing persisted per-question artefacts.
Smaller domains with 10–20 handbooks are runnable on a single H100 in under 24 hours.
Table 7 reports per-organ deltas for the 32B HERO-QA reference run versus the earlier 14B hybrid-retrieval run, showing lower absence and higher divergence prevalence without inflated per-pair divergence.