Source-linked AI summary
When Documents Disagree: Measuring Institutional Variation in Transplant Guidance with Retrieval-Augmented Language Models
Yubo Li, Ramayya Krishnan, Rema Padman
TL;DR
Transplant patient education materials vary across U.S. centers, but no systematic method previously quantified this heterogeneity at scale. The paper grounds common patient questions in different handbooks with retrieval-augmented language models and compares the answers, finding substantial divergence alongside widespread coverage gaps.
Problem
No systematic method exists to quantify heterogeneity in solid-organ transplant patient education materials across U.S. centers at scale.
Method
The framework grounds the same patient questions in handbooks from different U.S. transplant centers and systematically compares the resulting responses.
Results
20.8% of non-absent pairwise comparisons were DIVERGENT, while 96.2% of all handbook pairs were ABSENT; reproductive health had a 95.1% absence rate.
Takeaways & Limitations
Topic-level absence and center-level divergence profiles provide actionable benchmarks for improving and harmonizing transplant patient education.
Takeaways & Limitations
The corpus is limited to English-language handbooks from 23 U.S. transplant centers, and the LLM-based pairwise judge may introduce systematic classification biases.
Abstract
from arXiv · showhide
Patient education materials for solid-organ transplantation vary substantially across U.S. centers, yet no systematic method exists to quantify this heterogeneity at scale. We introduce a framework that grounds the same patient questions in different centers' handbooks using retrieval-augmented language models and compares the resulting answers using a five-label consistency taxonomy. Applied to 102 handbooks from 23 centers and 1,115 benchmark questions, the framework quantifies heterogeneity across four dimensions: question, topic, organ, and center. We find that 20.8% of non-absent pairwise comparisons exhibit clinically meaningful divergence, concentrated in condition monitoring and lifestyle topics. Coverage gaps are even more prominent: 96.2% of question-handbook pairs miss relevant content, with reproductive health at 95.1% absence. Center-level divergence profiles are stable and interpretable, where heterogeneity reflects systematic institutional differences, likely due to patient diversity. These findings expose an information gap in transplant patient education materials, with document-grounded medical question answering highlighting opportunities for content improvement.
1 Introduction
Existing evidence shows substantial variation across transplant centers and their patient-facing materials, but the paper asks whether handbook-grounded answers to the same questions differ clinically across institutions.
- Prior work documents center-level variation in transplant access, recipient-selection information, handbook readability, quality, and clinical-guidance availability.
- The paper grounds identical patient questions in handbooks from different U.S. transplant centers and systematically compares the resulting responses.
- The framework measures institutional variation across organs, topics, and centers rather than identifying a single universally correct answer.
2.1 Data Sources and Processing
The study combines a 102-handbook corpus from 23 U.S. transplant centers with a curated benchmark of 1,115 patient questions. Handbooks and questions are organized by organ, care phase, topic, and subtopic to support cross-center comparison.
- Transplant Patient Handbooks: 102 patient education handbooks from 23 major U.S. transplant centers form the study corpus.The corpus represents 16 of the nation’s 20 largest programs by volume and includes heart, lung, kidney, liver, and pancreas materials.
- Transplant Patient Handbooks: The geographically dispersed corpus covers both large academic medical centers and community-based transplant programs.The handbook collection includes centers distributed across the United States.
- Transplant Patient Handbooks: The handbook corpus includes 37 pre-transplant, 39 post-transplant, and 26 combined handbooks treated as distinct units.This accounts for differences in how centers organize pre-transplant and post-transplant education materials.
- Transplant Patient Question Set: 1,115 patient questions comprise the benchmark for cross-center comparison.Questions were collected from healthcare institutions, community forums, medical organizations, and other sources reflecting transplant patients’ and caregivers’ information needs.
- Transplant Patient Question Set: Each question is annotated with an organ label, one or more of 13 clinical topic categories, and 43 fine-grained subtopic tags.General questions form 27.9% of the benchmark and are answered by every handbook, whereas organ-specific questions are matched to handbooks for the corresponding organ.
2.2 Document Extraction
Raw PDF handbooks are converted into structured JSON that preserves document organization and metadata. This representation supports section-aware downstream processing and resume-safe incremental execution.
- Document Extraction: Raw PDF handbooks are converted into structured JSON representations using LlamaParse.The output preserves section headings, paragraph boundaries, page metadata, full text, and handbook-level metadata.
- Document Extraction: Structured extraction enables section-aware chunking in the downstream retrieval stage.Each JSON handbook contains sections with headings, body text, and page numbers.
- Document Extraction: The extraction pipeline detects already-processed files and skips them for resume-safe execution across incremental corpus updates.This makes the pipeline idempotent.
2.3 Retrieval and Answer Generation
The framework grounds each question–handbook pair with hybrid retrieval and generates standardized answers from center-specific passages. Sparse and dense retrieval are fused, reranked, and limited to the top five passages as context.
- Retrieval: Hybrid retrieval combines sparse lexical matching, dense semantic retrieval, and neural reranking to ground answers in center-specific content.The approach is designed to combine complementary retrieval signals.
- Indexing: Handbook sections are chunked at boundaries, with sections exceeding 512 tokens split at sentence boundaries while retaining their parent headings.Chunks are indexed in parallel BM25 and FAISS structures.
- Hybrid Retrieval with Reranking: Reciprocal Rank Fusion with k_RRF = 60 merges the independently ranked lexical and semantic candidate lists before cross-encoder reranking.The top-5 passages are selected as the retrieval context.
- Answer Generation: For each question–handbook pair, Qwen3-14B at temperature 0 generates an answer from the retrieved passages and original question.The model must rely exclusively on the provided context and return NOT ADDRESSED when relevant handbook information is absent.
2.4 Heterogeneity Analysis
The analysis detects handbook coverage gaps, classifies relationships between non-absent center answers, and aggregates heterogeneity across questions, topics, organs, and centers. These metrics distinguish pervasive divergence from concentrated disagreement and support interpretable center-level profiles.
- Absence Detection and Coverage Analysis: Coverage is assessed first by detecting whether each handbook substantively addresses a question.A heuristic checks for the canonical NOT ADDRESSED prefix, followed by an LLM-based binary classifier for other responses.
- Pairwise Consistency Assessment: Each pair of non-absent answers is assigned one of five relationship labels by an LLM judge.The judge returns a label, clinical justification, optional divergence sub-topic, and clinical significance rating for divergent or contradictory pairs.
- Question-Level Heterogeneity: Question-level divergence and consistency rates quantify clinically meaningful disagreement and full agreement among non-absent center pairs.The residual share consists of complementary pairs, so divergence and consistency rates need not sum to one.
- Topic- and Organ-Level Aggregation: Topic- and organ-level rates average question-level divergence and consistency across each group, while also reporting the proportion of questions with any divergence.This distinguishes groups where divergence is widespread from groups where it is concentrated in a few questions.
- Center-Level Profiles: Center-level profiles aggregate pairwise labels across questions and partner centers to identify systematic institutional differences and outlier guidance.Consistently high divergence may reflect different institutional policies, whereas high consistency relative to peers indicates alignment with prevailing practice norms.
3 Results
Across 102 handbooks from 23 centers and 1,115 benchmark questions, the analysis found substantial coverage gaps and structured heterogeneity in transplant guidance. Divergence was concentrated in monitoring and lifestyle topics, while center profiles remained distinct after controlling for data volume.
- Global Label Distribution: 96.2% of 1,772,261 handbook pairs were ABSENT, leaving 68,019 non-absent pairs for label comparison.Among non-absent pairs, COMPLEMENTARY was most common at 66.0%, followed by DIVERGENT at 20.8%, CONSISTENT at 13.0%, and CONTRADICTORY at 0.2%.
- Coverage Gaps: Lung had the lowest organ-specific absence rate at 72.5%, whereas pancreas had the highest at 85.6%.General-type questions had a 90.5% absence rate because organ-specific handbooks rarely cover cross-cutting topics.
- Coverage Gaps: Reproductive Health had the highest topic-level absence rate at 95.1%, indicating that most handbooks omit fertility, contraception, and pregnancy guidance.Financial & Administrative topics had the lowest absence rate at 72.4%.
- Organ-Level Heterogeneity: Kidney and lung had the highest divergence prevalence at 39.8% and 41.8%, while pancreas had the lowest at 11.0%.Divergence prevalence is the percentage of questions with at least one divergent or contradictory pair.
- Topic-Level Heterogeneity: Monitoring & Follow-up had the highest mean divergence rate at Rdiv = 0.277, followed by Lifestyle & Daily Living at Rdiv = 0.235.Their divergence prevalences were 38.9% and 40.4%, respectively, reflecting variation in surveillance, diet, exercise, and activity guidance.
- Center-Level Profiles: Among 23 centers, divergence rates ranged from Rdiv = 0.139 to 0.255 and consistency rates from Rcon = 0.082 to 0.194.The highest-divergence and most-consistent center profiles persisted after controlling for the number of non-absent pairs.
- Illustrative Comparison Matrices: Comparison matrices revealed structured agreement and disagreement, including consensus on blood-type waiting-time guidance and extreme divergence for blood-work frequency.Dental-care antibiotic prophylaxis displayed all five labels, including contradiction, reflecting clinical controversy.
4 Discussion and Conclusion
The framework reveals substantial but uneven institutional heterogeneity in transplant guidance: clinically meaningful divergence coexists with pervasive coverage gaps, and center-level patterns are structured enough to support quality improvement. These findings also expose methodological and scope boundaries for interpreting the results.
- Principal Findings: 20.8% of 68,019 non-absent pairwise comparisons were classified as DIVERGENT, while 0.2% were CONTRADICTORY, with divergence concentrated in monitoring and lifestyle topics.Lung and kidney questions had the highest divergence prevalence, whereas pancreas had the lowest.
- Principal Findings: 96.2% of question-handbook pairs lacked relevant content, including a 95.1% absence rate for Reproductive Health.The omission includes guidance on fertility, contraception, or pregnancy.
- Principal Findings: Center divergence rates ranged from 0.139 to 0.255, indicating stable institutional profiles that may reflect clinical philosophy or authoring conventions.The analysis characterizes these profiles as systematic differences rather than sample-size artifacts.
- Principal Findings: Question-level comparison matrices showed structured agreement and disagreement on topics including dental antibiotic prophylaxis and monitoring frequency.These patterns connect observed heterogeneity to documented clinical controversies rather than random noise.
- Implications: Document selection in retrieval-augmented medical question answering functions as an implicit clinical decision because single-center systems inherit institutional omissions and positions on contested topics.Coverage-gap and center-profile analyses can identify expansion priorities and institutions misaligned with peer consensus.
- Limitations: The study is limited by possible systematic bias in the LLM-based pairwise judge, imperfect retrieval, text-only processing, and a corpus restricted to English-language handbooks from 23 U.S. centers.Multimodal extraction and international, non-English materials could improve gap estimates and generalizability.