Source-linked AI summary
Evaluating Biomedical Reranking for LLM-Based Question Answering over Longitudinal Clinical Notes
Maryam Shahbaz Ali, Laura B. Strachan, Caitlin Sherman, Mark Kovler, Eleanor Mackey, Syed Muhammad Anwar
TL;DR
Longitudinal clinical QA must locate patient-specific evidence scattered across heterogeneous notes. This study combines hybrid retrieval with MedCPT reranking and finds improved exact-source retrieval alongside a smaller gain in answer correctness.
Problem
Patient-specific QA must locate relevant evidence scattered across encounters, copied-forward notes, and differing clinical terminology in long records.
Source: §1 · p. 1Method
The locally deployed, patient-restricted RAG evaluation combines PubMedBERT dense retrieval, BM25, weighted reciprocal-rank fusion, and MedCPT cross-encoder reranking.
Source: §1 · pp. 1–2, §2.1 · p. 2Results
Hit@10 rose from 46.6% to 60.6%, while automated answer correctness rose from 44.8% to 48.6% with MedCPT reranking.
Source: §5 · p. 4Takeaways & Limitations
Improved evidence retrieval did not translate proportionally into answer correctness, underscoring a distinction between evidence selection and downstream answer quality.
Source: §4 · p. 4The same Qwen3-8B family generated and judged answers, the development cohort was reused for configuration selection, and independent human evaluation and held-out validation are needed to establish generalization.
Source: §4 · p. 4
Abstract
from arXiv · showhide
Patient-specific clinical question answering requires locating the right evidence within long, heterogeneous longitudinal clinical records in which relevant facts may be scattered across encounters, repeated in copied-forward notes, or expressed using different clinical terminology. We evaluated whether biomedical reranking can improve evidence selection and downstream answer quality in a locally deployed retrieval-augmented generation pipeline for longitudinal clinical notes. The pipeline combines PubMedBERT dense retrieval, BM25 lexical retrieval, weighted reciprocal-rank fusion, and MedCPT cross-encoder reranking. Across 1,000 open- and closed-ended question-answer pairs from a cohort of 200 bariatric surgery patients, reranking increased exact source-chunk retrieval within the top 10 items, Hit@10 from 46.6% to 60.6% and mean reciprocal rank from 0.2371 to 0.3252. With Qwen3-8B generation, local judge-assessed answer correctness increased from 44.8% to 48.6%. These results show that biomedical reranking can improve the placement of relevant clinical evidence within a limited context window, although gains in retrieval do not translate proportionally into gains in answer correctness.
1. INTRODUCTION
Patient-specific QA must find informative evidence scattered across a patient’s longitudinal record, and the value of an answer depends on both retrieval and interpretation. This study evaluates biomedical cross-encoder reranking in a local, patient-restricted RAG pipeline.
1. INTRODUCTION
- Relevant facts may be scattered across encounters, repeated in copied-forward notes, or expressed with different terminology, making evidence selection difficult within a limited context window.The evidence may be buried in one patient’s longitudinal record rather than contained in model pretraining.Source: §1 · p. 1
- RAG supplies patient-specific evidence to a language model, but retrieving relevant evidence and correctly interpreting it are separate pipeline stages.Prior medical QA evaluation frameworks include MIRAGE and MedRAG.Source: §1 · p. 1
- Prior work covers medical QA, EHR summarization, clinical prediction, and longitudinal reasoning, but does not directly evaluate biomedical cross-encoder reranking for patient-specific QA over longitudinal notes.The cited work includes RAG for EHR summarization and information extraction, RAM-EHR for clinical prediction, and studies of biomedical and longitudinal EHR reasoning -.Source: §1 · p. 1
- The study evaluates biomedical cross-encoder reranking, relates retrieval gains to answer correctness by question type, and conducts local patient-restricted RAG evaluation with identifier screening.The evaluation uses 1,000 QA pairs from 200 bariatric surgery patients, with retrieval restricted to each patient’s longitudinal record.Source: §1 · pp. 1–2
2. METHODS
The methods combine dense and lexical candidate retrieval with weighted rank fusion, then rerank candidates and generate answers locally. Retrieval and generation use patient-specific clinical notes and a selected set of top-ranked chunks.
2. METHODS
- Figure 1 presents an overview of the proposed method, with details about each component provided in the following text.Source: §2 · p. 2
2.1. Dense and lexical candidate generation
- PubMedBERT dense retrieval and BM25 lexical retrieval each return up to 30 candidates, which are combined using weighted reciprocal-rank fusion.The fusion weights are 0.30/0.70, selected during development with greater weight on BM25 because it performed better on this cohort; the denominator constant 60 is the standard RRF offset.Source: §2.1 · p. 2
2.2. Biomedical reranking and generation
- MedCPT scores the union of dense and BM25 candidates, and the ten highest-scoring chunks are supplied as context to local Qwen3-8B answer generation.The reranker processes question–passage pairs with a 512-token combined input limit, truncating only the passage; generation can also run without retrieved context.Source: §2.2 · p. 2
3. EXPERIMENTS AND RESULT
Across a 200-patient cohort, the evaluation tested patient-restricted retrieval and answer generation on 1,000 questions. MedCPT reranking improved exact-source retrieval and answer correctness, while chunk-window changes did not improve retrieval and evaluation remained limited by the absence of independent human validation and a held-out test split.
3.1. Dataset
- The corpus comprised 71,289 patient-linked chunks from 40,197 cleaned notes, with retrieval restricted to each corresponding patient's records.Patient-restricted retrieval assumes patient identity is already known and does not evaluate cross-patient record discovery.Source: §3.1 · p. 3
3.2. Evaluation questions and reference evidence
- The frozen set contained 1,000 questions—500 open- and 500 closed-ended—from 200 patients, but evaluation lacked independent human validation and a held-out test split.An initial automated quality check classified 993 questions as pass, four as requiring review, and three as fail.Source: §3.2 · p. 3
3.3. Retrieval and answer metrics
- Exact-source Hit@10 counted designated chunks among ten results, while answer judging compared blinded outputs against frozen references without retrieved passages.RAG-variant comparisons used separate automated judging runs and were described as exploratory; patient-cluster intervals were the principal uncertainty summaries.Source: §3.3 · pp. 3–4
3.4. Evidence retrieval
- 60.6% vs 46.6% Hit@10 and 0.3252 MRR: MedCPT reranking improved exact-source retrieval over weighted fusion across 1,000 questions.The top-30 dense and top-30 BM25 candidate union contained the designated chunk for 77.8% of questions; 222 targets were absent from that pool.Source: Table 1 · p. 3, §3.4 · p. 4
3.5. Downstream question answering
- 48.6% vs 44.8% answer correctness after reranking; 222 of 606 retrieval-hit questions were still judged incorrect.Correctness was 63.4% for questions with top-ten retrieval hits and 25.9% for retrieval misses.Source: §3.5 · p. 4
3.6. Window ablation and identifier screening
- 5,528 chunks (7.75%) exceeded 512 tokens; the 448-token overlapping-window ablation did not improve hybrid retrieval, and flagged outputs required review.Identifier screening flagged 44 weighted-hybrid answers and 40 reranked answers; different flagged subsets preclude a matched comparison of filtered accuracies.Source: §3.6 · p. 4
4. DISCUSSION
The discussion reports larger gains in evidence retrieval than in answer correctness, and identifies several factors that constrain interpretation and generalization.
4. DISCUSSION
- Hit@10 increased by 14.0 percentage points, while automated answer correctness increased by 3.8 points overall.Stratified correctness gains were 2.2 points for open-ended and 5.4 points for closed-ended QA.Source: §4 · p. 4
- Remaining errors may reflect incomplete evidence, distracting context, generation errors, reference limitations, or judge errors.Source: §4 · p. 4
- Interpretation is limited by shared Qwen3-8B generation and judging, reuse of the development cohort, possible undercounting of equivalent evidence, and the patient-association assumption.Independent human evaluation and held-out validation are needed to establish generalization.Source: §4 · p. 4
5. CONCLUSION
In a local patient-specific QA evaluation, MedCPT reranking improved exact-source retrieval and automated answer correctness. The conclusion distinguishes gains in evidence retrieval from downstream answer quality and calls for independent human evaluation and held-out validation to establish generalization.
5. CONCLUSION
- Exact-source Hit@10 rose from 46.6% to 60.6%, while automated answer correctness rose from 44.8% to 48.6%.Source: §5 · p. 4
- The results support a distinction between evidence retrieval and downstream answer quality.Source: §5 · p. 4
- Independent human evaluation and held-out validation are needed to establish generalization.Source: §5 · p. 4