PaperlayerView exact revision on arXiv

Source-linked AI summary

Where Does Retrieval-Based Open-Ended Evaluation Fail? Automatic Taxonomy Induction from Long-Form Medical Answer Factuality Verification

Heyuan Huang, Jirui Dai, Alexandra DeLucia, Sonal Joshi, Mahsa Yarmohammadi, Jie Gao, Bernal Jiménez Gutiérrez, Mark Dredze

arXiv:2609.30467v1cs.CLcs.IR

TL;DR

Open-ended medical factuality evaluation lacks gold answers and evidence for diagnosing where retrieval-based verification fails. The paper induces retrieval and verifier-error taxonomies with an LLM-as-judge pipeline, and finds that scaling models or changing retrieval conditions does not resolve the severe open-ended performance gap.

  • Problem

    Open-ended factuality pipelines lack annotated verdicts and gold passages, leaving retrieval as a black box and evidence recall@k undefined for some claims.

    Source: §2 · p. 2

  • Method

    The authors induce a taxonomy of retrieval-stage and verifier-reasoning errors with an LLM-as-judge pipeline that needs no gold answers or annotated gold evidence.

    Source: §1 · pp. 1–2

  • Results

    On MedExpert, strong retrievers paired with frontier verifiers recover only a small fraction of clinician-flagged errors, an order-of-magnitude collapse from closed-ended biomedical benchmarks that persists across tested interventions.

    Source: §1 · p. 1

  • Takeaways & Limitations

    The authors recommend task-oriented multidimensional evaluation of retrieved evidence and intermediate verifier behavior rather than reliance on aggregated end-to-end metrics.

    Source: §1 · p. 2, §7 · p. 8

    The study’s open-ended analysis is based mainly on MedExpert and MedRAG, the only publicly available clinician-annotated dataset and authoritative medical corpus identified by the authors as of May 2026.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Retrieval-based factuality evaluation, where LLM-generated claims are verified against evidence from authoritative medical corpora, has become the dominant paradigm for scalable hallucination detection in high-stakes clinical settings. Despite the urgency of reliable and transparent medical fact verification, most systems measure performance with aggregate metrics like F1, which obscure where and why failures occur. Existing RAG diagnostics require gold answers or annotated gold evidence, neither of which exists in this regime. We introduce two comprehensive taxonomies, grounded in a case study on the open-ended MedExpert dataset and 3 closed-ended datasets, decomposing failures into retrieval-stage errors along five quality dimensions, and verifier-reasoning errors into six consecutive steps. We adapt an automatic pattern induction pipeline using LLM-as-Judge to label evidence quality and classify verifier reasoning errors at scale, and then stress-test our findings across 4 retrieval methods and 6 frontier verifier models. Our analysis reveals that scaling model size, adding reasoning effort, expanding to authoritative web sources, and applying medical fine-tuning do not resolve these failure modes, demonstrating that they represent fundamental limitations of the retrieve-then-verify paradigm in open-ended medical settings rather than artifacts of outdated systems. We release our code and data at https://anonymous.4open.science/r/Medical_RAG_eval-4AB5 for the full reproducibility of our results.

1 Introduction

Retrieval-based factuality evaluation is widely used for generated medical text, but aggregate scores do not reveal where verification pipelines fail. This work introduces a scalable taxonomy of retrieval and verifier errors and finds that strong systems still struggle on open-ended medical QA.

1 Introduction

  • On MedExpert, even strong retrievers paired with frontier verifiers recover only a small fraction of clinician-flagged errors—an order-of-magnitude drop from closed-ended biomedical benchmarks.The gap persists with larger models, more reasoning effort, medical fine-tuning, and authoritative web sources.Source: §1 · p. 1
  • The paper induces a model-agnostic taxonomy separating retrieval-stage evidence bottlenecks from verifier-reasoning errors.Retrieval errors include relevance, clinical scope, trustworthiness, polarity, and chunking; verifier errors include selection, comprehension, reasoning, calibration, consistency, and grounding.Source: §1 · p. 1
  • A human-in-the-loop LLM-as-judge pipeline induces and scales the taxonomy without requiring gold answers or annotated gold evidence.Those annotations are unavailable in this open-ended regime, where existing closed-ended RAG diagnostics do not apply.Source: §1 · pp. 1–2
  • The authors analyze six verifier LLMs, four retrievers, and three knowledge corpora, and recommend assessing retrieved evidence and verifier behavior separately from end-to-end scores.They argue this multidimensional protocol reflects system capability more faithfully and surfaces failure modes more directly than aggregate metrics.Source: §1 · p. 2

2 Related Work

Closed-ended fact verification can diagnose retrieval using gold evidence, but open-ended factuality evaluation lacks comparable annotations and standardized retriever evaluation. This paper addresses that gap with systematic error analysis and automatic pattern extraction.

2 Related Work

  • Open-ended factuality pipelines typically return aggregate verdict scores and plain-text reasoning, making it difficult to diagnose failures within the retrieve-then-verify process.Prior approaches such as FActScore and MedScore retrieve external snippets for claim verification.Source: §2 · p. 2
  • Closed-ended benchmarks provide gold passages and verdicts, enabling direct retriever precision@k and recall@k evaluation and separation of retrieval misses from verifier errors.Diagnostic frameworks build on this annotated protocol.Source: §2 · p. 2
  • Open-ended claims are generated from long-form answers without gold verdicts or passages, and some may lack support in any corpus, making evidence recall@k undefined in principle.Existing pipelines therefore treat retrieval as a black box, with no standardized framework or benchmark for evaluating the retriever itself.Source: §2 · p. 2
  • Existing medical-RAG diagnostics still depend on gold answers or must-have statements from physician-written answers, while this work analyzes retrieval without a reference answer.The authors describe their work as the first systematic retrieval-stage error analysis in that setting.Source: §2 · p. 2
  • The paper focuses on categorizing stage-specific failures through automatic LLM-based pattern extraction, rather than building a better fact-verification system.The automation is intended to scale the analysis beyond manual inspection.Source: §2 · p. 2

3 Experimental Setup

The study combines clinician-annotated open-ended medical QA with closed-ended medical fact-checking datasets adapted to a shared large evidence corpus. It evaluates four retrieval methods and six verifier models, then uses LLM-as-judge pattern extraction to classify retrieval quality and verifier reasoning errors.

3.1.1 Open-ended Datasets

  • MedExpert provides 540 open-ended medical QA pairs in mental health and prenatal care, with clinician-highlighted false spans, labels, and reasons; a subset is double-checked by five clinicians.The study also uses MedExpert-gold and reports that findings generalize to simulated semi-open-ended datasets.Source: §3.1.1 · p. 3

3.1.2 Closed-Ended Datasets

  • Four closed-ended medical fact-check datasets are converted to a unified claim format and searched against the 30.3M-passage MEDIC corpus rather than their original small evidence corpora.The supplied dataset descriptions include SciFact, HealthVer, and CovidFact.Source: §3.1.2 · pp. 3–4

3.2 Evaluation Setup

  • Claims receive one of four verdicts—Supported, Refuted, No Evidence, or Ambiguous—with Ambiguous used when retrieved passages conflict.The authors note that such disagreement is common across medical studies, guidelines, and clinical sources.Source: §3.2 · p. 4

3.3 Methods

  • Four retrievers—MedCPT, RRF-2, RRF-4, and Qwen3-Embedding-8B—retrieve the top 10 passages for each decomposed claim.The hybrids combine BM25 with biomedical, scientific, and general-domain semantic retrievers.Source: §3.3 · p. 4
  • The three knowledge sources are MEDIC, Google general search, and Google Scholar search.Google search tests authoritative-domain filtering and current evidence against static corpus retrieval; Scholar offers broader scope than MEDIC but narrower scope than general Google search.Source: §3.3 · p. 4
  • Six verifier LLMs span general and medical fine-tuning, including GPT-5.4, MedGemma 27B, Gemma 3 27B, Qwen3.6-27B, and two Mistral Small versions.The full pipeline and automatic error analysis are run for each retrieval-method, corpus, and verifier combination.Source: §3.3 · p. 4

3.4 Automatic Pattern Extraction

  • An LLM-as-judge pipeline expands a clinician-informed seed codebook, rating evidence on entity match, clinical scope, trustworthiness, and polarity and classifying verifier traces across six reasoning steps.The final codebooks are manually synthesized and judge labels are validated against a human-annotated subset.Source: §3.4 · p. 5

4 Open-Ended vs. Closed-Ended Fact Verification Results

The section contrasts low factuality-verification performance on open-ended MedExpert with stronger results where relevant evidence is available, and identifies evidence coverage, identifiability, and claim–evidence alignment as key challenges.

Open-Ended vs. Closed-Ended Fact Verification Results

  • MedExpert F1 is 0.06, versus 0.4–0.5 on COVID-related datasets and 0.7 on SciFact and SciFact-Open once relevant evidence is retrieved.The strongest pipeline still has consistently low Recall and F1 on open-ended MedExpert.Source: §4 · p. 5, Table 2 · p. 6
  • In MedExpert, evidence coverage is not guaranteed, so missing gold evidence cannot be remedied by better retrievers or verifiers or simple corpus expansion.By contrast, approximately 90% to 81% of SciFact and SciFact-Open gold passages are contained in MEDIC.Source: §4 · pp. 5–6
  • When gold evidence exists among millions of passages, it is harder to locate than in the small, curated search spaces of closed-ended datasets.The authors characterize this identifiability gap as scaling-friendly: better retrievers can shrink it and reduce the corresponding share of end-to-end errors.Source: §4 · p. 6
  • Medical claims and evidence can differ in units, population, or the number of passages needed, requiring domain knowledge and cross-passage reasoning.These alignment demands account for a substantial share of errors that end-to-end metrics fail to reveal.Source: §4 · p. 6

5 Quantitative Analysis

The quantitative analysis finds that stronger retrievers, broader corpora, and stronger verifier configurations do not remove the open-ended medical verification challenge. Across these comparisons, end-to-end performance and benchmark standing do not consistently track the quality of evidence use or reasoning.

5 Quantitative Analysis

  • Recall remains stable on balanced MedExpert-gold because most clinician-found factual errors cannot be automatically detected.The analysis uses MedExpert-gold for detailed results and reports MedExpert-full results separately.Source: §5 · p. 6

5.1 Better Retrievers Can’t Solve the Challenge

  • Qwen3 is the strongest retriever by Recall and F1, but MedCPT/RRFs have lower Recall and F1, and Table 5 shows Qwen3 produces more decisive Supported labels.Qwen3 has the fewest No Evidence labels (8.04%) and turns most such labels from other retrievers into Supported (80.14%).Source: §5.1 · p. 7, Table 5 · p. 7

5.2 Better Search Can’t Solve the Challenge

  • Replacing MEDIC with Google General Search, Google Scholar Search, or their merged corpus does not improve Recall or F1.The comparison uses GPT-5.4 and Qwen3 on MedExpert-gold; authoritative web sources can also contain unfactual content.Source: §5.2 · p. 7, Table 6 · p. 7

5.3 Better Verifiers Can’t Solve the Challenge

  • Mistral Small 3 has the highest Recall despite its small size, while GPT-5.4 and Qwen 3.6 have much lower Recall, so healthcare benchmark rank does not predict open-ended verification performance.Figure 2 compares MedExpert-gold Recall with healthcare benchmark rank across verifier models.Source: §5.3 · p. 7, Figure 2 · p. 7
  • Increasing GPT-5.4’s reasoning effort consumes substantially more reasoning tokens without improving Recall, and its reasoning is described as useless hedging.The comparison is the Table 7 reasoning-effort ablation with Qwen3 on MedExpert-gold.Source: §5.3 · p. 7, Table 7 · p. 7
  • MedGemma has roughly 2 times Gemma’s Recall but remains far behind the strongest verifier, with a similar overall gap in F1.The passage concludes that medical fine-tuning does not yield significant improvements beyond the base architecture and pretraining capabilities.Source: §5.3 · pp. 7–8

6 Qualitative Analysis

The qualitative analysis applies induced taxonomies to characterize retrieval quality and verifier reasoning, then quantifies the patterns found. It identifies distinct limitations across models and retrievers that end-to-end scores alone do not reveal.

6.1 Verifier Behavior Taxonomy

  • Verifier traces are judged across six consecutive steps, and a trace counts as correct only if every step is correct.GPT-5.4 has the highest full-trace correctness rate (90%), while Mistral Small 3 has the lowest (66%); its highest Recall is described as mostly a lucky guess based on wrong reasoning traces.Source: §6.1 · p. 8
  • Mistral Small 3 often misses or misunderstands evidence and is overconfident, while Gemma3 attends to irrelevant passages and GPT-5.4 retains inference- and confidence-step errors.MedGemma anchors attention to correct passages given the same evidence; newer and larger models generally show stronger reasoning performance.Source: §6.1 · p. 8

6 Qualitative Analysis

  • The retrieval taxonomy evaluates passages across entity match, scope match, content trustworthiness, evidence polarity, and chunking issues.The final codebook did not differ significantly from the seed codebook after six iterations; the codebooks were applied to 800 claim-evidence pairs and 576 claim-reasoning traces, with human-LLM agreement of 0.86 and 0.8, respectively.Source: §6.2 · p. 8, §6 · p. 8

6.2 Retrieval Quality Taxonomy

  • Qwen3’s high-quality passage rate is 14.5%, seven times MedCPT’s 2%, but its remaining challenges are clinical scope and direct affirmation or contradiction signals.A passage is counted high quality only when it meets all five quality dimensions.Source: §6.2 · p. 8

7 Conclusion

The conclusion recommends task-oriented, multidimensional evaluation of retrieved passages and intermediate verifier behavior rather than relying on aggregate metrics. The retrieval taxonomy omits temporal information because of data limitations.

7 Conclusion

  • Researchers should assess retrieval and re-ranking with customized task-oriented dimensions and evaluate intermediate verifier behavior instead of competing on aggregate P/R/F1.The passage identifies locating information scattered across sources and aligning evidence with claims as a central difficulty.Source: §7 · p. 8
  • The retrieval taxonomy omits temporal quality because the available data are limited.The passage notes that temporal quality could be one dimension.Source: §7 · p. 8

Limitations

The analysis is limited by missing temporal metadata, unavailable gold labels for formula-based RAG metrics, and budget-constrained pattern tagging.

Limitations

  • MedRAG lacks passage publication-date metadata, preventing evaluation of the Retriever Taxonomy’s temporal dimension.The authors note that this dimension could be enriched if other resources become publicly available.Source: Limitations · p. 9
  • Without gold passage labels or gold answers, the study cannot calculate retriever precision@k and recall@k or generator metrics.Additional resources could allow researchers to compare formula-based metrics with the qualitative findings.Source: Limitations · p. 9
  • API-budget limits confined automatic pattern-percentage tagging to MedExpert-gold, so tagging more data could make those percentages more comprehensive.Source: Limitations · p. 9

A.1 Full Retriever and Verifier Taxonomy Codebook

The codebook reports retrieval quality by dimension and verifier correctness by reasoning step, rather than relying only on aggregate pipeline metrics.

Retriever Taxonomy

  • Table 8 reports each retrieval dimension’s high-quality rate in isolation, while its metrics block reports passages high quality across all dimensions.For each error pattern, the highest retriever error rate is bolded and the dominant error italicized; best metric scores are bolded.Source: §A.1 · p. 11

Taxonomy tables

  • The tables identify the strongest system per metric in bold and each system’s dominant error in italics.Source: §A.1 · p. 11

Verifier Taxonomy

  • Table 9 reports verifier correctness per reasoning step because end-to-end accuracy alone cannot certify trustworthiness in high-stakes settings.Green and yellow sub-codes within a step are mutually exclusive, but multiple yellow errors may co-occur, so row percentages can exceed 100%.Source: §A.1 · p. 11

A.2 Data Preprocessing and Alignment

The preprocessing and alignment procedures form adjudicated MedExpert-gold annotations, normalize and map closed-ended datasets, and describe the evaluation labels and table denominators.

MedExpert

  • Five clinicians adjudicated 225 MedExpert sentences initially marked incorrect, retaining 86 as errors and reclassifying 139 as factually correct.Source: §A.2 · p. 11

MedExpert-gold

  • MedExpert-gold includes all original Has Error sentences, making its Recall generalizable to the full set, but not its Precision or F1.The five-clinician adjudication was intended to provide reliable annotations while reducing analysis costs.Source: §A.2 · p. 11

SciFact datasets

  • SciFact-Open was deduplicated from 279 to 276 claims, with evidence labels aggregated into claim-level verdicts; SciFact training and validation sets supplied 1,109 labeled claims.Source: §A.2 · pp. 11–12

HealthVer

  • HealthVer whitespace normalization merged variants into 1,851 claims, whose evidence labels were aggregated into Supported, Refuted, No Evidence, or Ambiguous verdicts.Source: §A.2 · p. 12

CovidFact

  • CovidFact duplicate removal reduced 4,086 claims to 4,081, and its original binary labels mapped directly to Supported or Refuted.Source: §A.2 · p. 12

Binary evaluation alignment

  • Binary evaluation uses three mappings that vary whether No Evidence and Ambiguous count as false, while MedExpert alignment uses an any-error sentence criterion.Source: §A.2 · pp. 11–12

Taxonomy table evaluation

  • Table 8 percentages use 200 annotated passages per retriever, and its bottom block reports GPT-5.4 performance on MedExpert-gold.Source: Table 8 · p. 13
  • Table 9 percentages use 96 claims per verifier, allow co-occurring error codes, and report Qwen3-retriever end-to-end performance on MedExpert-gold.Source: Table 9 · p. 14

A.3 Closed-ended Datasets’ Evidence Overlap

The section estimates how much benchmark evidence is represented in MEDIC using document-title overlap, finding markedly different coverage across the closed-ended datasets.

Method

  • Evidence overlap is computed as the fraction of unique benchmark evidence titles also present among MEDIC’s unique normalized paper titles.MEDIC passages and benchmark evidence are collapsed and normalized at the title level before comparison.Source: §A.3 · p. 15
  • The evaluation corpus supplies the denominator because the measure estimates MEDIC’s retrievable evidence coverage, not relative corpus size.Source: §A.3 · p. 15

Evidence overlap results

  • 90.12% and 81.46% evidence overlap for SciFact and SciFact-Open, respectively, exceed CovidFact’s 14.13% and HealthVer’s 41.72%.The authors characterize SciFact and SciFact-Open as comparatively unchallenging when evidence is retrieved, while the lower-overlap datasets are more challenging under MEDIC retrieval.Source: §A.3 · p. 15, Table 10 · p. 15

A.4 Retrieval and Verification Prompts

This appendix documents the retrieval and verification prompts and the induction and validation of evaluation taxonomies. Examples distinguish retrieval-quality and verifier-reasoning failures across evidence selection, interpretation, inference, confidence, label consistency, and grounding.

A.4 Retrieval and Verification Prompts

  • Zero-shot retrieval seeks biomedical passages that support or refute each claim, while a four-label verifier judges claims from retrieved evidence.The prompt design aims to assess evidence use as well as final-label performance.Source: §A.4 · p. 15, Table 11 · p. 16

A.5 Experiment Computation and Hyperparameters

  • All retrieval experiments used MEDIC and returned 10 documents per claim, balancing evidence coverage against verification cost.The full MedExpert retrieval runs ranged from 5h13m for MedCPT to 27h04m for Qwen3-Embedding-8B.Source: §A.5 · p. 15

A.6 Details of Taxonomy Construction and Validation Process

  • The team iteratively expanded seed codebooks with Claude Opus 4.7 until no new patterns emerged, then removed extremely rare unmatched leaves.After six seed-codebook versions, the retriever taxonomy saturated on 8,080 passages and the verifier taxonomy on 1,212 reasoning traces.Source: §A.6 · p. 17
  • Claim-evidence labels had high pooled agreement, while verifier-reasoning labels showed moderate agreement, strongest for Grounding and Label Consistency and weakest for Evidence Selection.Evidence-label agreement was 91.2% raw with κ = 0.90 and α = 0.90 at the dimension level; reasoning-label agreement was 80.0% raw with pooled κ = α = 0.57. Category-level κ was 0.80 for Grounding, 0.78 for Label Consistency, and 0.26 for Evidence Selection; annotator κ ranged from 0.24 to 0.89, reflecting the greater interpretive judgment required for reasoning traces.Source: §A.6 · pp. 17–18

A.7 Google Search Example; MedRAG

  • In the orthostatic-hypotension example, retrieved sources define a blood-pressure drop on standing, supporting the verifier’s Refuted judgment that dizziness is not another name for the condition.The example contrasts a symptom with the diagnosis itself.Source: §A.7 · p. 19, MedRAG · p. 19

A.8 MedExpert Full Results; A.9 Statistical Significance Analysis of Experimental Results

  • No intervention significantly improved recall or F1 after uncertainty and multiplicity were accounted for; every significant effect favored the baseline configuration.The full MedExpert verifier ablation results are reported in Table 15. The static MEDIC corpus significantly outperformed live web search on recall (Cochran’s Q p = 0.0360) and consistently on F1 (∆F1 up to −11.8 pp); baseline-verifier F1 was higher than for newer and medically tuned Gemma alternatives (p ≤0.005).Source: §A.9 · p. 21, Table 15 · p. 22, Table 16 · p. 22

A.10 Granularity Alignment Between System Outputs and Sentence-Level Labels

  • Under the strict label mapping, any Refuted, No Evidence, or Ambiguous atomic claim makes its parent sentence Has Error; about 80% of system reasons aligned with clinician reasons.Only sentences whose claims were all Supported were predicted as No Error; 10 system reasons were not aligned.Source: §A.10 · p. 21, Table 19 · p. 23

A.11.1 Retriever-Behavior Examples

  • Retriever-behavior examples distinguish entity match, PICOT scope, source trustworthiness, and evidence polarity, including irrelevant evidence and sources ranging from systematic reviews to case reports.The examples operationalize direct, partial, and irrelevant entity matches; exact, partial, and inapplicable scope; and reliable versus unreliable sources.Source: §A.11.1 · pp. 24, 26, 27
  • Direct affirmation or contradiction gives evidence a stance on the claim, while neutral passages mention the topic without taking a stance.Examples include evidence that labor may be prolonged and a question about labor that states no proposition.Source: §A.11.1 · p. 27
  • When entity match is irrelevant, polarity is not applicable; chunking is appropriate when a passage retains enough context, but titles, headers, or citations alone are fragmented.The examples contrast a self-contained monitoring definition with table-header-only and citation-only chunks.Source: §A.11.1 · pp. 27–28

A.11.2 Verifier-Behavior Examples

  • Evidence selection ranges from covering relevant passages to selective coverage or misallocated attention toward omitted, conflicting, or off-topic evidence.Selection is assumed from the citations in the verifier’s reasoning rather than treated as an explicit reasoning step.Source: §A.11.2 · p. 28

Faithful read

  • Faithful reading preserves qualifiers, gaps, and competing evidence; comprehension failure includes misreading claims, flipping evidence stance, or overgeneralizing conclusions.Examples include clinical-construct conflation and misunderstanding a claim about placenta previa.Source: Faithful read · p. 29

Sound inference; Inferential overreach; Conflict synthesis failure

Appropriate confidence; Overconfident; Underconfident

Underconfident; Mismatch

  • Label consistency requires the final verdict to match the verifier’s reasoning; mismatch includes a label that contradicts the stated conclusion or its uncertainty.A reasoning conclusion of support paired with a “No Evidence” label is one mismatch example.Source: Underconfident · p. 31, Mismatch · p. 31

Grounded reasoning; Ungrounded reasoning

  • Grounded reasoning keeps cited information traceable to evidence, while ungrounded reasoning fabricates citations or quotes or relies on internal knowledge instead.The examples contrast traceable quotations with invented figures and outside-knowledge conclusions.Source: Grounded reasoning · p. 32, Ungrounded reasoning · p. 32
Loading 2609.30467v1…