PaperlayerView exact revision on arXiv

Source-linked AI summary

How Ground Truth Shapes Retrieval Scores: Evidence-Aware Evaluation of a Retrieval-Augmented Generation System for Breast Lesions

Miguel Mochizuki Silva, Samuel Ismael, Mateus C. Dias, Thaís Gaudêncio, Yuri Malheiros, Barufaldi

arXiv:2610.31606v1cs.CLcs.AI

TL;DR

Clinical RAG retrieval scores can depend on which supporting passages are annotated as relevant, especially when chunking creates overlapping alternatives. This study evaluates retrieval and generation separately using evidence-aware annotations on a small breast-lesion benchmark. With cross-encoder reranking, accepted evidence appeared in the top three for 17 of 18 queries, and expanded annotations recognized more evidence without changing rankings.

  • Problem

    Retrieval scores depend on annotated relevance: chunks containing the same supporting evidence may be scored nonrelevant when only one chunk is labeled.

    Source: §1 · p. 2

  • Method

    The study maps canonical evidence spans to chunks, audits additional supporting chunks with rankings fixed, and evaluates retrieval configurations and generation separately on a breast-lesion benchmark.

    Source: §1 · p. 2

  • Results

    17 of 18 queries had accepted evidence among the top three with hybrid RRF and cross-encoder reranking (Recall@3 = 0.94), while curation expanded evidence spans from 25 to 37 without changing rankings.

    Source: §5 · p. 14

  • Takeaways & Limitations

    Expanded relevance annotations increased recognized evidence without changing rankings, illustrating that retrieval evaluation depends on annotation coverage.

    Source: §5 · p. 14

    The 18-query development benchmark used for configuration selection and evaluation, and retrieval-informed curation, require independent validation of generalization and evidence sufficiency.

    Source: §4.1 · p. 13

Abstract

from arXiv · show

Purpose: To describe an evidence-aware retrieval evaluation protocol for a clinical retrieval-augmented generation (RAG) system over breast lesion management literature, addressing the dependence of retrieval scores on annotation coverage. Methods: An 18-document corpus and an 18-query development benchmark, with one Yes/No or WH-question per document, were used. Retrieval combined dense embeddings, BM25, and cross-encoder reranking. Chunk sizes of 128 to 1024 tokens, overlaps of 0 to 20%, and four retriever configurations were evaluated. Relevance was defined by evidence spans mapped to all containing chunks, treating overlapping chunks with equivalent evidence as relevant. Retrieval was evaluated using R-Precision, Recall@K, MRR, and nDCG@K. Separately, generation was assessed on a 512-token index at K=1 to 5 using DeepEval claim-level metrics, with deepseek-v4-pro as the LLM judge, and BLEU/ROUGE metrics. Results: The selected hybrid retriever, using 384-token chunks and 20% overlap, retrieved accepted evidence in the top three results for 17 of 18 queries. Recall@3, R-Precision, MRR@3, and nDCG@3 each reached 0.94. Ground-truth curation increased evidence spans from 25 to 37 and accepted chunk targets from 36 to 51 without changing retrieval rankings. Conclusion: Cross-encoder reranking achieved the highest retrieval scores. Expanded relevance annotations increased recognized evidence without changing rankings, highlighting annotation sensitivity. The small development set and retrieval-informed curation require independent validation, including radiologist assessment of evidence sufficiency.

1 Introduction

Clinical breast-lesion management draws on sources whose recommendations can differ and depend on clinical conditions, while medical RAG evaluation remains heterogeneous. This study evaluates retrieval separately from answer generation and examines how relevance annotations shape retrieval scores.

1 Introduction

  • For uncertain-potential B3 lesions, international consensus supports vacuum-assisted excision for selected categories, while U.S. guidance permits surveillance for selected concordant lesions.Retrieving a recommendation therefore requires preserving the clinical conditions under which it applies.Source: §1 · p. 2
  • RAG conditions answers on retrieved passages and exposes sources for inspection, but missing evidence can limit correctness and longer contexts may go unused.Medical question-answering improvements vary with corpus and retrieval strategy.Source: §1 · p. 2
  • Heterogeneous medical RAG evaluation and expert-identified retrieval and evidence-selection failures motivate evaluating retrieved evidence separately from generated answers.Prior work covers clinical question answering and simulated oncology cases, while radiology-focused results vary across models and sources.Source: §1 · p. 2
  • Retrieval scores can depend on relevance annotations: a supporting chunk may be scored nonrelevant when only another chunk containing equivalent evidence is labeled.Overlapping segmentation can create such alternatives, and additional relevant passages may remain unannotated; overlap alone does not establish a fixed penalty.Source: §1 · p. 2
  • Chunk-based evaluation is sensitive to how evidence is partitioned and repeated across overlapping segments, so evidence-level relevance requires explicitly defined and audited alternatives.Benchmarks have represented support through spans, sentence-level facts, and provenance, while conventional ranking metrics may not capture answer-supporting information.Source: §1 · p. 2
  • The study defines canonical evidence spans independently of chunk identifiers, maps them to each segmentation, and audits newly recognized evidence while holding retrieval rankings fixed.It separately evaluates retrieval configurations and generation quality on a curated 18-document corpus and 18-query benchmark.Source: §1 · p. 2

2 Methods

The methods combine a scoped breast-lesion literature corpus with hybrid retrieval, evidence-to-chunk relevance mapping, and separate retrieval and generation evaluations. The benchmark uses focused questions and evaluates alternative evidence chunks under multiple chunking and retrieval configurations.

2 Methods

  • Retrieval ranks dense and BM25 results with reciprocal-rank fusion, reranks fused candidates with a cross-encoder, and returns top-ranked chunks to the answer generator.The overview describes corpus preparation, hybrid retrieval, response generation, and evaluation.Source: §2 · p. 3, Figure 1 · p. 3

2.2 Retrieval-Augmented Generation Pipeline

  • Dense retrieval and BM25 rankings are combined with reciprocal-rank fusion, then up to 20 fused candidates are reranked by a cross-encoder before the top three are returned.The ablation changes only the retrieval component across BM25, semantic, hybrid RRF, and hybrid RRF plus reranking.Source: §2.2 · p. 6, §2.2.2 · p. 7

2.2.1 Chunking

  • The selected segmentation uses 384-token chunks with a 77-token overlap after chunk-size and overlap sweeps.Chunks pack extracted text units rather than arbitrary character spans, and retain page ranges, contributing blocks, and section metadata.Source: §2.2.1 · p. 7

2.2.2 Embedding, reranking, and lexical retrieval

  • Dense and lexical retrieval use BGE-large embeddings and BM25 with a biomedical-aware tokenizer, respectively, before RRF fusion and cross-encoder reranking.Clinically meaningful lexical forms such as HER2+, BI-RADS, percentages, and comparison operators are preserved by tokenization.Source: §2.2.2 · p. 7

2.2.3 Answer generation

  • The generator receives the highest-ranked chunks in retrieval order, each capped at 3,500 characters, with source-identifying metadata.The selected implementation uses Qwen 3.6 (27B); it does not enforce a separate global token budget or project-level context-window check.Source: §2.2.3 · p. 8

2.3 Retrieval Evaluation

  • Canonical evidence spans are mapped to chunks per segmentation, and each query's accepted alternatives are evaluated separately using the maximum metric score before macro-averaging.The 18-query benchmark has one required evidence group per query and no complementary multi-chunk routes; retrieval-informed curation adds relevant chunks while rankings remain fixed.Source: §2.3 · pp. 8–9

2.4 Generation Evaluation

  • Generation context depth varies from one to five chunks while the ordered Top-5 retrieval results stay fixed, separating generation evaluation from retrieval changes.Generation is assessed with BLEU/ROUGE and DeepEval metrics, including faithfulness and answer relevancy, using an LLM judge.Source: §2.4 · p. 9

3 Results

The selected 384-token, 20%-overlap hybrid retriever with cross-encoder reranking found accepted evidence in the top three for 17 of 18 queries. Curation expanded accepted evidence annotations without changing rankings, while generation metrics varied across context depths and qualitative cases exposed both successful retrieval and a Top-3 miss.

3 Results

  • 17 of 18 queries had accepted evidence in the top three with the selected hybrid retriever and cross-encoder reranker (Recall@3 = 0.94).Ground-truth curation increased evidence spans from 25 to 37 without changing retrieval rankings.Source: §3 · p. 9

3.1 Segmentation

  • 384-token chunks achieved the highest overall retrieval performance: Recall@3 = 0.94, R-Precision = 0.77, MRR@3 = 0.89, and nDCG@3 = 0.90.Figure 2(A) compares chunk sizes from 128 to 1024 tokens at approximately 20% overlap.Source: §3.1 · p. 9, Figure 2 · p. 10

3.2 Retrieval Ablation

  • 0.94 was reached for Recall@3, R-Precision, MRR@3, and nDCG@3 after adding cross-encoder reranking to hybrid RRF.Table 4 compares four retrieval configurations against curated ground truth.Source: §3.2 · p. 10, Table 4 · p. 10

3.3 Effect of Ground-Truth Curation

  • 25 to 37 evidence spans and 36 to 51 accepted chunk targets followed ground-truth curation, with retrieval rankings unchanged.Hybrid RRF + Reranker retrieved accepted evidence within the top three for 17 of 18 queries (Recall@3 = 0.94).Source: §3.3 · p. 10

3.4 Generation Quality by Context Depth

  • 0.22 BLEU-4, 0.56 ROUGE-1, and 0.37 ROUGE-2 were highest at K = 5, while ROUGE-L peaked at K = 1 (0.51).Table 5(A) reports BLEU/ROUGE scores across K = 1 to K = 5.Source: §3.4 · p. 11, Table 5 · p. 11
  • 0.72 at K = 1 to 0.47 at K = 5 was the Contextual Relevancy change; no single context depth maximized all DeepEval metrics.Contextual Recall reached 0.94 at K = 2, K = 4, and K = 5; Faithfulness remained between 0.96 and 1.00.Source: §3.4 · p. 11, Table 5 · p. 11

3.5 Qualitative Retrieval Analysis

  • The radial-scar retrieval was accepted as successful, but its excerpt did not explicitly address sampling adequacy.The excerpt supports that routine excision is not required for radiologic–pathologic concordant lesions without atypia, but does not establish coverage of every qualifier in the question.Source: §3.5 · p. 12
  • The PASH rank-1 passage did not compare biopsy techniques; the required comparative evidence appeared at rank 7, outside the Top-3 cutoff.The query asked which biopsy technique is preferred for diagnosing pseudoangiomatous stromal hyperplasia.Source: §3.5 · p. 12

4 Discussion

The discussion shows that retrieval scores depend on both system configuration and ground-truth coverage, while benchmark success does not establish that retrieved evidence is clinically sufficient. The findings remain bounded by a small development benchmark and call for expert assessment.

4.1 Technical Development

  • 17 of 18 queries had accepted evidence in the top three with 384-token chunks, 20% overlap, and Hybrid RRF with cross-encoder reranking (Recall@3 = 0.94).Recall@3 records whether at least one accepted alternative appears in the top three, not the fraction of relevant chunks retrieved; the approach does not establish invariance to overlapping evidence.Source: §4.1 · p. 13
  • Ground-truth curation expanded evidence spans from 25 to 37 and accepted chunk targets from 36 to 51 without changing rankings, though auditing the selected configuration may favor it in later comparisons.The score changes reflect expanded annotation coverage rather than improved retrieval.Source: §4.1 · p. 13
  • Hybrid RRF raised R-Precision to 0.78 versus 0.61 for BM25 and 0.56 for semantic retrieval, while reranking raised Recall@3, R-Precision, MRR@3, and nDCG@3 to 0.94.In the size sweep, 384-token chunks had the highest MRR@3 (0.89) and nDCG@3 (0.90); 20% overlap led on three metrics, but performance was not monotonic across overlap settings.Source: §4.1 · p. 13
  • Generation metrics varied with context depth: Contextual Relevancy fell from 0.72 at K = 1 to 0.47 at K = 5, while no setting consistently maximized all answer-quality measures.Contextual Precision peaked at K = 5 (0.91), Contextual Recall reached 0.94 at K = 2, K = 4, and K = 5, and Faithfulness ranged from 0.96 to 1.00.Source: §4.1 · p. 13
  • The qualitative cases distinguish benchmark success from evidence sufficiency: retrieved passages could support an answer or address a technique yet omit sampling adequacy or the requested comparison.In Q09, accepted comparative evidence appeared only at rank seven; topical relevance and agreement with annotated targets may not cover every clinical qualifier.Source: §4.1 · p. 13
  • Interpretation is limited by 18 documents and 18 simple, direct queries, reuse of the benchmark for selection and evaluation, and generation assessment without human validation.A single query changes Recall@3 by approximately 0.06; generalization to independent questions, complex evidence needs, and conflicting clinical sources remains untested.Source: §4.1 · p. 13

4.2 Clinical Implications

  • Retrieving an accepted target does not establish clinical sufficiency when a passage omits necessary management conditions; expert assessment is needed to judge whether evidence preserves relevant conditions.A planned reader study will compare a breast imaging radiologist’s assessment of retrieved passages with automated ranking; clinical-workflow effectiveness remains unestablished.Source: §4.2 · p. 14

5 Conclusion and Future Work

The conclusion emphasizes that retrieval results reflect both system configuration and relevance-annotation coverage, while neither retrieval nor generation results establish clinical sufficiency. Future work prioritizes independent evaluation and expert review.

5 Conclusion and Future Work

  • On the 18-query development benchmark, hybrid RRF with cross-encoder reranking retrieved an accepted evidence alternative in the top three for 17 queries (Recall@3 = 0.94).Curation expanded evidence spans from 25 to 37 and accepted chunk targets from 36 to 51 without changing rankings; generation metrics varied across context depths without one setting consistently maximizing them all.Source: §5 · p. 14
  • The findings are limited to a small development benchmark and do not establish clinical evidence sufficiency or clinical effectiveness.Priorities include a blinded audit pooled across retrievers, independent questions without further tuning, and radiologist review of retrieved passages and answer faithfulness.Source: §5 · p. 14
  • Future technical work will compare fixed-size with semantic and section-aware segmentation, assess penalties for noisy chunks, and explore graph-based retrieval for relational and multi-hop evidence.The proposed segmentation comparisons particularly target evidence spanning paragraphs, tables, and quantitative results.Source: §5 · p. 14

A.1.1 Experimental Setup

The query-formulation sensitivity analysis tested seven variants on a selected index using curated ground truth and fixed retrieval settings; answer generation and LLM-based judging were disabled.

A.1.1 Experimental Setup

  • Seven query-formulation conditions were evaluated, including Short, Medium, Long, Canonical, Paraphrased, Yes/No, and WH variants, with 18 queries per condition.Medium and Canonical retained original queries; mean Short, Medium, and Long lengths were 11, 13, and 18 words.Source: §A.1.1 · p. 18
  • All 126 variants used the selected 384-token index with 77-token overlap and 757 chunks, curated ground truth, and final-ablation retrieval settings; generation and LLM judging were disabled.GPT-2 perplexity was reused for identical query texts, and analyses were descriptive.Source: §A.1.1 · p. 18

A.1.2 Results

The query-formulation results show descriptive differences across length and question-type conditions, but neither query length nor perplexity had a consistent monotonic association with retrieval performance.

A.1.2 Results

  • Medium-length queries led the length variants in R-Precision (0.89), MRR@3 (0.92), and nDCG@3 (0.91), while Yes/No questions exceeded WH questions on Recall@3 (0.89 versus 0.83), R-Precision (0.87 versus 0.84), and nDCG@3 (0.91 versus 0.85).WH questions had slightly higher MRR@3 (0.92 versus 0.91); these descriptive differences were not tested for statistical significance.Source: §A.1.2 · p. 18
  • Table S1 summarizes retrieval performance across seven query-formulation conditions, with ideal retrieval scores indicated toward 1 and lower perplexity preferred.Canonical and paraphrased queries had similar R-Precision values: 0.89 and 0.88, respectively.Source: Table S1 · p. 18, §A.1.2 · p. 18

Related papers

Loading 2610.31606v1…