Source-linked AI summary
When Retrieval Hurts: Measuring and Explaining Retrieval-Induced Hallucination in Chest X-ray Report Generation
Emmanuel Idoko, Abdusshakur Olabisi, Shiloh Oni, Adesola Josiah
TL;DR
Chest X-ray reporting requires findings grounded in the target image, but retrieved reports can supply unsupported clinical content. The paper measures retrieval-induced hallucination and examines its mechanism and gating implications. It finds that embedding similarity reflects anatomy and acquisition more than disease presence, complicating similarity-based gating.
Problem
Chest X-ray reporting needs image-grounded findings, yet retrieval can introduce clinical content absent from the target image.
Source: §1 · p. 2Method
The paper measures retrieval-induced hallucination separately from chance overlap, examines the retriever’s embedding geometry, and derives an empirical constraint on relevance gating.
Source: §1 · p. 2Results
Image-embedding similarity is dominated by anatomy and acquisition, and a normal chest X-ray retrieves an abnormal precedent in 24 of 28 cases, making similarity-based gating ineffective.
Source: §8 · p. 9Takeaways & Limitations
Retrieval-augmented clinical systems should be evaluated under retrieval failure, not only retrieval success.
Source: §8 · p. 9The gating analysis uses ground-truth labels unavailable at inference, and the authors describe the work as a pilot rather than a final benchmark.
Source: §5.4 · p. 7, §6 · p. 7
Abstract
from arXiv · showhide
Retrieval-augmented generation is an attractive way to improve chest X-ray reporting, because reports from similar prior studies supply clinical context a general vision-language model lacks. We show the same mechanism is a reliable source of clinical error. Over 100 MIMIC-CXR studies with a frozen LLaVA-1.5 generator and BioMedCLIP retrieval, CheXbert clinical F1 doubles under relevant retrieval (0.201 to 0.402) and collapses to 0.043 under clinically mismatched retrieval, a fifth of the image-only score; the retrieval-induced hallucination rate, counting only unsupported findings traceable to retrieved evidence, rises from 0.00 to 0.76 and 0.98. To show this is not an artefact of evidence-set coverage, we introduce a coincidental-overlap control that scores image-only generations against evidence they never saw, placing the chance base rate at 0.18, four to five times below the observed rates. The generator does not merely acquire findings, it transcribes text: 95% of reports produced under relevant retrieval contain an eight-word span occurring verbatim in the retrieved evidence but absent from the reference, against 0% without retrieval. We then explain the mechanism: a normal chest X-ray retrieves at least one abnormal precedent in 24 of 28 cases, because medical image-embedding similarity is dominated by anatomy and acquisition rather than by the presence of disease. This has a direct design consequence. Retrieval similarity does not predict harm (RIH rates 0.80/0.80/0.60/0.84 across similarity quartiles), so relevance gates conditioned on embedding similarity cannot work; gating on predicted pathology agreement removes 61% of unsupported evidence at no cost to useful coverage. We argue that retrieval-augmented clinical systems must be evaluated under retrieval failure, not only under retrieval success.
1 Introduction
Chest X-ray reports must ground findings in the target image, yet conflicting visual, language, and retrieved signals can produce unsupported clinical findings. The paper measures retrieval-induced hallucination, investigates its mechanism, and derives a constraint for relevance gates.
1 Introduction
- Chest X-ray report generation is high-stakes because generated findings must be grounded in the target image despite conflicting visual, language, and external-context signals.Conflicting signals may yield clinically meaningful findings unsupported by the image.Source: §1 · p. 2
- Retrieval-induced hallucination is a generated finding present in retrieved reports but absent from the target reference, isolating a measurable retrieval-specific form of hallucination.The paper distinguishes this narrower case from hallucination in general.Source: §1 · p. 2
- The paper measures retrieval-induced hallucination against chance overlap, identifies a mechanism in retriever embedding geometry, and derives an empirical constraint on relevance gates.Retrieval-Guided Cross-Attention is specified but not claimed to work.Source: §1 · p. 2
2 Related Work
Clinical multimodal models and retrieval-augmented generation provide capabilities relevant to medical reporting, but prompt-concatenated retrieved findings are not checked against the target image. This paper frames retrieval as a distinct source of image-unsupported findings.
2 Related Work
- Medical multimodal systems include LLaVA-Med, Med-Flamingo [10], CheXagent [2], and BioMedCLIP [14], but clinical competence does not solve image-grounded faithfulness.BioMedCLIP supplies the embeddings used for retrieval in this paper.Source: §2 · p. 2
- Retrieval-augmented generation conditions output on retrieved documents [6], while prompt concatenation supplies findings without checking their support in the target image.The approach has also been scaled to trillion-token corpora.Source: §2 · p. 2
- Medical VLM hallucination is commonly framed as findings unsupported by the image [11], while retrieval introduces a distinct source of such findings.Source: §2 · p. 2
3 Method
The method defines retrieval-induced hallucination by comparing generated findings with retrieved reports and the reference, then evaluates no-retrieval, prompt-retrieval, and mismatched-retrieval conditions. It also specifies RGCA, whose report-alignment signal is acknowledged to be a poor predictor of harm.
3.1 Problem Definition
- A retrieval-induced hallucination occurs when a clinically meaningful finding appears in both the generated report and retrieved evidence but not the reference report.Source: §3.1 · p. 3
3.2 Baseline Pipeline
- The baseline compares image-only generation, prompt retrieval with top-k reports, and deliberately mismatched retrieval to stress-test copying unsupported findings.Source: §3.2 · p. 3
3.3 Retrieval-Guided Cross-Attention
- RGCA gates retrieved reports using image-report alignment and feeds gated report tokens into cross-attention, but the paper says alignment poorly predicts harm and the formulation needs revision.The intended role is to guide visual grounding rather than supply text for the decoder to copy.Source: §3.3 · p. 3
4 Experimental Setup
The experiments use study-level image-embedding retrieval from MIMIC-CXR and evaluate a frozen LLaVA-1.5 generator with CheXbert-based labels. The patient-separated setup and reported compute make the pilot reproducible within its stated scope.
4.1 Dataset and Retrieval Setup
- The study uses a MIMIC-CXR/MIMIC-CXR-JPG study-level index, retrieving reports by image-text embedding similarity for each target image.Source: §4.1 · p. 3
4.2 Compute, splits, and runtime
- The 400-study retrieval pool and 100-study evaluation pool are patient-disjoint, use frontal PA/AP views, and exclude self- and same-patient retrieval.Index construction took under ten minutes on a laptop GPU; 300 generations took 20–40 minutes on one NVIDIA T4, and an independent rerun reproduced neighbours exactly.Source: §4.2 · p. 4
4.3 Generator and Evaluation
- The generator is LLaVA-1.5, and CheXbert labels generated, reference, and retrieved reports using the same condition vocabulary, with uncertain mentions treated as negative.Automatic labels do not settle individual cases; manual review remains the arbiter.Source: §4.3 · p. 4
5 Results
The results show that retrieval quality is asymmetric for normal and abnormal targets, and that retrieved evidence can improve clinical F1 while also supplying unsupported findings and copied text. Embedding similarity does not track harm, while pathology-agreement gating can reduce unsupported evidence without reducing coverage at a permissive threshold.
5.1 Retrieval quality
- Normal targets retrieve at least one abnormal precedent in 24 of 28 cases, despite retrieval sharing a target pathology in 55% of cases overall.Overall pathology coverage is 60%, while precision is 0.272.Source: §5.1 · p. 4, Table 1 · p. 5
5.2 Generation under the three retrieval conditions
- Across 100 evaluation studies, Clinical F1 doubled from 0.201 to 0.402 with relevant retrieved reports, but collapsed to 0.043 when retrieved reports were clinically mismatched, roughly one-fifth of image-only score.Source: Table 2 · p. 5, §5.2 · p. 5
- The accidental-overlap base rate is 0.18, versus observed RIH rates of 0.76 with relevant retrieval and 0.98 with mismatch.The attributable excesses are +0.58 and +0.80, respectively.Source: §5.2 · p. 5
- 95% of relevant-retrieval reports contain an eight-word span copied verbatim from retrieved evidence but absent from the reference, versus none without retrieval.The median longest verbatim run is 17 words, and the maximum is 70.Source: §5.2 · p. 6
5.3 Retrieval similarity does not predict harm
- RIH rates across similarity quartiles are 0.80, 0.80, 0.60, and 0.84, with no monotone trend and high harm in the most-similar quartile.Top-1 similarities range from 0.860 to 0.979, compressing the score range.Source: §5.3 · p. 6
5.4 What the gate should condition on
- At matched selectivity, cosine top-1 leaves 0.99 unsupported findings available at 0.33 coverage, while label agreement at J ≥0.4 leaves 0.35 at 0.47 coverage.The label-agreement condition retains 0.92 reports versus roughly one for cosine top-1.Source: §5.4 · p. 6, Table 3 · p. 7
- At J ≥0.2, label-agreement gating removes 61% of unsupported evidence (2.18 →0.84) while coverage remains 0.60.The table compares gating signals using unsupported findings, coverage, and normal-case protection.Source: §5.4 · p. 7, Table 3 · p. 7
- The gate comparison is an oracle using ground-truth labels unavailable at inference; substituting image-only reports yields coverage of 0.16.The image-only report has an omission rate of 0.86.Source: §5.4 · p. 7
6 Limitations
The findings are a pilot, and their scope is limited by the small evaluation and retrieval pool, the general-purpose generator, and limits on attribution and gating. RGCA is specified but not implemented or evaluated.
6 Limitations
- The evaluation covers 100 studies from a 500-study MIMIC-CXR subset, and its 400-study retrieval pool may make retrieval quality pessimistic relative to a full-corpus index.The authors characterize this as a pilot rather than a full benchmark.Source: §6 · p. 7
- Conclusions concern how a general VLM integrates retrieved evidence, not the attainable quality of automated reporting.The LLaVA checkpoint is not radiology-specialized, and its omission rate is 0.74–0.80 across conditions.Source: §6 · pp. 7–8
- The RIH rate cannot distinguish findings copied from evidence from independently generated findings that happen to appear there; individual attribution therefore rests on manual review.The coincidental-overlap control bounds this ambiguity at the population level.Source: §6 · p. 8
- Retrieval similarity does not separate harmful from harmless retrievals at n = 100, making the gating formulation unsupported as specified.The authors say the design should be revised before implementation rather than assumed to work.Source: §6 · p. 8
- RGCA is specified but not implemented or evaluated, and the authors make no claim that it reduces hallucination.Source: §6 · p. 8
7 Impact in African and other low-resource clinical contexts
The paper cautions that retrieval augmentation can introduce unsupported findings even under ordinary retrieval, with particular concern for smaller local corpora and normal studies. It argues that retrieval-corpus composition, especially the balance of normal studies, matters for patient-facing behavior.
7 Impact in African and other low-resource clinical contexts
- 76% of studies acquired an unsupported finding traceable to retrieved evidence under ordinary retrieval, without corruption.The authors say reasonable retrieval alone does not establish safety.Source: §7 · p. 8
- A 400-study pool sent 24 of 28 normal targets to an abnormal precedent, and a smaller or less diverse local corpus makes an appropriate neighbour less likely.The authors identify resource-constrained deployments with only a few hundred prior reports as being in this regime.Source: §7 · p. 8
- Retrieval inflated findings on normal studies while reducing omission only modestly, from 0.80 without retrieval to 0.74 with retrieval.The authors connect this error asymmetry to unnecessary imaging and referral where specialist review is limited.Source: §7 · p. 8
- Retrieval corpora should be treated as clinical infrastructure whose composition, particularly its balance of normal studies, materially affects patient-facing behavior.Source: §7 · p. 8
8 Conclusion
The conclusion attributes retrieval-induced hallucination to the retriever’s embedding geometry: similarity reflects anatomy and acquisition, so normal images often retrieve abnormal precedents that the generator reproduces. It presents predicted pathology agreement, rather than embedding similarity, as a constraint for a corrected gate, while making no claim that RGCA reduces hallucination.
8 Conclusion
- Image-embedding similarity is dominated by anatomy and acquisition, and a normal chest X-ray retrieves an abnormal precedent in 24 of 28 cases that the generator often reproduces verbatim.Source: §8 · p. 9
- A gate conditioned on retrieval similarity cannot work; gating on predicted pathology agreement is the constraint a corrected RGCA must satisfy.Source: §8 · p. 9
- RGCA is specified but not implemented, and the authors do not claim it reduces hallucination.The conclusion calls for evaluating retrieval-augmented clinical systems under retrieval failure, not only retrieval success.Source: §8 · pp. 8–9