Source-linked AI summary
RadMatch: Auditable Radiology Report Evaluation via Finding-Level Matching
Charles Corbière, Léo Machado, Aubin Charley, Baptiste Callard, Pierre Manceron, Corentin Dancette
TL;DR
Reliable radiology report evaluation must capture clinically meaningful findings and remain interpretable, but existing metrics often produce opaque scores or treat discrepancies uniformly. RadMatch decomposes comparison into structured finding extraction, clinical-equivalence matching, and significance-aware scoring. Across two expert benchmarks, it achieves the strongest clinical alignment, matching radiologist agreement on ReXVal and more than doubling the best prior metric on RadEvalExpert.
Problem
Existing radiology report metrics incompletely capture clinical quality, while LLM-based metrics remain opaque and difficult to interpret or audit.
Method
RadMatch extracts structured findings, matches candidate and reference entities, and scores matched and unmatched findings by clinical significance with seven attribute dimensions.
Results
Across two expert-annotated benchmarks, RadMatch is the most clinically aligned metric, matching radiologist agreement on ReXVal and reaching |τb| ≈0.58 versus 0.24 for CRIMSON on RadEvalExpert.
Takeaways & Limitations
The actionable-error count and persisted finding-level records make evaluation interpretable, auditable, and complemented by safety, attribute, and subset views.
Takeaways & Limitations
Evaluation is limited to chest X-ray benchmarks, the pipeline uses multiple stochastic LLM calls, and clinical use still requires final radiologist validation.
Abstract
from arXiv · showhide
As AI systems are increasingly used to draft radiology reports, reliably evaluating their clinical quality remains a critical challenge. Large language model (LLM)-based metrics are now the best-correlated with radiologist judgment, yet they output a single opaque score that neither a clinician nor a model builder can easily interpret or audit. We introduce RadMatch, a multi-stage, LLM-based metric that decomposes report comparison into a structured finding-level matching with significance-aware scoring and error characterization across seven clinical attribute dimensions (status, location, severity, morphology, certainty, longitudinal comparison, and measurement). The main score is the actionable-error count, both interpretable and auditable. Candidate findings are graded correct, partial, or incorrect, and unmatched findings are counted as missed or hallucinated. Triage and actionable safety recall/precision and per-subset views add complementary, deployment-oriented lenses. Across two expert benchmarks, RadMatch is the most clinically aligned metric, matching inter-radiologist agreement on ReXVal and more than doubling the best prior metric on the harder RadEvalExpert. Relying only on few-shot prompting, it is designed to extend to other modalities and anatomies. We will release RadMatch as open-source code with an interactive dashboard for inspecting results.
1 Introduction
Radiology report evaluation must distinguish clinically important errors rather than treat all wording discrepancies equally, while existing LLM metrics remain opaque. RadMatch addresses this by making report comparison finding-level, significance-aware, and auditable, with strong agreement against expert judgments.
- Traditional language-generation and clinical-concept metrics capture only fragments of clinical quality because they weight discrepancies equally and ignore clinical severity.
- LLM-based metrics correlate better with radiologists’ judgment, but their single quality score or error count is difficult to interpret or audit.
- RadMatch extracts atomic findings, matches candidate and reference findings, and applies significance-aware scoring to create traceable, finding-level error records.
- RadMatch characterizes each match across seven clinical attributes and reports actionable-error counts alongside triage and actionable safety precision and recall.
- |τb| ≈0.58 versus 0.24 for CRIMSON on RadEvalExpert, while ReXVal agreement matches agreement among radiologists themselves.
2 Related Work
Radiology report generation has advanced through increasingly structured and clinically grounded methods, but evaluation remains challenging as systems extend beyond chest X-ray. Existing metrics range from surface-text overlap to extracted clinical concepts, with limited sensitivity to clinical severity.
- Radiology report generation has progressed through image–language modeling, evidence grounding, retrieval, disease-aware alignment, self-critique, specialized decoding, and structured reporting.
- Extending report generation from radiographs to volumetric CT and MRI remains markedly harder to produce and evaluate.
- Lexical metrics reward surface or contextual overlap and can favor fluent paraphrases regardless of factual correctness.
- Clinical-concept metrics score extracted entities, relations, or labels but weight every discrepancy equally and are brittle.
3 RadMatch Evaluation Framework
RadMatch evaluates a candidate report against a reference by extracting structured findings, matching clinically equivalent entities, and scoring matched or unmatched findings by clinical consequence. Its actionable-error score and complementary safety and attribute views preserve interpretable records for post-hoc inspection.
- RadMatch estimates clinically actionable errors through three stages: structured finding extraction, clinical-equivalence matching, and consequence-based scoring.
- Structured Finding Extraction: Each finding record includes status, location, severity, morphology, certainty, longitudinal comparison, measurements, and a clinical-significance tier.
- Structured Finding Extraction: Significance tiers range from critical to routine and are modulated by clinical context, so the same observation can receive different urgency levels.
- Finding-Level Matching: Matching groups findings describing the same entity using direct or many-to-many aggregate matches, while unmatched findings become omissions or spurious findings.
- Finding-Level Matching: Status conflicts remain matched as one entity-level error, whereas anatomical mismatches produce an omission and a spurious finding; generic boilerplate is excluded from safety scoring.
- Scoring: Matches are labeled COR, PAR, or INC, while unmatched findings are labeled MIS or SPU; error type is tracked separately from significance magnitude.
- Metrics: The actionable-error count includes errors at critical, urgent, or notable tiers, excluding routine findings.
- Metrics: Safety precision and recall, attribute-level breakdowns, and subset views complement the count, with findings, matches, and verdicts retained for inspection.
4 Experiments
Across two expert-annotated chest-X-ray benchmarks, RadMatch consistently aligned with radiologist error counts while supporting finding-level auditing, safety views, and robustness across judge models. Its strongest results matched inter-radiologist agreement on ReXVal and substantially exceeded prior metrics on RadEvalExpert.
- Agreement with Radiologists: RadMatch reaches |τb| = 0.79 on ReXVal and 0.58 on RadEvalExpert with Opus 4.8, outperforming every other metric on both benchmarks.The ReXVal result matches inter-radiologist agreement, while the RadEvalExpert result more than doubles the best prior metric.
- Agreement with Radiologists: 0.58 on findings and 0.44 on impressions, with Opus 4.8 outperforming prior metrics on both RadEvalExpert sections.Agreement is lower on impressions because those sentences are more interpretive and combine several observations into one diagnostic statement.
- Robustness to LLM Capability: |τb| ∈[0.74, 0.79] on ReXVal and [0.53, 0.58] on RadEvalExpert for the strongest judges, showing stability across a broad capability band.Agreement degrades modestly for mid-tier judges, remains usable for GPT-5.4-mini, and collapses only for MedGemma 1.5 4B.
- Beyond a Single Score: Diagnostic and Safety Views: Single-call count and enumerate baselines match RadMatch on raw correlation but provide no explicit finding-level matching, significance tier, error kind, or auditable persisted record.The enumerate baseline lists errors as unstructured free text, preventing triage/actionable safety and per-subset views.
- Beyond a Single Score: Diagnostic and Safety Views: Triage recall is 0.48 on ReXVal and 0.28 on RadEvalExpert, while omissions and hallucinations dominate ReXVal errors and abnormal or longitudinal-comparison findings carry most actionable errors.These per-report and per-subset diagnostics expose safety-relevant failure patterns hidden by aggregate scores.
- Qualitative Analysis: RadMatch resolves fluent or high-scoring drafts into traceable finding-level errors, including status inversions, hallucinations, longitudinal inversions, and severity overcalls.In one pair, it identifies three actionable errors matching radiologists’ count; in another, it flags two urgent safety-relevant errors despite strong scalar scores from prior metrics.
5 Discussion and Limitations
RadMatch reframes report evaluation as an auditable finding-level record, prioritizing interpretable and actionable error analysis over opaque correlation alone. Its evaluation remains limited in modality and still requires expert oversight.
- Discussion: RadMatch decomposes reports into findings, matches them, and scores clinical significance, exposing missed, hallucinated, and mis-described findings by attribute dimension.Every penalty traces to a candidate–reference finding pair, making scores inspectable and contestable.
- Discussion: Once capable judges saturate correlation, RadMatch’s distinctive value shifts toward interpretability, actionability, and auditability.A single-call LLM error count can match RadMatch on agreement while providing no finding-level structure.
- Limitations: RadMatch’s evaluation is limited to chest X-ray because the two expert benchmarks lack radiologist error-count annotations for CT or MRI.Few-shot prompting supports potential extension, but the reported evaluation does not establish performance beyond chest X-ray.
- Limitations: Clinical deployment still requires final validation by radiologist judgment, because RadMatch complements rather than replaces expert oversight.The metric is intended to help developers evaluate and iterate on report-generation systems.
6 Conclusion
RadMatch evaluates radiology reports through significance-aware finding-level matching, converting opaque scores into auditable records. It tracks radiologist judgment closely across two expert benchmarks and can run with on-premise open judges.
- Conclusion: RadMatch uses finding-level matching with significance-aware scoring to turn opaque report evaluation into an auditable record.The conclusion presents this as the paper’s central contribution.
- Conclusion: Across two expert benchmarks, RadMatch tracks radiologist judgment as closely as any evaluated metric.The conclusion also notes operation with on-premise open judges.
A Implementation Details
RadMatch combines multiple LLM extraction, matching, and attribute-grading calls with deterministic comparison logic. The implementation uses structured clinical thresholds while deferring context-dependent judgments to the LLM attribute judge.
- A Implementation Details: Each report pair uses four LLM calls: independent finding extraction, batched matching, and batched free-text attribute grading.An optional indication-extraction call supplies clinical context when a study indication is available.
- A Implementation Details: The deterministic tail handles structured comparators and message-understanding typing after the LLM stages.The supplied implementation passage identifies the remaining work as deterministic, while the comparator passage specifies its clinical rules.
- Structured comparators: Status inversions are major errors, while longitudinal differences are major across benign-to-active trajectories and minor within a trajectory.These rules encode clinically motivated distinctions in structured dimensions.
- Structured comparators: Measurement errors are thresholded by category, including size differences above 20% with a 2 mm floor, ratio differences above 30%, count changes, and attenuation differences above 20 HU.Reference omissions are major and candidate-added measurements are minor; context-dependent cutoffs go to the LLM judge.
B Additional Agreement Breakdowns
Additional breakdowns show that RadMatch remains competitive across candidate generators and performs better on findings than impressions. Endpoint analysis highlights strengths on false prediction and weaker performance on inarticulate reports.
- Per-candidate breakdowns: RadMatch matches or exceeds every baseline across ReXVal candidate generators and remains stable across generators.Table 4 reports Kendall’s τb and Pearson’s r with 95% confidence intervals for four generator-specific subsets.
- Per-section breakdowns: On RadEvalExpert, RadMatch reaches |τb|=0.58 on findings versus 0.44 on impressions.The table caption specifies separate Findings and Impression scopes with K=3 candidates per study.
- Per-endpoint breakdowns: RadMatch is strongest on false prediction, competitive on omission, and weakest on the non-clinical inarticulate category by design.Label-based metrics lead on omission in the reported endpoint comparison.
- Per-endpoint breakdowns: For the reported endpoints, RadMatch (Opus 4.8) is aligned for severity, spurious comparison, and omission of change, but not for inarticulate reports.The supplied entries report −0.131 for severity, −0.192 for spurious comparison, −0.114 for omission of change, and −0.062 for inarticulate reports.
C Cost and Latency
RadMatch’s four-call configuration has substantial prompt-driven token use, with cost and latency varying by judge model and reasoning mode. The reported measurements separate per-pair resource requirements from benchmark totals and stage-level timing.
- C Cost and Latency: With Opus 4.8, each report pair costs approximately $0.06, totaling $12 for ReXVal and $35 for RadEvalExpert.These figures are reported for the chest-X-ray few-shot, non-reasoning configuration.
- C Cost and Latency: GPT-5.4 costs approximately $0.02 per pair, totaling $4 for ReXVal and $13 for RadEvalExpert, while producing a small drop in agreement.The paper characterizes GPT-5.4 as approximately three times cheaper than Opus 4.8.
- C Cost and Latency: ≈17–33k input tokens and ≈0.3–1.3k output tokens are used per report pair across RadMatch’s four calls.The input includes shared system prompts and few-shot exemplars.
- C Cost and Latency: End-to-end latency is measured over a fixed 25-pair ReXVal subset with one serial worker, covering extraction, matching, and scoring.This setup targets intrinsic per-report cost rather than provider- or hardware-specific throughput.
- C Cost and Latency: Non-reasoning judges take 5–11 seconds per pair, whereas reasoning judges are slower because latency tracks judge thinking effort.API judges use hosted endpoints; local judges run on 2×RTX 5090 via vLLM.
- C Cost and Latency: Table 7 decomposes per-report latency by pipeline stage and reports lower latency as better.The optional indication-extraction call is absent because ReXVal reports carry no indication.
D LLM Prompts
The RadMatch prompts specify a structured extraction process for atomic findings, including status, significance, comparison, and measurements, with rules for splitting, merging, exclusions, and attribute grading. The appendix also documents the four runtime calls and their schema-constrained outputs.
- D LLM Prompts: RadMatch documents four runtime LLM calls: optional study-indication extraction, finding extraction, finding-level matching, and attribute grading.Per-stage few-shot exemplars are supplied separately, and schema-constrained JSON output is enforced.
- D.1 Study-Indication Extraction (optional preprocessor): When available, the study indication is extracted first and injected as context into the three downstream prompts.This preprocessor is optional.
- D.2 Finding Extraction: Finding extraction independently converts each reference and candidate report into atomic, single-sentence observations with clinical annotations.The extracted fields include text, clinical status, clinical significance, comparison, and measurements.
- D.2 Finding Extraction: Examination-quality hedges are omitted, whereas statements describing a real anatomic state remain findings even when phrased as “not seen” or “not visualized.”The extraction criteria emphasize completeness, accuracy, atomicity, and consistency.
- D.2 Finding Extraction: Splitting rules separate distinct organs, negated conditions, pathologies, abnormalities, and sequential observations into separate findings.Examples include separating liver and spleen findings, shared negations, and multiple conditions or pathologies.
- D.2 Finding Extraction: A single observation remains combined when descriptors, qualifiers, or measurements describe the same finding, while high-confidence same-observation sentences may be merged.Merging requires the same anatomy, pathology entity, and compatible clinical status; uncertain cases remain separate.
- D.2 Finding Extraction: Clinical status uses normal or abnormal, with uncertain pathology treated as abnormal and explicit absence, resolution, patency, or normal size treated as normal.The default for uncertainty is abnormal.
- D.2 Finding Extraction: Clinical significance is recorded independently of status across critical, urgent, notable, and routine tiers, including context-dependent significance for normal findings.For example, the absence of pneumothorax can be critical in a trauma rule-out scenario.