Source-linked AI summary
AtomiMed: Hierarchical Atomic Fact-Checking for Universal Clinical-Aware Medical Report Evaluation
Yuan Wang, Wanxing Chang, Songtao Jiang, Shujian Gao, Xiaotian Zhang, Ruifeng Yuan, Weiwei Cao, Bowen Shi, Ling Zhang, Zuozhu Liu, Jianpeng Zhang
TL;DR
Medical report evaluation metrics often miss clinically critical errors and lack robust coverage across imaging modalities. AtomiMed decomposes reports into hierarchical Atomic Clinical Facts and cross-verifies them, achieving τ = 0.42 and MAE = 0.281 on MRI versus GREEN’s τ = 0.18 and MAE = 0.379.
Problem
Existing metrics lack simultaneous clinical factuality, fine-grained attribute assessment, modality universality, and reliable evaluation of generated medical reports.
Method
AtomiMed decomposes reports into Disease- and Attribute-level Atomic Clinical Facts and bidirectionally cross-verifies them through an agentic loop.
Results
τ = 0.42 and MAE = 0.281 on MRI, compared with GREEN’s τ = 0.18 and MAE = 0.379.
Takeaways & Limitations
AtomiMed provides meaningful cross-modal radiologist correlation with granular per-finding audit traces for disease- and attribute-level deficiencies.
Takeaways & Limitations
Future work must reduce inference cost, extend the attribute hierarchy to longitudinal comparisons, and broaden benchmark coverage to additional specialties.
Abstract
from arXiv · showhide
Traditional metrics for Medical Report Generation (MRG) predominantly rely on surface-level n-gram overlap, which fails to capture clinical factual accuracy and often overlooks catastrophic diagnostic errors. We address this fundamental limitation by proposing \textbf{AtomiMed}, a universal, modality-agnostic evaluation framework that decomposes complex medical narratives into a standardized, multi-level hierarchy of Atomic Clinical Facts, encompassing Disease-level entities and Attribute-level descriptors, including location, morphology, and severity. By implementing an Agentic Cross-Verification loop between ground-truth and predicted reports, AtomiMed simulates a multi-radiologist peer-review process to verify clinical consistency, thus enabling the decoupled assessment of diagnostic detection and descriptive accuracy. To facilitate standardized evaluation, we introduce \textbf{MRGEvalKit}, an open-source toolkit for automated hierarchical extraction, and curate \textbf{OmniMRG-Bench}, a comprehensive multi-modal benchmark covering X-ray, CT, MRI, and Ultrasound. Extensive experiments on multiple expert-annotated reader studies demonstrate that AtomiMed achieves significantly higher correlation with human radiologist judgment compared to traditional and model-based metrics. Our code are release at https://github.com/Venn2336/MRGEvalkit
1 Introduction
AtomiMed targets a core weakness in medical report evaluation: surface overlap and modality-specific metrics can miss clinically dangerous errors. It instead decomposes reports into atomic clinical facts and verifies them bidirectionally.
- Evaluation challenge: Medical report metrics must detect clinically critical errors such as missing pneumothorax or inverted laterality.Lexical metrics can assign near-identical scores to “no pleural effusion” and “pleural effusion.”
- Core idea: AtomiMed represents reports as disease-level presence claims plus attribute-level descriptors such as location, severity, and morphology.
- Core idea: Agentic Cross-Verification bidirectionally checks ground-truth and predicted reports, separating diagnostic detection from descriptive accuracy.The framework produces an aggregated scalar score and question-level audit traces for error attribution.
- Contributions: AtomiMed extracts hierarchical Disease-level and Attribute-level QA pairs and computes precision, recall, and F1 at both levels.
- Contributions: OmniMRG-Bench spans X-ray, CT, MRI, and Ultrasound with expert radiologist annotations, while MRGEvalKit supports reproducible hierarchical scoring.
2 Method
The method converts reports into hierarchical atomic facts, then uses an evidence-reading agent to cross-verify clinical consistency between reference and generated reports. This enables modality-universal scoring with traceable disease- and attribute-level errors.
- Atomic decomposition: AtomiMed decomposes each report into Disease-level and Attribute-level QA within a two-level Atomic Clinical Fact hierarchy.Attribute facets include location, size, morphology, severity, quantity, and temporal change.
- Atomic decomposition: The pipeline constrains the language model to emit valid JSON through a fixed prompt template and robust parsing pipeline.
- Cross-verification: Agentic Cross-Verification uses each report as evidence for questions generated from the other report.The loop compares a reference report Rgt and generated report Rinf in both directions.
- Cross-verification: Disease-level recall queries ground-truth questions against the generated report, while precision queries generated-report questions against ground truth.
- Scoring: Attribute verification is conditioned on disease-level agreement, so only correctly aligned findings contribute attribute scores.
3 Experiments and Results
AtomiMed is evaluated against expert judgments across multimodal benchmarks and preference studies. Its hierarchical scoring delivers strong agreement while exposing errors that holistic metrics conceal.
- Evaluation framework: AtomiMed separates diagnostic detection from descriptive accuracy through disease-level and attribute-level scoring.Scores are traceable to mismatched questions, enabling interpretable error attribution.
- Benchmark: 178K+ expert-verified hierarchical ACF pairs cover 9 anatomical systems, 6 attribute categories, and four imaging modalities.OmniMRG-Bench spans X-ray, CT, MRI, and Ultrasound.
- Radiologist correlation: On ReXVal, AtomiMed reaches Spearman’s ρ 0.806 versus GREEN (0.798), aligning closely with radiologist error judgments.The framework captures omissions and laterality shifts that radiologists penalize heavily.
- Pairwise preference: 95.71% ACC, τ = 0.9807, and MAE 0.0214 on X-ray outperform GREEN (MAE 0.1857, ACC 63.57%) in pairwise preference matching.These results indicate substantially closer agreement with radiologist rankings.
- Cross-modal robustness: AtomiMed reaches 84.33% ACC on CT and 49.86% on Ultrasound, while GREEN scores 33.83% and RaTEScore 35.00% on the latter.GREEN’s Kendall’s τ falls from 0.6481 on X-ray to 0.3283 on CT and 0.1513 on MRI.
- Error analysis: Granular analysis finds stronger morphology performance but sharp weaknesses in severity and size, alongside uneven anatomical coverage.Severity scores range from 1.3–5.9 and size scores from 1.0–6.9; respiratory performance dominates while digestive, reproductive, and urinary systems lag.
4 Conclusion
AtomiMed offers a modality-agnostic framework for evaluating medical report generation through hierarchical Atomic Clinical Facts and bidirectional verification. Future work targets efficiency, longitudinal comparisons, and broader clinical coverage.
- AtomiMed decomposes reports into Disease-level and Attribute-level Atomic Clinical Facts and verifies consistency through a bidirectional Agentic Cross-Verification loop.
- Fig. 4 uses heatmaps to show category-level performance across medical attributes and disease-level performance across anatomical systems.Higher scores indicate better alignment with human-verified atomic facts.
- Future work will develop distilled backbone models, extend attributes to longitudinal imaging comparisons, and broaden coverage to additional clinical specialties.These directions address inference cost, temporal comparison, and scope limitations.