AtomiMed: Hierarchical Atomic Fact-Checking for Universal Clinical-Aware Medical Report Evaluation

1 Introduction

• Clinically critical misses such as omitted pneumothorax or reversed laterality can create false confidence and risk patient safety; evaluation must be factually grounded, structure-aware, and robust across imaging modalities.Source: §1 · p. 2
• Lexical metrics miss semantic reversals (BLEU, ROUGE, METEOR); structure-based metrics lack modality breadth, while GREEN lacks per-finding audit traces and incurs substantial inference costs.No existing metric simultaneously provides modality universality, fine-grained attribute correctness, and interpretable error attribution.Source: §1 · p. 2
• AtomiMed decomposes reports into Disease-level and Attribute-level Atomic Clinical Facts, then bidirectionally cross-verifies reports to separately assess diagnostic detection and descriptive accuracy with question-level audit traces.It computes precision, recall, and F1 at both hierarchy levels.Source: §1 · pp. 2–3
• Across four radiologist-annotated benchmarks, AtomiMed achieves significantly higher correlation with radiologist judgment than prior metrics, including GREEN, while exposing granular per-finding errors.The experiments cover multiple modalities and report systematic disease- and attribute-level deficiencies invisible to prior metrics.Source: §1 · pp. 2–3
• OmniMRG-Bench spans X-ray, CT, MRI, and Ultrasound with standardized expert radiologist annotations, alongside MRGEvalKit, an open-source toolkit for reproducible hierarchical atomic scoring.Source: §1 · p. 3

2.1 Atomic Decomposition

• AtomiMed prompts an instruction-tuned LLM to extract binary disease-level findings and location, size, morphology, severity, quantity, and temporal-change attributes as structured atomic facts.A fixed prompt template and robust parser constrain the model to emit valid JSON.Source: §2.1 · pp. 3–4

2.2 Agentic Cross-Validated Assessment

• A bidirectional cross-verification loop asks ground-truth and generated-report disease questions against each other, then scores attributes only for findings aligned in both directions.Disease names are matched to attribute keys using fuzzy string similarity (θ = 0.8); when both reports contain no disease statements, scores are set to 1.Source: §2.2 · pp. 4–5, Figure 1 · p. 4
• Equal-weight aggregation separates diagnostic detection from descriptive accuracy, with each score traceable to a mismatched question for interpretable error attribution.Source: §2.2 · p. 5

2.3 OmniMRG-Bench

• OmniMRG-Bench spans X-ray, CT, MRI, and Ultrasound across 9 anatomical systems and 6 attribute categories, with over 178K radiologist-verified hierarchical QA pairs.Its attribute pairs are dominated by location (∼62.9K) and size (∼31.5K) descriptors; reports come from public datasets and de-identified clinical archives.Source: Figure 2 · p. 5, §2.3 · p. 5

3.1 Experiment Settings

• AtomiMed is evaluated across two complementary experimental paradigms.The supplied passage does not specify the paradigms or complete the purpose of the evaluation.Source: §3.1 · p. 5

Evaluation Benchmarks and Paradigms.

• Four expert-annotated benchmarks—ReXVal, ReFiSco-v0, RadEvalX, and RaTE-Eval—support metric–radiologist error-count correlation evaluation.They provide radiologist error counts, line-level severity annotations, or typed error annotations; RaTE-Eval is derived from MIMIC-IV and Radiopaedia.Source: Evaluation Benchmarks and Paradigms · p. 6, Table 2 · p. 6
• The dimensionless pairwise paradigm compares 10 state-of-the-art models on 20 cases each across X-ray, CT, MRI, and ultrasound against human expert judgment.Each case produces a 10×10 model-comparison matrix for validation against human judgment.Source: Evaluation Benchmarks and Paradigms · p. 6

Baselines.

• The baseline suite includes lexical metrics BLEU-1/4, ROUGE-L, METEOR, and CIDEr for n-gram overlap.Source: Baselines · p. 6
• BERTScore serves as the embedding-based baseline for semantic similarity via contextual embeddings.Source: Baselines · p. 6
• Medical-specialized baselines are F1 RadGraph [6], SembScore, and graph-based RaTEScore.Source: Baselines · p. 6
• GREEN is the LLM-as-a-Judge baseline for holistic clinical validation.Source: Baselines · p. 6

Implementation and Evaluation Protocol.

• AtomiMed uses Qwen3-235B-A22B for atomic decomposition and cross-verification, with decoding temperature T = 0 and fuzzy attribute matching threshold θ = 0.8.Source: Implementation and Evaluation Protocol · pp. 6–7
• Correlation with radiologist error counts is assessed using Kendall’s τ and Spearman’s ρ.Source: Implementation and Evaluation Protocol · p. 7
• A board-certified radiologist reviewed 10 models’ outputs across 80 sampled cases to create the Human Preference Gold Standard, evaluated by MAE, ACC, and Kendall’s τ.Source: Implementation and Evaluation Protocol · p. 7

Correlation with Radiologist Judgments.

• 0.806 Spearman’s ρ on ReXVal exceeds GREEN’s 0.798 and traditional NLP metrics, supporting AtomiMed’s strong alignment with expert clinical judgment.The passage attributes this alignment to detection of errors radiologists penalize, including omissions and laterality shifts.Source: Correlation with Radiologist Judgments · p. 7
• On RaTE-Eval, AtomiMed maintains comparable performance while providing fine-grained, interpretable audit traces for each clinical finding, unlike holistic LLM judges.Table 3’s stated scope is dimensionless pairwise-preference consistency across imaging modalities, measured with MAE, Acc, and Kendall’s τ against standard NLP and specialized medical metrics.Source: Correlation with Radiologist Judgments · p. 7, Table 3 · p. 7

Pairwise Preference and Clinical Awareness.

• 95.71% ACC and τ = 0.9807 on X-ray, with MAE 0.0214, distinguish AtomiMed from GREEN (MAE 0.1857, ACC 63.57%) and lexical baselines (13%–21% ACC).The passage describes AtomiMed’s MAE as an order of magnitude lower than GREEN’s and over twenty times lower than any lexical baseline.Source: Pairwise Preference and Clinical… · p. 7
• 84.33% ACC on CT and 49.86% on Ultrasound show AtomiMed’s cross-modal performance; GREEN reaches 33.83% on CT and RaTEScore 35.00% on Ultrasound.GREEN’s Kendall’s τ falls from 0.6481 on X-ray to 0.3283 on CT and 0.1513 on MRI.Source: Pairwise Preference and Clinical… · p. 8
• In MRI, AtomiMed’s scores cluster around the regression line (τ = 0.42, MAE = 0.281), unlike GREEN’s wide scatter (τ = 0.18, MAE = 0.379).The passage characterizes GREEN’s MRI agreement as near-random.Source: Pairwise Preference and Clinical… · p. 8, Figure 3 · p. 8

Attribute-level analysis

• Morphology scores are 6.0–13.2 across models, versus 1.3–5.9 on Severity and 1.0–6.9 on Size, revealing weak quantification of clinical significance and extent.HuatuoGPT-34B scores 1.27 and Qwen2.5-VL-7B scores 0.80 on Severity, near the floor.Source: Granular Performance Analysis via… · p. 8

Disease-level analysis

• Respiratory scores reach 21.84 for HuluMed-7B, while Digestive scores are 0.32–10.93, Reproductive 1.31–5.34, and Urinary 0.07–6.91.The passage describes respiratory dominance and the other systems as severely underserved, consistent with chest-centric pretraining biases.Source: Granular Performance Analysis via… · p. 8
• InternVL3.5-38B scores 26.20 on Endocrine but 0.97 on Urinary, exemplifying uneven anatomical coverage invisible to prior evaluation metrics.AtomiMed identifies these fine-grained deficiencies through its granular analysis.Source: Granular Performance Analysis via… · p. 8

• AtomiMed decomposes reports into Disease- and Attribute-level Atomic Clinical Facts and applies bidirectional Agentic Cross-Verification as a computational protocol for radiological peer review.Source: §4 · p. 8
• Figure 4 presents category-level medical-attribute and disease-level anatomical-system performance heatmaps, where higher scores indicate better alignment with human-verified atomic facts.Source: Figure 4 · p. 9
• Future work will develop distilled backbones to reduce inference cost, extend the attribute hierarchy to longitudinal imaging, and broaden benchmark coverage to more clinical specialties.Source: §4 · p. 9