Source-linked AI summary
A multicenter benchmark and clinically structured metric for coronary CTA report generation
Zhiyu Ye, Yue Sun, Limiao Zou, Cheng Xu, Keting Xu, Tong Hu, Yue Yu, Hairong Zheng, Yining Wang, Tong Zhang
TL;DR
Reliable CCTA report-generation evaluation requires standardized multicenter benchmarks and metrics that capture clinical correctness and anatomical specificity. The study establishes a four-hospital benchmark and develops CSMCCTA, which structures report findings across anatomical levels and calibrates component weights using expert assessments. CSMCCTA aligned strongly with expert judgments, while the CCTA-trained C2RG model led benchmark performance across hospitals but remained far from optimal.
Problem
Existing CCTA report-generation evaluation lacks standardized multicenter benchmarks and clinically structured metrics that prioritize clinical correctness over textual fluency.
Method
The study builds a four-hospital benchmark and develops CSMCCTA using guideline-informed patient-, vessel-, and segment-level variables, finest-level matching, and expert-calibrated weights.
Results
CSMCCTA showed strong expert alignment, while C2RG achieved the highest CSMCCTA scores across all four hospitals but remained far from optimal.
Takeaways & Limitations
The benchmark supports standardized model comparison, while CSMCCTA makes finding agreement and reporting specificity visible in CCTA report evaluation.
Takeaways & Limitations
CSMCCTA relies on a CCTA-specific schema, deterministic keyword extraction may miss uncommon or institution-specific expressions, and its expert calibration requires broader multicenter validation.
Abstract
from arXiv · showhide
Reliable evaluation of automated coronary computed tomography angiography (CCTA) report generation requires standardized multicentre benchmarks and clinically structured metrics. We established a four-centre benchmark comprising 3,021 CCTA series from 818 patient-report pairs to evaluate seven open-source three-dimensional vision-language models. We developed CSM$_{\text{CCTA}}$, a clinically structured metric for CCTA report evaluation, with patient-, vessel-, and segment-level variables defined according to clinical guidelines. Report pairs are compared at the finest shared anatomical level, and the contributions of different clinical components are weighted based on expert assessments. We estimated these weights using 70 expert-scored cases and evaluated clinical alignment in a non-overlapping set of 30 cases. CSM$_{\text{CCTA}}$ showed a strong correlation with radiologist scores (Pearson's $r=0.97$, $p<0.001$), exceeding the next-best metric, FORTE ($r=0.70$), by 0.27, and agreed with expert preferences in 115 of 160 pairwise comparisons (71.9\%). Under controlled perturbations, CSM$_{\text{CCTA}}$ remained stable to clinically equivalent wording and decreased monotonically with progressive information omission. In the multicenter benchmark, the CCTA-trained C2RG model achieved the highest CSM$_{\text{CCTA}}$ scores across all four hospitals, although its performance remained far from optimal. In contrast, CCTA-irrelevant reports accounted for up to 98.7\% of the outputs from generalist models. Together, the benchmark provides a standardized setting for model comparison, while CSM$_{\text{CCTA}}$ enables clinically structured evaluation of finding agreement and anatomical specificity. These results support a more clinically aligned and anatomically resolved approach to evaluating CCTA report generation. Code is available at https://openi.pcl.ac.cn/OpenMedIA/CSM_CCTA.
Introduction
CCTA report generation needs evaluation that prioritizes clinical correctness and anatomical specificity over fluent wording. The study introduces a multicenter benchmark and CSMCCTA, a structured metric designed for clinically interpretable comparisons.
- Motivation: CCTA report generation remains difficult because radiologists must synthesize complex 3D findings into accurate clinical reports.Automated reporting could reduce this time-intensive workload, but clinical translation requires reliable evaluation.
- Motivation: Fluent model-generated reports can contain severe clinical errors, making readability insufficient for medical report evaluation.Evaluation must identify abnormality types, anatomical locations, and severity gradings.
- Metric validation: CSMCCTA uses expert-scored derivation and validation cases to calibrate its weighting scheme and showed the highest observed correlation among 13 compared metrics.Weights are estimated by constrained linear regression using the derivation set, with validation performed on held-out cases.
- Motivation: Traditional NLG metrics emphasize semantic similarity or fluency and may provide unreliable, poorly interpretable scores when reports differ in clinical detail.These metrics do not explicitly account for clinically critical information or reporting-level differences.
- Contribution: CSMCCTA parses reports into patient-, vessel-, and segment-level clinical variables and compares findings at the finest shared anatomical level.Segment-level representations include vessel, segment, abnormality, and severity.
Results
Across four hospitals, C2RG achieved the highest CSMCCTA scores, while CSMCCTA showed stronger clinical alignment and robustness than compared metrics. Generalist models often produced irrelevant or superficial reports, and CSMCCTA preserved anatomical specificity while handling wording variation and information loss.
- 3,021 CCTA series and 818 patient-report pairs from four hospitals formed the multicenter benchmark for seven 3D medical VLMs.
- Model comparison across the benchmark: C2RG achieved the highest CSMCCTA scores at PUMCH (0.456), SJTH (0.520), TCH (0.355), and FAHXMU (0.479).
- Analysis across anatomical levels with CSMCCTA: Generalist models produced superficial patient-level information or CCTA-irrelevant reports, whereas detailed fine-grained findings characterized stronger performance.
- Clinical relevance of metrics: CSMCCTA achieved the highest observed correlation with expert scores (r = 0.97, p < 0.001) and the narrowest observed 95% confidence interval.
- Clinical relevance of metrics: CSMCCTA agreed with expert preferences in 115/160 pairwise comparisons (71.9%), compared with 51.3% for FORTE.
- Robustness analysis of evaluation metrics: CSMCCTA remained stable under clinically equivalent wording and aligned linearly with clinical information completeness under progressive omission (β = 1.00; R2 = 0.87; boundary error = 0.05).Its score drop under semantic-preserving lexical variation was 0.00, while CIDEr’s was 0.50.
Discussion
The benchmark and CSMCCTA provide a clinically structured basis for comparing CCTA report-generation models while separating finding accuracy from reporting specificity. The discussion emphasizes actionable anatomical detail but limits interpretation to the evaluated models and validation settings.
- Discussion: CSMCCTA separates finding accuracy from anatomical reporting detail by comparing reports at the finest shared anatomical level.This reduces unfair penalization from mismatched reporting detail, while preserving the distinction between agreement and specificity.
- Discussion: CCTA-adapted models generated more relevant and anatomically specific findings, whereas several generalist models produced coarse or unrelated reports.Anatomical-level outputs can reveal whether failures involve overall relevance, vessel-level description, or segment-level localization.
- Discussion: Fluent report-style language alone is inadequate for judging quality in this specialized imaging task.Anatomical-level outputs provide diagnostic information beyond lexical similarity when selecting data, training objectives, or models.
- Discussion: CSMCCTA is robust to clinically equivalent wording and omitted findings, but its adaptive level matching creates an interpretive trade-off.A report may be accurate at the shared anatomical level while lacking detail required for a specific task; component outputs and selected level should therefore be considered.
- Discussion: The conclusions are constrained by a CCTA-specific schema, PUMCH-derived expert and perturbation studies, variable reference reports, and illustrative rather than prospective real-world examples.Larger multi-institutional expert studies are needed to assess transportability of the learned weights.
- Discussion: The benchmark and CSMCCTA make both finding agreement and reporting specificity visible while preserving clear boundaries on what each evaluation mode can establish.In single-report mode, CSMCCTA assesses structural completeness and anatomical specificity, not clinical accuracy without a reference report.
Methods
The study constructs a four-hospital CCTA report-generation benchmark and evaluates seven open-source 3D vision-language models using a hierarchical, clinically structured metric. CSMCCTA represents findings across patient, vessel, and segment levels, aggregates detailed information when needed, and compares reports at their shared anatomical level.
- Benchmark: The benchmark evaluates CCTA report generation from 3D images against paired clinical reference reports across four hospitals.The dataset includes 3,021 CCTA series and 818 patient-report pairs.
- Benchmark: Seven open-source 3D medical VLMs, including C2RG and generalist models, are assessed with CSMCCTA and 13 established metrics.The comparison includes lexical, contextual, and medical report metrics such as BLEU, ROUGE, BERT-F1, GREEN, RaTEScore, and FORTE.
- Metric design: CSMCCTA evaluates report quality through hierarchical clinical variables at patient, vessel, and segment anatomical levels.The representation includes coronary dominance, calcification severity, maximal stenosis severity, plaque type, and segment-level vessel-abnormality-type/severity quadruples.
- Metric design: The metric preserves explicitly reported variables while deriving missing coarser variables from more detailed vessel or segment findings.Aggregation uses maximal stenosis severity for vessel and patient levels and unions plaque types across corresponding segment findings.
- Metric design: CSMCCTA selects the coarser anatomical level shared by the reference and prediction before scoring their available variables.This avoids rewarding unsupported segment-level detail and avoids penalizing predictions when only a coarser comparison is supported.