Source-linked AI summary
CT2Rep: Automated Radiology Report Generation for 3D Medical Imaging
Ibrahim Ethem Hamamci, Sezgin Er, Bjoern Menze
TL;DR
Radiology report generation had not extended to 3D imaging amid computational complexity and scarce paired datasets. CT2Rep generates reports for chest CT, with a longitudinal multimodal extension; it outperforms a designed baseline, while CT2RepLong performs comparably to CT2Rep on a limited longitudinal dataset.
Problem
Automated report generation had advanced for 2D imaging but not 3D, where volumetric complexity and scarce paired datasets present challenges.
Source: §1 · pp. 1–2Method
CT2Rep generates reports for 3D chest CT with an auto-regressive causal transformer and relational memory, and CT2RepLong adds cross-attention fusion for longitudinal multimodal data.
Source: §1 · p. 2, §2 · p. 3, §2.1 · p. 4Results
CT2Rep outperforms its designed 3D vision-encoder baseline, while CT2RepLong performs comparably to CT2Rep despite longitudinal data covering only 13% of patients.
Source: §1 · p. 2, §3.1 · p. 7, §3.2 · p. 8Takeaways & Limitations
The authors provide trained models and code to lay a foundation for further research on 3D report generation.
Source: §4 · p. 9CT2RepLong was evaluated with longitudinal data from only 13% of patients.
Source: §3.2 · p. 8
Abstract
from arXiv · showhide
Medical imaging plays a crucial role in diagnosis, with radiology reports serving as vital documentation. Automating report generation has emerged as a critical need to alleviate the workload of radiologists. While machine learning has facilitated report generation for 2D medical imaging, extending this to 3D has been unexplored due to computational complexity and data scarcity. We introduce the first method to generate radiology reports for 3D medical imaging, specifically targeting chest CT volumes. Given the absence of comparable methods, we establish a baseline using an advanced 3D vision encoder in medical imaging to demonstrate our method's effectiveness, which leverages a novel auto-regressive causal transformer. Furthermore, recognizing the benefits of leveraging information from previous visits, we augment CT2Rep with a cross-attention-based multi-modal fusion module and hierarchical memory, enabling the incorporation of longitudinal multimodal data. Access our code at https://github.com/ibrahimethemhamamci/CT2Rep
1 Introduction
Radiology report generation had advanced for 2D imaging but not 3D, where paired-data scarcity and volumetric complexity pose challenges. CT2Rep introduces automated report generation for chest CT, with a longitudinal multimodal extension.
1 Introduction
- Earlier machine-learning progress in radiology reporting focused on 2D imaging, while 3D reporting remained unaddressed amid computational complexity and scarce paired datasets.Volumetric data also makes clinically relevant report generation more challenging.Source: §1 · pp. 1–2
- CT2Rep outperforms a designed baseline built from a state-of-the-art 3D chest CT vision encoder.The authors note that directly comparable 3D report-generation methods did not exist.Source: §1 · p. 2
- CT2Rep is presented as the first automated radiology report-generation framework for 3D medical imaging, using an auto-regressive causal transformer.The framework targets chest CT volumes and uses CT-RATE, a dataset of chest CT volumes paired with radiology reports.Source: §1 · p. 2
- CT2Rep is extended with cross-attention-based multimodal fusion and hierarchical memory to incorporate longitudinal patient data from previous visits.The extension is named CT2RepLong.Source: §1 · p. 2
- The authors make trained models and source code publicly available.They state that this is intended to facilitate further research.Source: §1 · p. 2
2 Methods
The methods define CT2Rep as a 3D sequence-to-sequence report model and extend it to integrate prior-visit volumes and reports. The section also specifies CT-RATE preparation and longitudinal pairing.
2.1 The Proposed Method
- CT2Rep converts a chest CT volume into patch embeddings, then applies spatial and causal transformers while preserving volumetric information.Source: §2.1 · pp. 3–4
- Its decoder uses relational memory to retain report patterns across generation steps and MCLN to feed that memory into decoder scaling and shifting.Source: §2.1 · p. 4
2.2 Longitudinal Data Utilization
- CT2RepLong predicts a report for a new volume by cross-attending prior-volume and prior-report representations and concatenating their attended features.Source: §2.2 · p. 5
Multimodal transformer decoder.
- The longitudinal decoder adds cross-attention between prior reports and volumes in both directions, then applies RM and another memory cross-attention before MCLN.Source: Multimodal transformer decoder · p. 5
2.3 Dataset Preparation.
- CT-RATE contributes 25,701 non-contrast CT volumes from 21,314 patients; training uses findings sections, while longitudinal pairs draw from patients with more than two visits.Source: §2.3 · p. 6
3 Experiments and Results
The experiments compare CT2Rep with a CT-Net-based baseline and assess whether longitudinal information improves report generation. They also describe evaluation metrics and implementation settings.
3 Experiments and Results
- Evaluation used BLEU, METEOR, and ROUGE-L for language generation, plus precision, recall, and F1 for clinical efficacy.Clinical efficacy scores came from classifying ground-truth and generated reports for 18 abnormalities.Source: §3 · p. 6
3.1 Comparison with the Baseline Method
- CT2Rep significantly outperformed the CT-Net baseline; a case comparison showed more radiologist-like content flow and medical terminology.Table 1 provides the quantitative evaluation, while Fig. 3 compares generated reports with the ground truth.Source: §3.1 · p. 7, Table 1 · p. 7, Figure 3 · p. 7
3.2 Ablation Study on Longitudinal Data Utilization
- CT2RepLong benefited from prior multimodal data and its cross-attention mechanism, and performed comparably to CT2Rep despite longitudinal data covering only 13% of patients.The ablation compared previous-report embeddings, previous-volume embeddings, and their simple fusion without cross-attention; the case study notes that CT2RepLong included terms missed by the baseline.Source: §3.2 · p. 8, Figure 4 · p. 8
3.3 Implementation Details
- 35 seconds for CT2Rep and 50 seconds for CT2RepLong were the reported inference times.Models were trained for 20 epochs over one week on a single NVIDIA A100 GPU; CT2Rep and CT2RepLong used 49,138 and 28,441 training pairs, respectively.Source: §3.3 · p. 8
4 Discussion and Conclusion
The conclusion presents CT2Rep as a first framework for chest-CT report generation and describes CT2RepLong as an extension using longitudinal data. The authors report releasing trained models and code to support further research.
4 Discussion and Conclusion
- CT2Rep is presented as the first framework for automating report generation from 3D medical imaging, focused on chest CT.The conclusion describes its auto-regressive causal transformer and relational memory as components intended to enhance report-generation accuracy.Source: §4 · p. 9
- CT2RepLong extends the framework with longitudinal data integration to provide additional context and accuracy.The authors state that trained models and code are open-source to provide a foundation for further research.Source: §4 · p. 9
- The authors acknowledge support from the Helmut Horten Foundation and Istanbul Medipol University, and declare no relevant competing interests.Source: §4 · p. 9
Supplementary Material
The supplementary tables describe clinical-efficacy metrics for generated reports, comparing CT2Rep and CT2RepLong with a baseline across abnormalities.
Supplementary Material
- Table 2 presents abnormality-based precision, recall, and F1 for reports generated by CT2Rep and the CT-Net-based baseline, alongside test-set abnormality ratios.Source: Table 2 · p. 12
- Table 3 presents per-abnormality precision, recall, and F1 for the baseline and CT2RepLong, alongside the test-set abnormality distribution.The caption describes CT2RepLong as using cross-attention multimodal fusion and longitudinal memory to integrate previous reports and volumes.Source: Table 3 · p. 13