Source-linked AI summary
Generalist Foundation Models from a Multimodal Dataset for 3D Computed Tomography
Ibrahim Ethem Hamamci, Sezgin Er, Chenyu Wang, Furkan Almas, Ayse Gulnihan Simsek, Sevval Nil Esirgun, Irem Dogan, Omer Faruk Durugol, Benjamin Hou, Suprosanna Shit, Weicheng Dai, Murong Xu, Hadrien Reynaud, Muhammed Furkan Dasdelen, Bastian Wittmann, Tamaz Amiranashvili, Enis Simsar, Mehmet Simsar, Emine Bensu Erdemir, Abdullah Alanbay, Anjany Sekuboyina, Berkan Lafci, Ahmet Kaplan, Zhiyong Lu, Malgorzata Polacin, Bernhard Kainz, Christian Bluethgen, Kayhan Batmanghelich, Mehmet Kemal Ozdemir, Bjoern Menze
TL;DR
Limited open 3D medical imaging datasets paired with reports hinder development of generalist models. The paper introduces CT-RATE and uses it to build CT-CLIP and CT-CHAT; CT-CLIP outperforms supervised baselines across validation sets, while CT-CHAT outperforms other open-source assistants across tasks. The models and dataset provide an open foundation for further 3D medical imaging research, but clinical deployment still requires additional validation and safeguards.
Problem
Open-source 3D medical imaging datasets paired with textual reports are scarce, limiting development of advanced generalist models for 3D imaging.
Method
The authors curate CT-RATE and train CT-CLIP with contrastive alignment of 3D CT volumes and radiology reports, then combine its vision encoder with a pretrained language model to create CT-CHAT.
Results
CT-CLIP outperforms supervised methods across internal and external zero-shot abnormality-detection evaluations, while CT-CHAT outperforms other open-source vision-language assistants across tasks.
Takeaways & Limitations
The open release of CT-RATE, CT-CLIP, and CT-CHAT establishes a foundation for future multimodal AI research in 3D medical imaging.
Takeaways & Limitations
CT-RATE and the resulting models come from a single geographic and clinical setting, which may limit generalizability to other populations, and clinical deployment requires further validation.
Abstract
from arXiv · showhide
Advancements in medical imaging AI, particularly in 3D imaging, have been limited due to the scarcity of comprehensive datasets. We introduce CT-RATE, a public dataset that pairs 3D medical images with corresponding textual reports. CT-RATE comprises 25,692 non-contrast 3D chest CT scans from 21,304 unique patients. Each scan is accompanied by its corresponding radiology report. Leveraging CT-RATE, we develop CT-CLIP, a CT-focused contrastive language-image pretraining framework designed for broad applications without the need for task-specific training. We demonstrate how CT-CLIP can be used in multi-abnormality detection and case retrieval, and outperforms state-of-the-art fully supervised models across all key metrics. By combining CT-CLIP's vision encoder with a pretrained large language model, we create CT-CHAT, a vision-language foundational chat model for 3D chest CT volumes. Finetuned on over 2.7 million question-answer pairs derived from the CT-RATE dataset, CT-CHAT underscores the necessity for specialized methods in 3D medical imaging. Collectively, the open-source release of CT-RATE, CT-CLIP, and CT-CHAT not only addresses critical challenges in 3D medical imaging but also lays the groundwork for future innovations in medical AI and improved patient care.
Main
3D medical imaging AI is constrained by limited open datasets pairing volumes with reports, despite the flexibility of image-text foundation models in 2D imaging. The paper addresses this gap with CT-RATE and models built for broad 3D chest CT applications.
- 3D imaging provides more comprehensive patient views than 2D imaging, while CT offers speed and broad availability for abnormality diagnosis.
- Image-text foundation models can support classification, retrieval, and interactive diagnostics without task-specific training or explicit manual image annotations.
- Open-source 3D imaging datasets paired with textual reports remain scarce because their creation requires substantial resources, expertise, and technical knowledge.
- Existing 3D datasets contain only a few thousand cases, generally lack paired text, and offer fewer pretrained open-source models than 2D imaging.
- CT-RATE pairs 50,188 reconstructed 3D chest CT volumes from 25,692 scans and 21,304 patients with radiology reports, supporting CT-CLIP and CT-CHAT development.
Results
The authors curate and evaluate a heterogeneous report-linked CT dataset, then use it to develop CT-CLIP and CT-CHAT for zero-shot detection, retrieval, and multimodal interaction. CT-CLIP outperforms supervised baselines across internal and external validation, while CT-CHAT outperforms other open-source vision-language assistants across tasks.
- CT-RATE contains 25,692 non-contrast 3D chest CT volumes from 21,304 patients, totaling over 14.3 million slices and spanning varied scanners, protocols, and reconstructions.
- CT-RATE uses radiologist-authored impressions and findings, with both sections together outperforming either section alone for CT-CLIP abnormality classification.
- The zero-shot CT-CLIP approach exceeds the supervised baseline by 0.102 mean AUROC and 0.050 mean F1 internally, with gains across RAD-ChestCT and UPMC external sets.External gains are 0.085 AUROC and 0.073 F1 on RAD-ChestCT, and 0.087 AUROC and 0.037 F1 on UPMC.
- All 18 abnormalities achieve higher mean AUROC scores with zero-shot CT-CLIP than with the supervised baseline on internal validation.
- CT-CLIP retains open-vocabulary inference after VocabFine, supporting abnormalities not included during fine-tuning and broader multimodal downstream tasks.
- CT-CHAT generates clinically accurate answers across question answering, multiple-choice tasks, and detailed radiology report generation, outperforming other open-source vision-language assistants.Its performance is attributed to CT-CLIP’s 3D spatial representation and CT-RATE’s radiology-specific paired data.
Discussion
CT-RATE addresses the lack of large 3D medical imaging datasets paired with radiology reports by supporting CT-CLIP and CT-CHAT. These models show broad performance across detection, retrieval, and vision-language tasks, while remaining subject to deployment and generalizability limitations.
- CT-CLIP: CT-CLIP uses contrastive language-image pretraining to align 3D chest CT volumes with radiology reports for task-flexible inference.The framework supports zero-shot predictions beyond predefined classes without requiring task-specific training.
- Detection and validation: CT-CLIP’s zero-shot multi-abnormality detection outperforms the fully supervised method across all metrics on internal and two external validation datasets.The external datasets test generalization under distribution shifts from different countries.
- CT-CHAT: CT-CHAT combines CT-CLIP’s 3D vision encoder with a large language model and outperforms existing 2D models on visual question answering and report generation evaluations.Including the full CT volume is particularly important for report generation, while clinical efficacy is assessed across multiple findings.
- Limitations and future work: CT-CHAT remains far from clinical deployment because robustness, interpretability, regulatory compliance, and prospective validation still require evaluation.CT-RATE and the models also originate from a single geographic and clinical setting, which may limit generalizability to other populations.
- Retrieval: CT-CLIP supports volume-to-volume and report-to-volume retrieval, but finetuning for abnormality classification diminishes retrieval performance, especially for report-to-volume retrieval.The reduction is primarily associated with effects on the text transformer in VocabFine.
Online Methods
CT-RATE is curated from non-contrast chest CT volumes and paired radiology reports, with standardized metadata, anonymization, labeling, preprocessing, and planned annotation extensions.
- CT-RATE contains 50,188 reconstructed CT volumes from 25,692 experiments involving 21,304 unique patients, split into 20,000 training and 1,304 validation patients.
- The dataset includes varied reconstruction techniques, scanner manufacturers, resolutions, spatial spacings, slice thicknesses, and volume sizes to support evaluation across abnormalities.Volumes contain 100–600 slices, averaging 304.7 slices, with multiple scanner and resolution configurations.
- Reports are anonymized by removing personal identifiers and date information, then translated from Turkish to English and reviewed by bilingual medical students.
- A 1,000-report annotation set supports automated extraction of 18 abnormality labels by merging findings and impression sections and grouping similar conditions.
- Volumes are standardized to 0.75 mm in-plane and 1.5 mm through-plane spacing, then center-cropped or padded to 480 × 480 × 240.
- The dataset includes organ-level anatomical segmentation labels, while multi-abnormality segmentation and localization labels are planned for future work.
Adding external validation datasets to CT-RATE
Two independent external datasets provide distribution-shift evaluations for models developed on CT-RATE.
- RAD-ChestCT contributes 3,630 non-contrast chest CT volumes from another country, reconstructed with a single technique and acquired using Siemens and GE scanners.Its scans have 512 × 512 resolution and 46–1277 slices, averaging 511.6.
- The MedSyn validation split contributes 1,024 chest CT scans with paired radiology reports, each from a different de-identified subject at UPMC.The same automated report labeler extracts 18 abnormality labels for evaluation.
Developing the CT-CLIP model
CT-CLIP adapts CLIP to paired 3D chest CT volumes and radiology reports by aligning learned visual and textual representations through contrastive learning.
- CT-CLIP uses a 3D CT encoder and CXR-Bert text encoder to extract latent embeddings from CT volumes and radiology reports.
- Contrastive loss trains the model to align 512-dimensional CT and text projection layers, enabling meaningful image–report correlations and zero-shot classification.
- The CT encoder averages encoded tokens across the axial plane, then flattens and transforms them into a 512-dimensional representation.Axial attention preserves information from every slice while the transformation reduces GPU memory requirements.
- CXR-Bert processes up to 512 report tokens, allowing both findings and impression sections to contribute to the text representation.
CT-CLIP for zero-shot multi-abnormality detection
CT-CLIP performs zero-shot multi-abnormality detection using positive and negative prompts, while VocabFine and ClassFine provide alternative task-specific finetuning strategies.
- CT-CLIP for zero-shot multi-abnormality detection: CT-CLIP converts positive and negative prompt logits into abnormality-presence probabilities with softmax for zero-shot multi-abnormality classification.The prompts represent statements such as an abnormality being present or absent.
- CT-CLIP for zero-shot multi-abnormality detection: Prompt engineering compares seven radiology-style positive-negative formulations, including “There is {∅/no} {abnormality}.”
- Finetuning CT-CLIP for multi-abnormality detection: ClassFine adds a linear classification layer and trains the modified model using ground-truth labels, with evaluations on internal and external datasets.
- Finetuning CT-CLIP for multi-abnormality detection: VocabFine preserves open-vocabulary capabilities by finetuning positive and negative prompt logits against binary ground-truth arrays with binary cross-entropy.For 18 abnormalities, the unified representation contains 36 logits, divided into 12-element segments for gradient runs.
- Finetuning CT-CLIP for multi-abnormality detection: VocabFine uses the same inference strategy as CT-CLIP after finetuning.
Benchmarking the CT-CLIP models for multi-abnormality detection
CT-CLIP and finetuned models are benchmarked against CT-Net on the 18 CT-RATE abnormalities, using per-abnormality metrics and bootstrap-based statistical analysis.
- CT-Net serves as the fully supervised baseline for evaluating CT-CLIP and finetuned models on CT-RATE abnormalities.CT-Net is adapted to CT-RATE by resizing chest CT volumes to 420 × 420 × 402.
- Accuracy, precision, and weighted F1-scores are calculated separately for each of the 18 abnormalities.The overall weighted F1 averages per-abnormality binary-task scores so each abnormality contributes equally.
- 1,000 bootstrap iterations estimate metric distributions and dispersion on internal and external validation sets.Each iteration samples evaluated volumes with replacement and computes the standard deviation of the resulting metric values.
Benchmarking the CT-CLIP models for CT volume retrieval
CT volume retrieval uses shared CT-report and volume embeddings, cosine similarity, and retrieval metrics tailored to image-to-image and report-to-volume matching.
- CT-CLIP enables volume retrieval by comparing cosine similarities among volumes and reports in a unified embedding space.CT tokens are projected into the shared embedding space before similarity computation.
- MAP@K evaluates image-to-image retrieval by averaging precision among retrieved volumes relevant to the target based on abnormality-label overlap.A retrieved volume is relevant when its True abnormality labels overlap with the target labels by intersection over union.
- AP@K uses precision and relevance at each retrieved position, while final class retrieval accuracy is computed across samples.P_i denotes precision at position i and R_i denotes relevance at position i.
- Image-to-image retrieval is assessed with MAP@1, MAP@5, MAP@10, and MAP@50 on internal and external validation sets.Recall@K is excluded because identical query images would always have the highest cosine similarity, yielding Recall@K of 1.
- Report-to-volume retrieval ranks CT volumes by cosine similarity between report and volume embeddings and evaluates Recall@5, Recall@10, Recall@50, and Recall@100.The external validation set lacks text reports, limiting this evaluation to settings with available report queries.
Creating the dataset for CT-CHAT
CT-CHAT is trained on a report- and volume-derived VQA dataset organized into long-answer, multiple-choice, and validation components, with filtering for usable generated data.
- The CT-CHAT VQA dataset is derived from free-form radiology reports and chest CT volumes using the CT-RATE train-validation split.The training dataset is organized into four main components.
- Long-answer questions comprise conversation-based, description, free-response, and text-only subcategories.The first three are generated from radiology reports, whereas text-only questions use 48,343 chest-CT-related PubMed Central-OA articles.
- The long-answer training component contains 281,689 conversations and 518,053 conversation questions, alongside description, free-response, and text-only questions.All questions are generated with Llama 3.1 8B, and question-generation scripts and prompts are released.
- The validation set includes conversation, description, free-response, short-answer, multiple-choice, and report-generation questions.Its entries are curated from free-form reports in the CT-RATE validation set.
- Generated data are filtered for accurate, properly formatted examples that can be parsed as JSON for model training.The pipeline also limits the initial number of generated questions.
Developing the CT-CHAT model
CT-CHAT combines a frozen CT-CLIP vision encoder with a multimodal projector and LLM, using staged alignment and finetuning to generate radiology reports and answer questions about 3D CT volumes.
- CT-CHAT adapts the LLaVA framework by compressing 3D CT volumes, projecting visual features into the LLM embedding space, and generating contextualized outputs.The architecture integrates a frozen CT-CLIP vision encoder, multimodal projector, and large language model.
- The primary CT-CHAT configuration uses the 70B Meta Llama 3.1 model, while an 8B configuration provides a smaller alternative.The study evaluates four different LLMs.
- Training begins with medical concept alignment, freezing the LLM while training the projector to map CT volumes to radiology reports.This step is evaluated through an ablation study and is intended to align the projector with biomedical concepts.
- Each finetuning sample combines frozen CT-CLIP volume features, a structured nodule template, and a reference report.The nodule template includes count, identifiers, lobe, centroid, diameter, and volume information.
- Finetuning updates only LoRA weights and the projector in the 8B configuration, and segmentation-derived cues markedly improve performance.The workflow can incorporate additional abnormalities as more 3D segmentation models become available.
Benchmarking the CT-CHAT model
CT-CHAT is benchmarked against open-source 2D VQA systems using DRRs for fair comparison, automatic language and clinical-accuracy metrics, and blinded radiologist assessment. In the reader study, CT-CHAT received higher report-quality scores than RadFM and CT2Rep.
- Comparative evaluation: CT-CHAT is compared with four open-source 2D VQA models using DRRs derived from CT-RATE for a fair comparison.The baselines are LLaVA 1.6 with Mistral 7B, LLaVA 1.6 with Vicuna 13B, LLaVA-Med, and CXR-LLaVA.
- Automatic evaluation: Five measures evaluate generated responses: BLEU-1, METEOR, ROUGE-L, CIDEr, and Llama score for clinical accuracy.The Llama score compares generated responses with ground truth using Llama 3.1 70B.
- Blinded reader study: CT-CHAT achieved mean reader scores of 3.44, 4.10, and 4.52, compared with RadFM scores of 0.55, 0.91, and 0.86.The three scores correspond to readers R1, R2, and R3, respectively.
- Blinded reader study: CT-CHAT also exceeded CT2Rep, whose mean reader scores were 2.66, 2.52, and 2.74 for R1, R2, and R3.Reports were rated on finding description, localisation, terminology, and overall clinical adequacy using a 0–10 scale.
- Ablation and safeguards: An input-volume ablation study adds a guardrail because CT-CHAT is expected to hallucinate pathologies when the volume is absent.The guardrail is implemented through a system prompt.
Computational hardware and software
The experiments use separate Python and PyTorch environments for CT-CLIP and CT-CHAT, run primarily on four 80GB A100 GPUs with substantial memory and CPU resources.
- CT-CLIP environment: CT-CLIP experiments use Python 3.11.5, PyTorch 2.0.1, CUDA 11.7, and supporting scientific, vision, and transformer libraries.The listed libraries include SciPy, Torchvision, Scikit-learn, Pandas, NumPy, Transformers, and Accelerate.
- CT-CHAT environment: CT-CHAT experiments use Python 3.12.4, PyTorch 2.4.0, CUDA 12.4, and updated versions of several supporting libraries.The environment includes SciPy 1.14.0, Torchvision 0.19.0, Scikit-learn 1.2.2, Pandas 2.2.2, and NumPy 1.26.4.
- Hardware: Training primarily uses four 80GB A100 GPUs, 500GB RAM, and 24 CPUs.
Data availability
The CT-RATE dataset, external RAD-ChestCT validation set, trained models, and code are publicly available through online repositories.
- Datasets: CT-RATE is publicly accessible through its Hugging Face dataset repository.
- Datasets: The external RAD-ChestCT validation set is openly accessible through Zenodo.
- Models and code: The trained CT-CLIP and CT-CHAT models and codebases are publicly available through GitHub.