Source-linked AI summary
Large-scale and Fine-grained Vision-language Pre-training for Enhanced CT Image Understanding
Zhongyi Shui, Jianpeng Zhang, Weiwei Cao, Sinuo Wang, Ruizhe Guo, Le Lu, Lin Yang, Xianghua Ye, Tingbo Liang, Qi Zhang, Ling Zhang
TL;DR
Medical vision-language models need scalable supervision while global image-report matching can miss clinically meaningful local correspondences. The paper proposes fVLM, which explicitly aligns anatomy-level CT regions with report descriptions and reduces normal and abnormal false negatives through disease-aware pairing. Across 54 diagnosis tasks, fVLM achieves 81.3% average AUC, exceeding CLIP by 12.9% and supervised methods by 8.0%.
Problem
Global image-report contrastive learning can overlook local anatomy-report correspondences, while large-scale CT vision-language data and comprehensive annotations are difficult to obtain.
Method
fVLM explicitly matches anatomical CT regions with corresponding report descriptions and calibrates fine-grained contrastive learning using false-negative identification and disease-aware pairing.
Results
81.3% average AUC across 54 diagnosis tasks surpassed CLIP by 12.9% and supervised methods by 8.0%.
Takeaways & Limitations
Fine-grained anatomy-level pre-training demonstrates strong potential for versatile CT image interpretation across multiple anatomies and diagnosis tasks.
Takeaways & Limitations
Fine-grained alignment requires localizing anatomical structures and decomposing reports, adding resource consumption and time commitment.
Abstract
from arXiv · showhide
Artificial intelligence (AI) shows great potential in assisting radiologists to improve the efficiency and accuracy of medical image interpretation and diagnosis. However, a versatile AI model requires large-scale data and comprehensive annotations, which are often impractical in medical settings. Recent studies leverage radiology reports as a naturally high-quality supervision for medical images, using contrastive language-image pre-training (CLIP) to develop language-informed models for radiological image interpretation. Nonetheless, these approaches typically contrast entire images with reports, neglecting the local associations between imaging regions and report sentences, which may undermine model performance and interoperability. In this paper, we propose a fine-grained vision-language model (fVLM) for anatomy-level CT image interpretation. Specifically, we explicitly match anatomical regions of CT images with corresponding descriptions in radiology reports and perform contrastive pre-training for each anatomy individually. Fine-grained alignment, however, faces considerable false-negative challenges, mainly from the abundance of anatomy-level healthy samples and similarly diseased abnormalities. To tackle this issue, we propose identifying false negatives of both normal and abnormal samples and calibrating contrastive learning from patient-level to disease-aware pairing. We curated the largest CT dataset to date, comprising imaging and report data from 69,086 patients, and conducted a comprehensive evaluation of 54 major and important disease diagnosis tasks across 15 main anatomies. Experimental results demonstrate the substantial potential of fVLM in versatile medical image interpretation. In the zero-shot classification task, we achieved an average AUC of 81.3% on 54 diagnosis tasks, surpassing CLIP and supervised methods by 12.9% and 8.0%, respectively.
1 INTRODUCTION
The paper introduces fVLM, an annotation-free CT vision-language model that aligns anatomy-level image regions with report descriptions and addresses false negatives in fine-grained contrastive learning. On MedVL-CT69K, it achieves strong zero-shot diagnostic performance across 54 tasks and 15 anatomies.
- Motivation: Medical image interpretation is burdensome, while comprehensive disease labels are costly and difficult to define for complex clinical conditions.Radiologists must inspect many slices and anatomies, and predefined one-hot labels cannot cover all diagnostic requirements.
- Method: fVLM performs anatomy-level matching between CT regions and corresponding report descriptions instead of contrasting whole images with whole reports.The framework decomposes and matches both modalities, addressing vision-language misalignment and improving interpretability.
- Method: Fine-grained contrastive learning is calibrated with false-negative identification for both normal and abnormal samples, shifting from patient-level to disease-aware pairing.Anatomy-level healthy cases and similar abnormalities can create ambiguous negative pairs.
- Data and experiments: 69,086 patients and 272,124 CT scans comprise MedVL-CT69K, curated to address limited medical vision-language data.The dataset includes corresponding diagnostic reports and supports scalable, annotation-free pre-training.
- Results: 81.3% average AUC across 54 disease diagnosis tasks surpassed CLIP by 12.9% and a supervised baseline by 8.0%.On CT-RATE and Rad-ChestCT, fVLM also achieved absolute AUC gains of 7.4% and 4.8% over the state-of-the-art approach.
2 RELATED WORK
Existing medical vision-language pre-training methods mainly use global image-report alignment, which can miss local clinical correspondences. Related approaches add implicit local alignment, but the paper positions explicit anatomy-level alignment as the relevant direction for CT.
- Existing methods: Existing medical vision-language pre-training methods primarily target 2D images of single body parts, especially chest X-rays, using contrastive alignment with reports.These methods learn transferable representations from paired medical scans and reports.
- Global alignment: Global alignment contrasts entire images with reports and overlooks local similarities or disparities between image patches and report pieces.This can produce vision-language misalignment and coarse-grained understanding of clinically relevant details.
- Implicit local alignment: GLoRIA, LoVT, and MGCA combine global contrastive learning with implicit local alignment through cross-attention between image regions and report sentences.The passage describes these methods as effective for 2D chest X-ray data while raising questions about their applicability to 3D CT volumes.
3 METHOD
The method builds anatomy-level visual and textual representations for fine-grained contrastive pre-training, then reduces false negatives among normal and similarly abnormal anatomy pairs.
- 3.1 DATA PRE-PROCESSING: CT scans are segmented into 104 anatomical regions and grouped into 36 major anatomies matching report-description granularity.Grouping addresses reports' often imprecise lesion localization.
- 3.1 DATA PRE-PROCESSING: fVLM extracts anatomy-specific visual tokens with segmentation masks and query tokens, while reports are decomposed into anatomy-wise textual descriptions.Query tokens are updated through self-attention to produce anatomy-wise visual representations.
- 3.2 FINE-GRAINED CONTRASTIVE PRE-TRAINING: The model aligns each anatomy across CT scans using image-to-text and text-to-image contrastive similarities, with cosine similarity and a learnable temperature.The loss aggregates cross-entropy terms across anatomy categories and structurally complete samples.
- 3.3 REDUCING FALSE NEGATIVES IN IMAGE-REPORT PAIRS: Anatomy-level normal samples proliferate false negatives because unaffected anatomies remain normal even when another anatomy is abnormal.Normal anatomy descriptions also vary substantially rather than following a fixed template.
- 3.3 REDUCING FALSE NEGATIVES IN IMAGE-REPORT PAIRS: The method corrects normal-pair labels and bootstraps abnormal-pair similarities to support disease-aware rather than purely patient-level pairing.Two fVLMs are alternately trained so one model supplies similarity estimates for correcting the other's targets.
4 EXPERIMENTS
Experiments evaluate fVLM on zero-shot abnormality detection, report generation, ablations, and scaling. Across these settings, fVLM outperforms comparison methods and remains more data-efficient than CLIP.
- Zero-shot abnormality detection: 54 diseases across 15 anatomies are evaluated for zero-shot abnormality detection on MedVL-CT69K.The dataset uses expert-annotated validation and test sets, with 36 and 54 diseases respectively.
- Zero-shot abnormality detection: 12.9 points on AUC and 9.5 points on ACC separate fVLM from CLIP in zero-shot abnormality detection.Against Merlin, the second-best competitor, fVLM gains 9.4 points on AUC and 6.7 points on ACC.
- Public benchmarks: 7.4% and 4.8% absolute AUC gains over CT-CLIP are achieved in internal and external benchmark evaluations, respectively.Zero-shot fVLM also exceeds supervised fine-tuned CT-VocabFine and CT-LiPro on F1-score in both test sets.
- Radiology report generation: 4.2 points on ACC and 3.8 points on GREEN improve over CLIP in the frozen report-generation protocol.With fine-tuning, fVLM still improves over CLIP by 2.5 points on ACC and 2.6 points on GREEN.
- Ablation study: 7.8 points on AUC and 6.0 points on ACC result from combining fine-grained alignment with false-negatives correction between normals.The ablation finds these two components contribute the largest gains, while co-teaching label correction outperforms either single model.
- Scaling law: Across multiple training-data scales, fVLM consistently outperforms CLIP and exhibits superior data efficiency.The comparison is presented through data scaling-law curves.
5 CONCLUSION
The paper presents fVLM as a fine-grained CT vision-language pre-training method that aligns anatomical structures with report descriptions. Experiments across diagnosis, report generation, and visualization demonstrate its superiority, while the required preprocessing adds resource and time costs.
- 5 CONCLUSION: fVLM explicitly aligns discrete CT anatomical structures with corresponding descriptions in diagnostic reports.This addresses the misalignment of methods that contrast entire images and reports.
- 5 CONCLUSION: Quantitative diagnosis, report-generation, and qualitative visualization experiments demonstrate fVLM’s superiority.The conclusion summarizes evidence across multiple evaluation types.
- Limitations and future work: Localizing anatomical structures and decomposing reports into anatomy-wise descriptions require additional resources and time.Future work targets anatomy-wise report generation to better use fVLM for this application.
A APPENDIX
The appendix describes anatomy-specific prompting for clinical-report processing. Separate prompts identify mentioned anatomies and extract descriptions for each anatomy individually.
- Prompt design: A prompt judges whether each anatomy is mentioned in the Findings or Impression section of a clinical report.The prompt is shown in Appendix Figure 6.
- Prompt design: The LLM is queried separately for each anatomy to extract anatomy-specific descriptions from Findings or Impression sections.This divide-and-conquer strategy simplifies extraction and improves description quality.
A.1 DETAILS ABOUT THE TEXT CLASSIFIER
The text classifier converts generated radiology reports into predictions for 54 abnormalities using expert-labeled MedVL-CT69K reports.
- A.1 DETAILS ABOUT THE TEXT CLASSIFIER: A BERT-base encoder and classification head are trained to identify 54 abnormalities in generated radiology reports.The classifier uses reports and corresponding disease labels with binary cross-entropy loss.
A.2 IMPLEMENTATION DETAILS
The implementation preprocesses CT volumes with standardized orientation, spacing, and intensity normalization, while anatomy-aware cropping prevents incomplete large structures from being excluded. The section also presents anatomy-wise normal-sample proportions and comparisons with radiologists.
- Preprocessing: CT scans are reoriented, resampled to 1 mm in-plane resolution and 5 mm slice spacing, and normalized by clipping Hounsfield units to -300:400 before mapping them to 0:1.
- Evaluation setup: Figure 8 reports the percentage of normal samples for each anatomy, while Figure 9 compares the method with three radiologists and annotates each abnormality’s positive-sample count.
- Cropping: Uniform anatomy sampling ensures the selected anatomy is completely included in each crop, avoiding bias from incomplete larger anatomies and loss exclusion.
A.3 READER STUDY
The reader study evaluates the method against three board-certified radiologists using 100 randomly selected test patients. The method remains below radiologists overall but reaches comparable accuracy for liver cirrhosis and splenomegaly.
- Study design: The reader study randomly selects 100 patients from the MedVL-CT69K test set for comparison with three board-certified radiologists.
- Overall comparison: The method shows a noticeable overall performance gap relative to professional radiologists.
- Disease-level comparison: Diagnostic accuracy is comparable to radiologists for some diseases, including liver cirrhosis and splenomegaly.
A.4 FURTHER ABLATION ANALYSIS
The ablation compares label-correction models and finds that self-correction degrades performance, whereas the proposed CoT strategy outperforms momentum-based correction.
- Label correction: Using the training model itself for contrastive-label correction causes significant performance degradation, attributed to error accumulation.
- Label correction: The proposed CoT strategy produces greater performance gains than the momentum model.
- Analysis: The ablation further measures the difference between the training model and the label-correction model to investigate these correction behaviors.
A.5 VISUALIZATION
The visualizations assess anatomical-text alignment and embedding structure. Heatmaps localize abnormalities, while t-SNE comparisons show more compact positive-case clusters than CLIP, supporting improved semantic understanding and interpretability.
- Activation maps: Heatmaps correlate anatomy-specific visual tokens with abnormality text embeddings and activate in affected areas for localized bladder stones and diffuse fatty liver.
- Activation maps: The model precisely localizes pathological changes across both localized lesions and diffuse abnormalities.
- Embedding visualization: Compared with CLIP, the model produces more compact visual-embedding clusters among positive cases of each abnormality.
- Supporting analyses: The reported visualizations are accompanied by anatomy grouping, text-classifier performance, and abnormality-distribution tables for training, validation, and test data.