PaperlayerView exact revision on arXiv

Source-linked AI summary

CLIP-Driven Universal Model for Organ Segmentation and Tumor Detection

Jie Liu, Yixiao Zhang, Jie-Neng Chen, Junfei Xiao, Yongyi Lu, Bennett A. Landman, Yixuan Yuan, Alan Yuille, Yucheng Tang, Zongwei Zhou

arXiv:2301.00785v5eess.IVcs.CVcs.LG

TL;DR

The paper targets limited, partially labeled abdominal CT datasets and models that miss anatomical semantics and broader tumor coverage. It introduces a CLIP-driven Universal Model that combines text embeddings with segmentation, trained on 3,410 scans from 14 datasets. The model ranks first on MSD and BTCV, is 6x faster than dataset-specific models, generalizes across hospitals, and transfers to novel tasks.

  • Problem

    Small, partially labeled datasets and limited tumor coverage constrain models to specific organs or tumors while overlooking anatomical semantics and novel domains.

  • Method

    The Universal Model integrates CLIP text embeddings with segmentation models to encode anatomical relationships and learn from partially labeled datasets.

  • Results

    The model ranks first on MSD and BTCV, predicts fewer tumor-detection false positives while maintaining high sensitivity, generalizes across hospitals, and transfers across tasks.

  • Takeaways & Limitations

    The framework supports segmentation of 25 organs and detection of 6 tumors within a computationally efficient universal model.

  • Takeaways & Limitations

    Human-annotation quality was not compared for 19 organs and 6 tumors because suitable multiply annotated public CT scans were unavailable.

Abstract

from arXiv · show

An increasing number of public datasets have shown a marked impact on automated organ segmentation and tumor detection. However, due to the small size and partially labeled problem of each dataset, as well as a limited investigation of diverse types of tumors, the resulting models are often limited to segmenting specific organs/tumors and ignore the semantics of anatomical structures, nor can they be extended to novel domains. To address these issues, we propose the CLIP-Driven Universal Model, which incorporates text embedding learned from Contrastive Language-Image Pre-training (CLIP) to segmentation models. This CLIP-based label encoding captures anatomical relationships, enabling the model to learn a structured feature embedding and segment 25 organs and 6 types of tumors. The proposed model is developed from an assembly of 14 datasets, using a total of 3,410 CT scans for training and then evaluated on 6,162 external CT scans from 3 additional datasets. We rank first on the Medical Segmentation Decathlon (MSD) public leaderboard and achieve state-of-the-art results on Beyond The Cranial Vault (BTCV). Additionally, the Universal Model is computationally more efficient (6x faster) compared with dataset-specific models, generalized better to CT scans from varying sites, and shows stronger transfer learning performance on novel tasks.

1. Introduction

The paper addresses limited, inconsistently labeled abdominal CT datasets by combining them in a CLIP-driven Universal Model that represents anatomical relationships. The model segments 25 organs and detects 6 tumors while improving benchmark performance, efficiency, generalization, and transferability.

  • 3,410 CT scans from 14 public datasets provide partially labeled annotations for 25 organs and 6 tumors.
  • CLIP text embeddings encode semantic and anatomical relationships among organs and tumors for the Universal Model.
  • Ranking first on MSD and BTCV establishes high abdominal organ segmentation performance.
  • The model predicts fewer false positives than existing models while maintaining high sensitivity for tumor detection.
  • 6x faster testing than dataset-specific models improves computational efficiency.
  • Performance generalizes to CT scans from varied hospitals without additional tuning and transfers across diseases, organs, and datasets.

2. Related Work

Prior work addresses partially labeled datasets but commonly remains limited in scale, task scope, and semantic representation. This paper instead presents one framework for organ segmentation and tumor detection using CLIP embeddings to capture relationships between anatomical structures.

  • Partial labels arise because public abdominal imaging datasets annotate different organs and tumors with inconsistent taxonomies.
  • Earlier assembled datasets were small, performed similarly to dataset-specific models, and lacked official benchmark evaluation.
  • One-hot labels discard semantic relationships between organs and tumors and are unsuitable for diverse tumor subtypes.
  • Universal Model jointly handles organ segmentation and tumor detection within a single framework.
  • The broader field is shifting from dataset-specific models toward universal or foundation models for diverse medical imaging tasks.

3. Methodology

The method combines CLIP-based class conditioning with standardized CT processing and masked supervision to learn from partially labeled datasets. Its text and vision branches generate class-specific segmentation predictions while addressing label inconsistency and semantic orthogonality.

  • 3.1. Background: A partially labeled dataset omits annotations for at least one of its K unique classes in some data points.
  • 3.1. Background: Existing solutions predict all classes directly or condition predictions with one-hot class vectors, but both ignore anatomical semantics.
  • 3.2. CLIP-Driven Universal Model: The text branch generates CLIP embeddings from medical prompts and combines each class embedding with global image features through an MLP controller.
  • 3.2. CLIP-Driven Universal Model: Medical prompt choice is critical, and all three tested CLIP prompts significantly improve average DSC over one-hot labels and BioBERT.
  • 3.2. CLIP-Driven Universal Model: The vision branch standardizes CT scans before a vision encoder and text-driven segmentor produce class-specific masks.
  • 3.2. CLIP-Driven Universal Model: Masked back-propagation computes BCE loss only for classes annotated in each scan, preventing supervision from unlabeled classes.

4. Experiments & Results

Experiments evaluate the Universal Model on public benchmarks, external datasets, tumor detection, efficiency, and CLIP-based embedding structure. Across these evaluations, it achieves strong segmentation and detection performance while supporting efficient, anatomically structured predictions.

  • Experimental setup: 14 public datasets provide 3,410 training CT scans, while two public and one private datasets are used for testing.Evaluation uses DSC and NSD for segmentation, and sensitivity and specificity for tumor detection.
  • Organ segmentation: The Universal Model ranks first on both the MSD and BTCV challenges, surpassing competing methods.MSD results use the official competition test dataset and five-fold validation comparisons, while BTCV uses five-fold cross-validation.
  • Human comparison: AI pseudo labels show similar intra-observer variability to two human experts for six organs.The evaluated organs are spleen, liver, kidneys, stomach, gallbladder, and pancreas.
  • CLIP embedding: CLIP label encoding produces better feature clustering and anatomically structured semantics than one-hot encoding.The embedding brings related structures such as the right and left kidney closer together and improves performance over one-hot and BioLinkBERT embeddings.

5. Intriguing Properties

The Universal Model combines efficiency, flexible backbones, external generalization, and transfer learning within one segmentation framework.

  • Efficiency: >6× faster than dataset-specific models while maintaining a 74% average DSC.The comparison uses FLOPs as an inference-speed indicator.
  • Expansibility: A CNN-based U-Net backbone achieves 76.73% average DSC across 25 organs and 6 tumors, comparable to Swin UNETR's 76.11%.
  • Generalizability: On unseen 3D-IRCADb and JHH datasets, the Universal Model improves DSC by 5% and 4%, respectively, over previous methods.These evaluations use data from external sources without training exposure.
  • Transferability: Fine-tuning on four TotalSegmentator downstream tasks achieves DSC scores of 86.49%, 89.57%, 94.43%, and 88.95%.The model serves as a pre-training model for segmentation.

6. Conclusion

The paper presents a CLIP-Driven Universal Model for abdominal organ segmentation and tumor detection, using CLIP embeddings to learn anatomical relationships from partially labeled datasets. It reports strong benchmark performance alongside efficiency, generalizability, transferability, and expansibility, while identifying prompt-template choice as an open issue.

  • The Universal Model integrates CLIP embeddings with segmentation models to address label inconsistency and orthogonality in partially labeled datasets.
  • CLIP embeddings establish more meaningful anatomical relationships between organs and tumors than one-hot embeddings, according to the authors.
  • Prompt-template selection remains an open problem for medical image-text vision models and warrants further study.

B. Assembly of Datasets

The dataset assembly unifies heterogeneous public CT datasets with inconsistent annotation protocols, label indices, and organ definitions for training and evaluation.

  • All datasets are mapped into a standard label-index template to address inconsistent annotation protocols.
  • The assembled data include diverse organ and tumor annotations, including pancreas, kidney tumors, liver lesions, and MSD tasks covering six anatomical tasks.
  • Dataset characteristics vary across institutions, scanners, protocols, phases, resolution, and contrast conditions.
  • The assembly uses 14 public datasets for training and MSD, BTCV, 3D-IRCADb, TotalSegmentator, and a private dataset for testing or benchmarking.
  • The private JHH dataset contains 5,038 CT scans with 21 annotated organs and is used to investigate extensibility to new classes.

C.1. Data Augmentation

The implementation standardizes CT volumes and applies foreground-focused, randomized training augmentation, while combining CLIP text features with a Swin UNETR vision encoder.

  • Data Augmentation: CT scans are resliced to isotropic 1.5×1.5×1.5 mm^3 voxels, intensity-clipped to [−175, 250], and normalized to [0, 1].
  • Data Augmentation: Training samples random 96×96×96 foreground- or background-centered patches, with random 90-degree rotations and probabilistic intensity shifts.
  • Text Branch: The CLIP ViT-B/32 text encoder provides class features that can be precomputed because they depend only on the fixed class dictionary.
  • Architecture: Swin UNETR supplies the vision encoder, while a text-based controller combines CLIP embeddings with global image features to generate class-specific parameters.

C.3. Evaluation Metrics

The paper evaluates 3D segmentation results using the Dice similarity coefficient and Normalized Surface Distance. DSC compares ground-truth and predicted voxel values, while NSD details are deferred to cited work.

  • DSC and NSD are used to measure 3D segmentation results.
  • The Dice similarity coefficient is defined as a segmentation metric for the reported 3D results.
  • The DSC calculation uses ground-truth voxel values Y and predicted voxel values Ŷ.

D. Additional Evaluations

Additional evaluations examine transferability, pseudo-label quality, tumor detection, and the effect of CLIP embeddings. The reported comparisons indicate strong transferability, human-comparable pseudo labels, broad tumor detection, and improved anatomical semantics.

  • Transferability: The Universal Model shows the best transferability in per-class evaluations of TotalSegmentator and JHH.Tables 8–13 provide comparisons with Swin UNETR and per-class evaluations.
  • Pseudo-label quality: Universal Model predictions are roughly similar to annotations from two human experts in contour comparisons.The comparison is presented as evidence for the effectiveness of generated pseudo labels.
  • Tumor detection: The method detects small and large kidney and liver tumors without generating false-positive tumor predictions in the illustrated cases.The comparison includes four competitive baseline methods.
  • Embedding ablation: The CLIP-embedding ablation study reports Dice scores for each organ and tumor.The study extends Table 1 and evaluates the contribution of embedding choices.
  • Embedding space: The Universal Model exhibits stronger semantic relationships among anatomical structures than the baseline method in whole-embedding-space visualization.

E. Discussion of Open Challenges

The discussion identifies inconsistent labels and severe long-tail class imbalance as open challenges, while additional evaluations illustrate pseudo-label quality, transferability, and tumor-label limitations. Dataset assembly also exposes incomplete annotations and class-distribution imbalance.

  • Inconsistent label protocols: Inconsistent annotation definitions across datasets require considerable manual correction by experienced radiology experts.For example, AMOS annotates the entire aorta, whereas AbdomenCT-1K omits part of its upper region.
  • Long-tail problem: Severe long-tail class imbalance, especially for small tumors, can lead to unsatisfactory tumor-class performance.The paper uses data augmentation to alleviate this issue and calls for further research.
  • Pseudo-label quality: Contour comparisons show pseudo-label predictions for organs that are roughly comparable to annotations from two human experts.
  • Tumor detection: Qualitative cases show detection of small and multiple liver tumors while avoiding false positives, and detection of kidney tumors and cysts alongside kidneys.
  • Transferability: The Universal Model represents the best transferability in the reported TotalSegmentator vertebrae, cardiac, muscles, organs, and JHH evaluations.These table results are evaluated using DSC where specified.
  • Validation scope: Validation of the six-tumor segmentation accuracy through pathology reports remains a future direction.The model-generated masks for 25 organs across 20 CT datasets achieved a satisfactory level of accuracy.
Loading 2301.00785v5…