PaperlayerView exact revision on arXiv

Source-linked AI summary

SegVol: Universal and Interactive Volumetric Medical Image Segmentation

Yuxin Du, Fan Bai, Tiejun Huang, Bo Zhao

arXiv:2311.13385v5cs.CV

TL;DR

Volumetric medical segmentation lacks a foundation model that supports broad anatomical categories and easy interaction, while existing methods face limitations in generalization, 3D prompting, and inference. SegVol combines spatial and semantic prompts with a zoom-out-zoom-in mechanism, and it surpasses other SAM-like methods across 22 segmentation tasks. Its reported scope also includes CT-trained segmentation of MRI, while referring volumetric segmentation remains a limitation.

  • Problem

    Existing volumetric segmentation methods face challenges with inconsistent category labels, complex structures, costly inference, and limited direct 3D or semantic prompting.

    Source: §1 · pp. 1–2

  • Method

    SegVol combines spatial, semantic, and multi-prompts with a zoom-out-zoom-in inference mechanism for volumetric segmentation.

    Source: §1 · pp. 2–3

  • Results

    Across 22 volumetric segmentation tasks, SegVol surpassed other SAM-like interactive segmentation methods by a large margin.

    Source: §1 · p. 2

  • Takeaways & Limitations

    A model trained only on CT data can also segment MRI using semantic and spatial prompts.

    Source: §4 · p. 9

    SegVol still has a gap in referring volumetric segmentation; the authors identify training on data with diverse semantic and spatial prompts as future work.

    Source: §4 · p. 9

Abstract

from arXiv · show

Precise image segmentation provides clinical study with instructive information. Despite the remarkable progress achieved in medical image segmentation, there is still an absence of a 3D foundation segmentation model that can segment a wide range of anatomical categories with easy user interaction. In this paper, we propose a 3D foundation segmentation model, named SegVol, supporting universal and interactive volumetric medical image segmentation. By scaling up training data to 90K unlabeled Computed Tomography (CT) volumes and 6K labeled CT volumes, this foundation model supports the segmentation of over 200 anatomical categories using semantic and spatial prompts. To facilitate efficient and precise inference on volumetric images, we design a zoom-out-zoom-in mechanism. Extensive experiments on 22 anatomical segmentation tasks verify that SegVol outperforms the competitors in 19 tasks, with improvements up to 37.24% compared to the runner-up methods. We demonstrate the effectiveness and importance of specific designs by ablation study. We expect this foundation model can promote the development of volumetric medical image analysis. The model and code are publicly available at: https://github.com/BAAI-DCAI/SegVol.

1 Introduction

SegVol is presented as a universal, interactive model for volumetric segmentation, addressing gaps in anatomical-category generalization, 3D prompting, and inference efficiency. It combines semantic and spatial prompts with a zoom-out-zoom-in mechanism and is evaluated across 22 tasks.

1 Introduction

  • Volumetric segmentation extracts 3D regions such as organs, lesions, and tissues, supporting clinical applications including monitoring, surgical planning, diagnosis, and therapy optimization.The cited applications include tumor monitoring, surgical planning, disease diagnosis, and therapy optimization.Source: §1 · p. 1
  • Segmentation datasets use inconsistent category labels, making task-specific models difficult to generalize across datasets and motivating universal anatomical semantics.For example, CT-ORG labels “lungs,” while LUNA16 separates them into left and right lung.Source: §1 · p. 1
  • Prior methods struggle with complex structures and costly sliding-window inference, while existing prompt-based approaches often lack direct 3D processing or anatomical-semantic understanding.These limitations are described as constraining challenging tasks and human-interactive segmentation.Source: §1 · pp. 1–2
  • Across 22 tasks, SegVol surpassed other SAM-like methods by a large margin while supporting over 200 anatomical categories with spatial, semantic, and combined prompts.Combined prompts are described as enabling high-precision segmentation and semantic disambiguation.Source: §1 · pp. 2–3
  • The zoom-out-zoom-in mechanism is designed to reduce computational cost while preserving precise segmentation.The overview describes a rough zoom-out mask followed by zoom-in refinement of the identified region of interest.Source: Figure 1 · p. 2, §1 · p. 3

2 Methodology

SegVol combines a large, partly unlabeled CT dataset with text and spatial prompting, prompt-mask training, and a coarse-to-fine inference strategy. Its training uses masked-image pretraining followed by supervised fine-tuning with BCE and Dice losses.

2.1 Dataset Construction

  • The authors collected 25 public volumetric segmentation datasets spanning over 200 anatomical categories and used FH-generated pseudo masks to supplement unlabeled categories and reduce spurious correlations.Pseudo masks are replaced by ground truth when available, tiny structures are filtered, and masks are refined with dilation and erosion.Source: §2.1 · p. 3

2.2 Model Architecture

  • A 3D ViT image encoder, CLIP text encoder, prompt encoder, and mask decoder combine image and prompt embeddings to predict segmentation masks.The text encoder is frozen during training, and a parallel text input branch enhances text information.Source: §2.2 · pp. 3–4

2.3 Prompt Generation

  • Training pairs are generated from point, bounding-box, and text prompts, with pseudo-label masks receiving point and box prompts but no text prompts.Box prompts include random jitter; masks with IoU greater than 0.9 against an overlapping pseudo mask are integrated into the target.Source: §2.3 · p. 4

2.4 Zoom-out-zoom-in Mechanism

  • Zoom-out inference makes a coarse mask, then zoom-in inference crops and refines its region of interest before back-filling the prediction.Multi-size training resizes CTs for the zoom-out view and crops them into cubes for the zoom-in view; the mechanism is designed to reduce cost while producing precise ROI segmentation.Source: §2.4 · p. 4

2.5 Loss Function

  • Image-encoder pretraining uses masked-voxel reconstruction, while supervised fine-tuning combines binary cross-entropy and Dice losses.The pretraining passage identifies input and predicted voxel values, masked voxels, and the pretraining dataset; fine-tuning uses predicted and ground-truth masks.Source: §2.5 · p. 5

3 Experiments

Experiments compare SegVol with SAM-like methods across volumetric segmentation tasks, then examine its inference mechanism, training-data scale, prompt combinations, and case-study capabilities. The results include comparisons on organs and lesions and tests of semantic disambiguation and category identification.

3.1 Experimental Setup

  • The setup pre-trains SegVol on 96K CTs for 2,000 epochs, then fine-tunes on 25 labeled datasets for 270 epochs.Fine-tuning uses a frozen text encoder, batch size 32, and input size (32, 256, 256).Source: §3.1 · p. 5

3.2 Compared with SAM-like Interactive Methods

  • SegVol leads most organ and lesion tasks, exceeding SAM-MED2D by 19.25% average Dice on AMOS22 and SAM-MED3D by 4.28% on SegTHOR.On ULS23, SegVol is comparable to MedSAM for small patch-like masks.Source: Table 2 · p. 6, §3.2 · p. 7

3.3 Ablation Studies

  • The zoom-out-zoom-in mechanism achieves the best average Dice and competitive inference speed against resize and sliding-window strategies.It uses one global 3D CT inference followed by ROI scanning with dozens of windows, rather than scanning the whole volume with thousands of windows.Source: §3.3 · p. 7, Table 3 · p. 7
  • Dice score increases as training data scales from one dataset toward the final model trained on 25 datasets, especially with text prompts.The ablation compares models trained on 1, 2, and 8 datasets with the final 25-dataset model.Source: Figure 3 · p. 7, §3.3 · p. 8
  • Across 19 tasks, bbox+text raises average Dice by 5.85% over text alone, while point+text raises it by 4.62% over point alone.Spatial prompts locate a region; semantic prompts clarify its anatomical target.Source: Figure 3 · p. 7, §3.3 · p. 8

3.4 Case Studies

  • Semantic prompts disambiguate spatial prompts that could plausibly select different structures, such as a kidney tumor, left kidney, or both kidneys.The paper contrasts this with SAM returning multiple masks for different plausible targets.Source: §3.4 · p. 8
  • Given a spatial prompt on a liver tumor, SegVol assigns prediction scores of 0.619 to tumor and 0.339 to liver based on their spatial relationship.The identification experiment decodes semantic prompts from a category set and applies softmax to obtain category probabilities.Source: §3.4 · p. 8

4 Discussion

SegVol's discussion highlights gains from scaling, cross-modality generalization, and prompt-based interpretation, while noting a remaining gap in referring volumetric segmentation.

4 Discussion

  • More training data significantly improves 3D segmentation performance, and the study reports that SegVol has not yet reached its performance ceiling.The authors expect further improvement with more data and computational resources.Source: §4 · p. 8
  • The discussion illustrates how semantic prompts can disambiguate spatial prompts, while spatial-prompt results can also yield ranked anatomical-category predictions.Figure 4 contrasts prompt-conditioned segmentation cases; Figure 5 ranks the top eight categories by classification probability.Source: Figure 4 · p. 9, Figure 5 · p. 9
  • A model trained only on CT can segment MRI using semantic and spatial prompts, while joint multi-modality training remains future work.The authors present this generalization as evidence that SegVol understands human anatomical structure.Source: §4 · p. 9
  • SegVol still has a gap relative to referring volumetric segmentation; the authors propose diverse semantic and spatial prompt training data as future work.Source: §4 · p. 9
  • The authors expect the foundation model to benefit clinical studies and applications, and report no obvious negative societal impact.Source: §4 · p. 10

5 Conclusion

The conclusion presents SegVol as a universal, interactive model for volumetric segmentation across more than 200 anatomical categories. It summarizes its training data and inference design, and reports evaluation on 22 tasks.

5 Conclusion

  • SegVol supports spatial- and semantic-prompt segmentation of more than 200 anatomical categories.Source: §5 · p. 10
  • The model is trained using 90K unlabeled CTs and 25 open-source medical datasets, with a zoom-out-zoom-in mechanism for efficient, precise inference in regions of interest.Source: §5 · p. 10
  • Experiments cover 22 segmentation tasks, and ablations examine the zoom-out-zoom-in mechanism, dataset scale, and multi-prompt combination strategy.Source: §5 · p. 10

A Dataset Details and Availability

The dataset details describe 25 open-source datasets for supervised fine-tuning, supplemented by publicly accessible unlabeled CT volumes and coverage across major body regions.

A Dataset Details and Availability

  • The authors collect 25 open-source datasets for supervised fine-tuning and additional open-source datasets for comparative experiments, alongside 90K unlabeled CTs from Radiopaedia.Source: A Dataset Details and Availability · p. 15
  • The segmentation datasets cover the head, neck, thorax, abdomen, and pelvis, with over 200 organ and tissue categories and 28 lesion tasks.Source: A Dataset Details and Availability · p. 15
  • Table 4 summarizes dataset information for supervised fine-tuning and experiments, while Table 5 reports dataset availability.Source: Table 4 · p. 15

B Training Algorithm

The training procedure combines ground-truth and pseudo-mask supervision for each case, using separate prompted predictions and a weighted sum of their losses.

B Training Algorithm

  • Each case pairs an image with ground-truth and pseudo-mask sets, and training minimizes the weighted sum of ground-truth and pseudo losses.The ground-truth loss uses a sampled prompt; the pseudo loss uses a pre-designed prompt.Source: B Training Algorithm · pp. 15, 17
  • The supervised fine-tuning joint dataset groups structures into 47 important spatial regions, with examples shown as 2D slices and 3D shapes.Source: Figure 6 · p. 16, Figure 7 · p. 16
  • A reinforcement branch for semantic prompts combines text-embedding similarity with mask prediction before interpolation.Source: B Training Algorithm · p. 17
  • The ground-truth training loop uses six combinations of text, point, and bounding-box prompts and accumulates Dice and binary cross-entropy losses.Source: B Training Algorithm · p. 18
  • For pseudo-mask training, the loop randomly selects a mask, generates a spatial prompt, and accumulates Dice and binary cross-entropy losses.Source: B Training Algorithm · p. 18

C Additional Experimental Analysis

Additional experiments compare SegVol with task-specific and SAM-like interactive models, test the zoom-out-zoom-in mechanism and few-shot adaptation, and assess prompt consistency and MRI transfer.

Task-specific comparisons

  • Visual comparisons report more precise tumor segmentation than nnU-Net, better pancreas modeling than the baselines, and more consistent stomach modeling than other methods.For aorta and left kidney structure integrity, SegVol significantly outperforms 3DUX-NET and SwinUNETR and is comparable to nnU-Net.Source: C Additional Experimental Analysis · p. 18, Figure 10 · p. 20, Figure 11 · p. 21

Training algorithm

  • The training illustration combines ground-truth loss using sampled prompts with pseudo-mask loss using fixed prompts, then minimizes their weighted sum.Each illustrated training case includes an image, a ground-truth mask set, and a pseudo-mask set.Source: Figure 8 · p. 19

Task-specific comparisons

  • Across eight tasks, SegVol generally outperforms single-dataset task-specific models; on three lesion tasks, its average Dice score is 14.76% higher than nnU-Net.The comparisons cover organ and lesion tasks; Figure 9 summarizes Dice-score distributions, and Table 7 compares lesion-task averages.Source: C Additional Experimental Analysis · p. 18, Figure 9 · p. 19, Table 7 · p. 20

Prompt consistency

SAM-like comparisons

  • SegVol was compared with five SAM-like interactive methods on AMOS22[44], ULS23[74], and SegTHOR[75]; on the AMOS22[44] validation set under Sec. 3.2 settings, it used less computational cost while achieving much better performance.The comparison used bbox+text prompting; relatively poor text-prompt results on ULS23[74] were attributed to unclear dataset categories.Source: C Additional Experimental Analysis · p. 20

Zoom-out-zoom-in ablation

MRI generalization

Few-shot fine-tuning

  • Fine-tuning on dozens of cases approaches joint fine-tuning on all datasets for MSD-spleen, while FLARE22 performs much better with all-dataset fine-tuning.The few-shot experiments use 40 FLARE22 training cases and 32 MSD-spleen training cases.Source: Table 8 · p. 20, C Additional Experimental Analysis · p. 27

D Evaluation Metrics

The paper evaluates segmentation overlap with the Dice Similarity Coefficient, also called the Dice score.

Dice score

  • The Dice Similarity Coefficient is defined as DSC = 2|X∩Y|/(|X|+|Y|), where X is the predicted segmentation set and Y is the ground-truth set.The numerator uses the intersection cardinality; the denominator sums the cardinalities of both sets.Source: D Evaluation Metrics · p. 30
  • The metric compares the predicted mask and ground truth through their set intersection and respective sizes.The paper denotes these sets as X and Y, respectively.Source: D Evaluation Metrics · p. 30
  • The Dice score measures similarity between predicted and true segmentation and is suited to evaluating overlap in binary segmentation results.The paper identifies it as a commonly used image-segmentation metric.Source: D Evaluation Metrics · p. 31

E Additional Discussion

SegVol is presented as a universal interactive volumetric segmentation model trained using 90K unlabeled CTs and 25 open-source medical datasets. The authors describe clinical applications and identify complex referring-expression segmentation as a remaining gap.

Discussion

  • SegVol uses 90K unlabeled CTs and 25 open-source medical datasets to support accurate segmentation of more than 200 anatomical targets, with reported state-of-the-art performance in several comparisons.The authors also characterize its architecture as simple relative to other volumetric segmentation methods.Source: E Additional Discussion · p. 31

Clinical implications

  • The authors present tumor-change monitoring and surgical-pathway planning as potential clinical uses of precise interactive segmentation.They describe clinician assistance with identifying and quantifying tumor location, size, and shape changes, and segmenting relevant anatomy in high-resolution 3D images.Source: E Additional Discussion · p. 31

Limitations

  • SegVol still falls short of referring-expression segmentation involving complex semantic information and logical relationships; the authors leave this for future work.They identify curated data combining semantic and spatial prompts as a promising route.Source: E Additional Discussion · p. 31

NeurIPS Paper Checklist

The checklist reports that the paper’s claims reflect its contributions, describes experimental details and limitations, and provides affirmative responses on reproducibility and open access.

Claims and scope

  • The checklist says the abstract and introduction accurately reflect the paper’s contributions and scope.The paper’s main claims are marked as matching its contributions.Source: NeurIPS Paper Checklist · p. 32

Limitations and experimental details

  • The checklist states that the paper discusses limitations and provides experimental-setting and computer-resource information.The checklist directs readers to Sections 4, 3.1, and E for these disclosures.Source: NeurIPS Paper Checklist · pp. 32, 34, 35

Other checklist disclosures

  • The paper marks theory and human-subjects research as not applicable, reports no obvious negative societal impact, and states that asset creators and licenses are acknowledged.The societal-impact response is yes, and the misuse-safeguards question is marked not applicable.Source: NeurIPS Paper Checklist · pp. 33, 35, 36, 37

Reproducibility and access

  • The checklist reports that datasets are open-source, while the model, training algorithm, and experimental setup are described and the trained model and code are to be released after review.The paper marks both reproducibility disclosure and open access to data and code as yes.Source: NeurIPS Paper Checklist · pp. 33–34

Statistical reporting

  • The authors report multiple experiment runs, violin plots for 32 tasks, and medians and quartiles in comparative tables as statistical information.The referenced plots are Figures 2 and 9, and the tables are Tables 9 and 10.Source: NeurIPS Paper Checklist · p. 34
Loading 2311.13385v5…