PaperlayerView exact revision on arXiv

Source-linked AI summary

MedCLIPSeg: Probabilistic Vision-Language Adaptation for Data-Efficient and Generalizable Medical Image Segmentation

Taha Koleilat, Hojat Asgariandehkordi, Omid Nejati Manzari, Berardino Barile, Yiming Xiao, Hassan Rivaz

arXiv:2602.20423v1cs.CVcs.CL

TL;DR

Medical image segmentation faces limited annotations, ambiguous features, and domain shifts, while dense text-guided use of vision-language models remains underexplored. MedCLIPSeg adapts CLIP with probabilistic bidirectional cross-modal attention and soft patch-level contrastive learning, achieving strong data efficiency and generalization with interpretable uncertainty maps. Across 16 datasets, five modalities, and six organs, it outperforms prior methods in accuracy, efficiency, and robustness while improving uncertainty calibration.

  • Problem

    Medical image segmentation is challenged by limited annotations, ambiguous anatomical features, and domain shifts, while CLIP's potential for dense text-guided segmentation remains underexplored.

  • Method

    MedCLIPSeg adapts CLIP using probabilistic bidirectional cross-modal attention over patch and text tokens, explicit uncertainty modeling, and soft patch-level contrastive learning.

  • Results

    Across 16 datasets spanning five modalities and six organs, MedCLIPSeg achieves strong accuracy, data efficiency, OOD generalization, robustness, and calibrated uncertainty relative to prior methods.

  • Takeaways & Limitations

    MedCLIPSeg demonstrates the potential of probabilistic vision-language modeling for text-driven medical image segmentation with interpretable local reliability estimates.

  • Takeaways & Limitations

    The PVL adapter initializes scalar g through sigmoid(0), assigning equal initial emphasis to the original query and attended output.

Abstract

from arXiv · show

Medical image segmentation remains challenging due to limited annotations for training, ambiguous anatomical features, and domain shifts. While vision-language models such as CLIP offer strong cross-modal representations, their potential for dense, text-guided medical image segmentation remains underexplored. We present MedCLIPSeg, a novel framework that adapts CLIP for robust, data-efficient, and uncertainty-aware medical image segmentation. Our approach leverages patch-level CLIP embeddings through probabilistic cross-modal attention, enabling bidirectional interaction between image and text tokens and explicit modeling of predictive uncertainty. Together with a soft patch-level contrastive loss that encourages more nuanced semantic learning across diverse textual prompts, MedCLIPSeg effectively improves data efficiency and domain generalizability. Extensive experiments across 16 datasets spanning five imaging modalities and six organs demonstrate that MedCLIPSeg outperforms prior methods in accuracy, efficiency, and robustness, while providing interpretable uncertainty maps that highlight local reliability of segmentation results. This work demonstrates the potential of probabilistic vision-language modeling for text-driven medical image segmentation.

1. Introduction

Medical image segmentation is limited by costly and inconsistent annotations, ambiguous boundaries, and domain shifts that undermine reliability. MedCLIPSeg adapts CLIP with probabilistic bidirectional fusion to support data-efficient, uncertainty-aware, and generalizable text-driven segmentation.

  • Motivation: Expensive, inconsistent annotations, ambiguous anatomical boundaries, and scanner, protocol, and population shifts constrain reliable medical image segmentation.These obstacles motivate systems that are data-efficient, uncertainty-aware, and generalizable across domains.
  • MedCLIPSeg: Probabilistic cross-modal fusion models visual-textual variability as distributions, improving feature alignment, OOD generalization, and confidence calibration.Figure 1 contrasts deterministic and probabilistic fusion and reports smaller OOD performance drops and lower Brier scores for the probabilistic formulation.
  • Motivation: Conventional CNN- and ViT-based segmentation models rely on extensive pixel-wise supervision and can be over-confident on OOD inputs and fuzzy boundaries.Their deterministic feature learning does not adequately account for ambiguity or local disagreement among features.
  • Vision-language opportunity: CLIP patch tokens offer spatial semantics for label-efficient, natural-language-driven dense localization, but medical fine-grained segmentation remains underexplored.Biomedical vision-language models provide cross-modal alignment while requiring adaptation for subtle anatomical differences and detailed descriptions.
  • MedCLIPSeg: MedCLIPSeg uses probabilistic, bidirectional vision-language fusion through a PVL adapter that learns confidence-weighted attention between image patches and text tokens.Variational Key modeling reduces over-confidence, while Monte Carlo Value sampling produces mean masks and pixel-level uncertainty maps.

2. Related Work

Medical segmentation has progressed from CNN and Transformer architectures toward cross-modal methods that add semantic priors. Related work increasingly uses vision-language models, prompts, and few-shot support sets, but adaptation remains necessary for fine-grained medical semantics and robust domain transfer.

  • Medical Image Segmentation: CNN and Transformer architectures established strong medical segmentation foundations through skip connections, multi-scale context modeling, and long-range dependencies.Examples include U-Net variants, DeepLab, TransUNet, Swin-UNet, HiFormer, and UNETR.
  • Medical Image Segmentation: Vision-only approaches often depend on low-level appearance features and show limited robustness across scanners and imaging protocols.This motivates high-level semantic priors from cross-modal learning.
  • Vision-Language Models: Biomedical vision-language models such as BiomedCLIP, PubMedCLIP, and UniMedCLIP provide global alignment but require adaptation for fine-grained anatomical semantics.Their clinical image-text pretraining supports domain-specific learning, while dense segmentation demands more detailed representations.
  • Prompt-based Segmentation: Prompt-based segmentation methods extend vision-language models with lightweight decoders, cross-attention, and fine-grained localization strategies.Examples include CLIPSeg, CRIS, LAVT, DenseCLIP, ZegCLIP, SAN, and CAT-Seg.
  • Prompt-based Segmentation: Geometric-prompt methods may introduce instability, while few-shot frameworks segment unseen classes or modalities using small support sets without additional training.BiomedParse additionally explores structured knowledge parsing across modalities through natural language.

3. Methodology

MedCLIPSeg adapts CLIP for text-driven segmentation through probabilistic, bidirectional vision–language fusion across multiple layers. It models uncertainty in attention and value representations, uses residual stabilization, and produces segmentation from pixel–text similarity with soft patch-level contrastive alignment.

  • Probabilistic Multi-modal Adaptation: MedCLIPSeg inserts PVL Adapters between CLIP’s vision and language encoders for bidirectional multimodal interaction.The adapters project both modalities into a shared lower-dimensional space and are applied at multiple encoding layers.
  • Probabilistic Multi-modal Adaptation: Keys and Values are represented with learnable means and variances, allowing attention to penalize uncertain correspondences and sample stochastic value features.Softplus converts predicted log-variances into variance values, while the variance term uses squared query features to quantify uncertainty interactions.
  • Uncertainty Estimation: Monte Carlo forward passes produce posterior samples whose mean yields segmentation masks and whose predictive entropy quantifies total uncertainty.The authors report that 30 stochastic forward passes provide stable uncertainty estimates, combining aleatoric and epistemic uncertainty.
  • Fusion Stabilization: Residual gating gradually increases reliance on attended features, reducing early training instability while preserving a balanced initial mixture of original and attended representations.The learnable gate is initialized through sigmoid(0), giving equal initial emphasis to both inputs.
  • Segmentation via Pixel-Text Similarity: After fusion, normalized text and upsampled visual patch embeddings are combined by dot product and bilinear interpolation to produce segmentation logits.The framework preserves patch-level spatial information while aligning visual regions with the text embedding.
  • Soft Patch-level Contrastive Loss: A soft patch-level contrastive loss aligns text with regional image representations and replaces hard targets with targets derived from similarities among textual prompts.The approach averages image patch embeddings to preserve local semantics while reducing token-level noise.

4. Experiments and Results

Experiments evaluate MedCLIPSeg under limited supervision, cross-dataset domain shifts, component ablations, prompt variations, backbone choices, and uncertainty reliability. Across these settings, the reported results show improved data efficiency, generalization, and calibrated uncertainty.

  • Experimental Setup: 10%, 25%, and 50% training-data settings evaluate data efficiency, with fully supervised training serving as an upper-bound reference.Models are assessed using varying fractions of training data and pixel-level and textual annotations in the fully supervised setting.
  • Data Efficiency Evaluation: 2–3% gains at 10% data and 3–4% at 50% data over CAT-Seg demonstrate stronger data-efficient segmentation.MedCLIPSeg consistently outperforms unimodal and multimodal baselines, including the CLIP-based CAT-Seg.
  • Domain Generalization: 85.7% DSC on BUSI, 84.4% on BUSUC, 90.2% on Kvasir-SEG, 88.0% on BTMRI, and 92.5% on ISIC show cross-dataset performance under distribution shifts.The model preserves contour fidelity and segmentation quality when evaluated on unseen target datasets.
  • Effectiveness of Key Design Components: −7.9% ID and −23.8% OOD DSC after removing PVL Adapters identify them as the largest component ablation drop.Deterministic attention additionally reduces OOD DSC by 15.9%, while excluding LSoftCon decreases HM DSC by 1.92%.
  • Ablation Studies: 83.76% HM DSC at Layer 10 is the best result for layer interventions, while β=2.35 gives the best HM DSC among tested confidence weights.Deeper interventions improve ID and OOD segmentation until Layer 10, followed by a slight final-layer drop.
  • Uncertainty Visualization and Reliability: 87.57% and 80.41% Spearman correlations for ID and OOD uncertainty, respectively, accompany Brier-score reductions from 23.9% and 25.3% to 11.1% and 11.8%.Uncertainty concentrates near anatomical boundaries and regions prone to expert disagreement, while probabilistic modeling improves calibration.

5. Conclusion

MedCLIPSeg combines probabilistic confidence-weighted attention, bidirectional vision–language fusion, and soft patch-level contrastive learning for data-efficient, generalizable, uncertainty-aware medical segmentation.

  • MedCLIPSeg uses a probabilistic Vision–Language adapter for confidence-weighted attention and explicit uncertainty estimation.
  • Across six organs and five imaging modalities, MedCLIPSeg achieves strong segmentation performance, data efficiency, out-of-distribution generalization, and calibrated uncertainty.

Supplementary Material

The supplementary material details benchmark coverage, computational evaluation, automated caption generation, training settings, and prompt-design ablations.

  • Supplementary Material: The benchmarks cover six organs, five imaging modalities, and data-efficiency, fully supervised, and domain-generalization settings.
  • Supplementary Material: Computational complexity is compared across methods using parameter footprint, FLOPs, and inference time under identical BUSI test conditions.
  • Supplementary Material: Automated captions are generated by filling LLM-created templates with image- and mask-derived attributes such as class, location, number, shape, and color.
  • Supplementary Material: Training uses UniMedCLIP ViT-B/16, PubMedBERT, Adam optimization, cosine annealing, equally weighted Dice and binary cross-entropy losses, and λSoftCon = 0.1.
  • Supplementary Material: Prompt ablations vary conciseness, spatial specificity, verbosity, contradictions, and noise to assess their effects on segmentation and generalization.

F. 3D applicability

MedCLIPSeg extends to 3D segmentation by replacing the 2D image encoder with a 3D vision–language backbone, without changing the core method.

  • F. 3D applicability: MedCLIPSeg uses M3D-CLIP to replace the 2D image encoder and computes 3D segmentations from global text tokens and voxel features.
  • F. 3D applicability: 88.72% DSC is achieved on the CHAOS CT Liver dataset using the M3D-Seg data split.
  • F. 3D applicability: Average runtime per volume over 20 test volumes supports practical 3D feasibility, while further 3D analysis remains future work.
  • F. 3D applicability: Using 5–10 Monte Carlo samples only marginally changes DSC and uncertainty estimates, reducing computational cost relative to 30 samples.

H. Effect of λSoftCon

Ablations identify λSoftCon = 0.1, visual-first bidirectional fusion, average pooling, dimension 256, and difference-based uncertainty weighting as favorable design choices for ID and OOD segmentation.

  • H. Effect of λSoftCon: λSoftCon = 0.1 yields the highest harmonic mean between in-distribution and out-of-distribution performance.
  • H. Effect of λSoftCon: Removing the contrastive loss degrades both in-distribution and out-of-distribution performance by eliminating semantic alignment across image–text patches.
  • H. Effect of λSoftCon: Sigmoid(0) gate initialization achieves the highest in-distribution, out-of-distribution, and harmonic mean DSC scores.
  • H. Effect of λSoftCon: Vision First fusion achieves the best in-distribution, out-of-distribution, and harmonic mean results among the tested attention orders.
  • H. Effect of λSoftCon: Average Pooling provides the highest in-distribution, out-of-distribution, and harmonic mean DSC scores among the evaluated pooling strategies.
  • H. Effect of λSoftCon: A shared PVL-adapter dimension of 256 best balances in-distribution and out-of-distribution segmentation accuracy.
  • H. Effect of λSoftCon: Difference-based uncertainty weighting performs best across in-distribution, out-of-distribution, and harmonic mean DSC scores, whereas omitting uncertainty reduces domain-shift robustness.

O. Error vs. Uncertainty Correlation

MedCLIPSeg’s uncertainty maps closely track pixel-level segmentation error across imaging domains. The probabilistic design also improves confidence calibration and boundary handling under in-domain and cross-domain conditions.

  • 0.9248, 0.9921, 0.9201, and 0.9885 Pearson correlations link uncertainty with error in breast ultrasound, polyp endoscopy, skin dermatoscopy, and brain MRI, respectively.All correlations are statistically significant with p < 0.001.
  • Probabilistic MedCLIPSeg reduces false positives in non-lesion regions and false negatives around missed lesion boundaries compared with the deterministic variant.The result is lower combined error, smoother contours, and more balanced confidence calibration.
  • Across polyp endoscopy and breast ultrasound domain shifts, uncertainty remains concentrated along lesion boundaries, indicating calibrated confidence and robust generalization.The evaluated shifts include changes in texture, lighting, and instrument artifacts.

R. Effect of Supervised Segmentation

Supervised segmentation annotations are essential for reliable medical image segmentation because medical boundaries are subtle, low-contrast, and ambiguous. Removing the segmentation loss causes severe performance collapse, while MedCLIPSeg performs strongly against uncertainty-aware and domain-generalization baselines.

  • Removing LSeg and training only with LSoftCon reduces performance to approximately 20% DSC in-distribution and below 13% out-of-distribution.The ablation demonstrates that contrastive supervision alone is insufficient for this medical segmentation setting.
  • Adding LSeg yields gains exceeding +69% DSC in-distribution and +66% out-of-distribution.The supplied passage attributes these gains to the spatial guidance provided by segmentation annotations.
  • Pixel-accurate supervision helps distinguish anatomy from imaging artifacts and surrounding tissues when boundaries are subtle, low-contrast, and ambiguous.The passage identifies this spatial guidance as necessary for clinically meaningful decision boundaries.
  • MedCLIPSeg consistently outperforms alternative baselines in both segmentation robustness and uncertainty quality.The comparison includes nnUNet checkpoint ensembling, Ariadne’s Thread, EviVLM, VLSM-Ensemble, and deterministic or evidential variants.

T. Additional Evaluation Metrics

Additional domain-generalization evaluation reports Sensitivity, Specificity, and F1 for leading baselines. The supplied prompt examples span multiple lesion types, organs, imaging modalities, and spatial descriptions.

  • Sensitivity, Specificity, and F1 are reported for the top three baselines under domain generalization.These metrics support evaluation of boundary localization under domain shifts.
  • The representative prompts describe polyps, pulmonary infection, masses, melanomas, malignant and benign lesions, tumors, and normal MRI findings.The examples use varied attributes such as color, shape, size, location, and lesion status.
  • Prompt examples specify lesion locations including the center, left, right, upper section, top right, and center region of an image.The supplied examples include polyps, tumors, lesions, melanoma, and pituitary tumor descriptions.
  • The prompt set also includes a normal MRI description and additional polyp, melanoma, tumor, and lesion formulations.These examples broaden the textual conditions used across datasets.

V. Per-dataset Efficiency Results

MedCLIPSeg is evaluated across six medical image segmentation benchmarks using 10%, 25%, 50%, and 100% labeled supervision. It achieves the highest or second-highest performance across nearly all datasets and label fractions, indicating strong data efficiency and cross-domain adaptability.

  • MedCLIPSeg consistently achieves the highest or second-highest performance across nearly all datasets and labeled-data fractions.The evaluated fractions are 10%, 25%, 50%, and 100% supervision across six benchmarks.
  • DSC and NSD measure volumetric overlap and boundary accuracy across the six benchmarks.The tables report these metrics for 10%, 25%, 50%, and 100% labeled supervision.
  • The comparisons cover unimodal CNN and transformer baselines alongside text-driven and CLIP-based model families.This evaluation setup is used to examine behavior under annotation scarcity and cross-domain variation.
Loading 2602.20423v1…