PaperlayerView exact revision on arXiv

Source-linked AI summary

Uncertainty-Aware Vision-Language Segmentation for Medical Imaging

Aryan Das, Tanishq Rachamalla, Koushik Biswas, Swalpa Kumar Roy, Vinay Kumar Verma

arXiv:2602.14498v2cs.CVcs.LG

TL;DR

Medical vision-language segmentation must remain accurate and reliable when clinical images are poor and labelled data are limited. The paper combines structured cross-modal fusion with uncertainty-aware optimization, achieving strong results across multiple datasets while emphasizing computational efficiency and modality alignment.

  • Problem

    Existing vision-language segmentation methods largely neglect uncertainty modelling, while effective alignment between visual features and language cues remains challenging.

  • Method

    The framework uses MoDAB and lightweight SSMix for structured image-text fusion and long-range dependency modelling, with SEU Loss jointly guiding spatial, spectral, and uncertainty information.

  • Results

    The model achieves superior segmentation performance across QATA-COVID19, MosMedData++, and Kvasir-Seg, including 92.24% Dice and 84.9% mIoU in one evaluation.

  • Takeaways & Limitations

    The results support incorporating uncertainty modelling and structured modality alignment into efficient medical vision-language segmentation.

Abstract

from arXiv · show

We introduce a novel uncertainty-aware multimodal segmentation framework that leverages both radiological images and associated clinical text for precise medical diagnosis. We propose a Modality Decoding Attention Block (MoDAB) with a lightweight State Space Mixer (SSMix) to enable efficient cross-modal fusion and long-range dependency modelling. To guide learning under ambiguity, we propose the Spectral-Entropic Uncertainty (SEU) Loss, which jointly captures spatial overlap, spectral consistency, and predictive uncertainty in a unified objective. In complex clinical circumstances with poor image quality, this formulation improves model reliability. Extensive experiments on various publicly available medical datasets, QATA-COVID19, MosMed++, and Kvasir-SEG, demonstrate that our method achieves superior segmentation performance while being significantly more computationally efficient than existing State-of-the-Art (SoTA) approaches. Our results highlight the importance of incorporating uncertainty modelling and structured modality alignment in vision-language medical segmentation tasks. Code: https://github.com/arya-domain/UA-VLS

1. Introduction

Medical vision-language segmentation addresses limited clinical annotations and poor visual quality by combining images with textual clinical cues. The paper further targets uncertainty-aware training and efficient visual-language alignment for reliable segmentation.

  • Clinical image segmentation supports diagnosis, surgical planning, and research but often depends on extensive labelled data that is limited in practice.
  • Natural language reports or anatomical queries provide auxiliary supervision and contextual cues for vision-language segmentation.This paradigm connects low-level visual features with high-level clinical concepts and can reduce task-specific supervision needs.
  • Existing multimodal methods largely neglect uncertainty modelling despite the need for accurate and reliable predictions in clinical applications.Uncertainty-aware guidance can focus learning on ambiguous regions and reduce overconfident errors under noisy data.
  • The paper proposes MoDAB and SSMix for structured multimodal fusion and long-range dependency modelling.These modules are designed to improve alignment between visual features and language cues with limited model parameters.
  • SEU Loss unifies spatial, spectral, and uncertainty guidance within a single optimization objective.

2. Related Work

Prior work spans unimodal segmentation, efficient state-space sequence models, vision-language representation learning, multimodal segmentation, and uncertainty estimation. These lines of research establish the components that the paper combines for medical vision-language segmentation.

  • Unimodal Segmentation Models: Unimodal segmentation progressed from U-Net and enhanced convolutional variants to hybrid CNN-Transformer architectures for global and multiresolution context.
  • State Space Models: State Space Models offer linear-time, memory-efficient long-sequence modelling while capturing long-range dependencies.
  • Vision-Language Segmentation Models: Vision-language models such as ConVIRT, GLoRIA, CLIP, and BiomedCLIP learn joint image-text representations from paired medical images and reports.Their alignment objectives primarily operate at global or hierarchical representation levels.
  • Vision-Language Segmentation Models: Medical vision-language segmentation methods use cross-modal reasoning, report-based supervision, multimodal attention, and surrogate annotations to improve spatial localization.
  • Vision-Language Segmentation Models: Later frameworks tighten multimodal coupling through multilevel contrastive learning, cross-modal reconstruction, and shared latent representations.
  • Uncertainty in Medical Segmentation: Medical segmentation research uses frequency-domain dropout and entropy maps to estimate uncertainty, calibrate predictions, and identify ambiguous regions.

3. Methodology

The method encodes visual and textual modalities, fuses them through MoDAB with attention and SSMix, and decodes aligned features into segmentation masks. SEU Loss unifies spatial, spectral, and uncertainty guidance for medical vision-language segmentation.

  • Modalities Encoding: ConvNeXt-Tiny extracts hierarchical multiscale visual features, while frozen BioViL CXR-BERT produces contextualized token embeddings for cross-modal alignment.Visual features from four stages preserve fine-grained and abstract semantics; text embeddings encode contextual clinical information.
  • Modality Decoding Attention Block: MoDAB combines visual self-attention, visual-text cross-attention with sinusoidal positional encodings, and SSMix for structured multimodal fusion.Self-attention captures spatial dependencies, while cross-attention uses visual features as queries and state-space-enhanced text as keys and values.
  • Modality Decoding Attention Block: Cross-attended visual features are residually combined with the original visual representations through a learnable scalar, producing features containing spatial dependencies and aligned textual cues.These enriched features are propagated to the decoder for segmentation mask reconstruction.
  • State Space Mixer: SSMix transforms projected text using convolutional operations, selective state-space scanning, gating, and learned dynamics to capture global and local dependencies efficiently.The resulting text representation is fused with visual features and passed to the decoder.
  • Decoder: The decoder progressively restores spatial resolution through transposed-convolution upsampling, encoder-feature concatenation, convolutional refinement, pixel shuffling, pooling, and a final prediction layer.This coarse-to-fine pathway preserves semantic and spatial detail during segmentation reconstruction.
  • Objective Function: SEU Loss is a unified objective that integrates spatial, spectral, and probabilistic priors to guide anatomically precise, structurally consistent, and uncertainty-aware predictions.It is defined on predicted segmentation maps and one-hot ground truth, with entropy-based regularization encouraging lower-entropy predictions.

4. Experiments

Experiments evaluate the model on three publicly available vision-language medical segmentation datasets using a consistent training setup and standard efficiency metrics.

  • Datasets: Three datasets—QaTa-COV19, MosMed++, and Kvasir-SEG—were used to evaluate medical vision-language segmentation.The datasets were augmented with concise natural-language descriptions to support multimodal training.
  • Datasets: QaTa-COV19 contains 9,258 chest X-ray images with manually annotated COVID-19 lesion regions and textual descriptions.
  • Datasets: MosMed++ contains 2,729 annotated chest CT slices from patients diagnosed with COVID-19, enriched with severity scores and textual descriptions.
  • Datasets: Kvasir-SEG contains 1,000 high-resolution endoscopy images with pixel-wise polyp annotations and location-focused captions.Image resolutions range from 332×487 to 1920×1072 pixels.
  • Training setup: Images were resized to 224×224 pixels, trained with batch size 32, and run for up to 200 epochs with early stopping.A minimum of 20 epochs was enforced, with patience of 20 epochs for early stopping.
  • Evaluation: Table 1 compares Dice, mIoU, trainable parameters, and FLOPs across monomodal and multimodal SoTA methods on all three datasets.The table marks first-, second-, and third-best values using black, green, and blue, respectively.

5. Results and Discussion

The proposed model produces more focused attention and more precise segmentation than SoTA baselines, while achieving strong accuracy with substantially lower computational cost across datasets.

  • Qualitative analysis: The model exhibits more focused and semantically aligned attention than SoTA baselines in Grad-CAM visualizations.
  • Qualitative analysis: The model demonstrates superior precision in localizing and delineating target regions compared with SoTA methods.
  • Quantitative analysis: 92.24% Dice and 84.9% mIoU were achieved on QaTa-COV19, improving over MAdapter by +2.17% and U-Mamba by +11.73%.
  • Quantitative analysis: 79.67% Dice and 66.38% mIoU were achieved on MosMed++, improving over MAdapter by +1.27% and nnUNet by +7.35%.
  • Quantitative analysis: 93.83% Dice and 87.62% mIoU were achieved on Kvasir-SEG, improving over MAdapter by +2.46% and TransUNet by +3.3%.
  • Efficiency: 39.9M trainable parameters and 17.87G FLOPs make the model more compact and computationally efficient than many SoTAs.The model is reported to maintain the highest performance across all three datasets, indicating a strong performance-efficiency trade-off.

6. Ablation Studies

Ablation experiments on Kvasir-SEG show that SEU loss, textual guidance, and structured architectural components each contribute to segmentation performance.

  • Loss Function Analysis: Replacing SEU with Dice loss or BCE reduces Dice to 93.44% and 92.03%, respectively.These results support SEU’s role in capturing spatial and uncertainty-aware features.
  • Effect of Textual Guidance: Removing text prompts during inference lowers the Dice score to 87.28%.
  • Effect of Textual Guidance: Removing MoDAB and eliminating textual supervision lowers Dice to 85.15%.The larger drop reinforces the value of language-driven guidance.
  • Architectural Component Evaluation: Replacing cross-attention with point-wise addition reduces segmentation accuracy to 92.11%.This evaluates the contribution of structured cross-modal attention.
  • Architectural Component Evaluation: Replacing SSMix with a linear projection yields 91.72% segmentation accuracy.The result highlights the contribution of dynamic sequence modeling.

7. Conclusion

The study presents an uncertainty-aware vision-language segmentation model that combines visual and textual information through MoDAB, SSMix, and SEU Loss. Experiments across multiple medical datasets show improved accuracy and computational efficiency relative to existing SoTA methods, supporting more reliable and interpretable segmentation for clinical decision-making.

  • The model integrates visual and textual data through MoDAB and SSMix for cross-modal learning.
  • SEU Loss guides training to account for uncertainty while enhancing spatial precision and visual-linguistic alignment.
  • Experiments on multiple medical datasets show that the model outperforms existing SoTA methods in accuracy and computational efficiency.
  • The approach supports more reliable and interpretable segmentation for clinical decision-making.
Loading 2602.14498v2…