PaperlayerView exact revision on arXiv

Source-linked AI summary

Characterizing Text Branch Sensitivity in Medical Vision-Language Segmentation via Evidence Decoupling

Ziquan Liu, Zhewei Zhu, Xuyang Shi

arXiv:2609.02663v1cs.CV

TL;DR

The paper investigates how much clinical text contributes to pixel-level medical image segmentation, where prior work largely assumes text is beneficial. It introduces EDD to decompose image and text-modulated evidence, finding that text sensitivity varies strongly by dataset while fusion choice has limited effect. The analysis indicates that text is represented through global semantic modulation rather than an independent spatial text-to-pixel pathway.

  • Problem

    The study addresses the understudied question of how much the text branch contributes to pixel-level decisions in multimodal medical segmentation.

  • Method

    The paper evaluates fusion operators across pretrained medical VLMs and datasets, and introduces EDD using evidential deep learning and deep supervision to decompose image and text-modulated evidence.

  • Results

    Fusion operators yield nearly identical segmentation performance, whereas text perturbation causes severe degradation on BTMRI and BUSI but relatively small effects on ISIC and Kvasir-SEG.

  • Takeaways & Limitations

    Text sensitivity is dataset dependent, and text-modulated evidence is globally distributed with different semantic components driving sensitivity across datasets.

  • Takeaways & Limitations

    EDD’s text evidence is text-modulated visual evidence spatialized through visual features, not an independent text-to-pixel pathway.

Abstract

from arXiv · show

Pretrained vision-language models (VLMs) have shown promising performance in medical image segmentation by incorporating clinical text. However, it remains unclear how much textual information actually contributes to pixel-level predictions. In this work, we systematically investigate the role of text in multimodal medical image segmentation. We first analyze several commonly used fusion strategies and find that segmentation performance is largely insensitive to the choice of fusion module. To further understand modality interactions, we propose an Evidence Decoupling Decoder (EDD) based on evidential deep learning and deep supervision. EDD serves as an internal representation analysis tool that decomposes image evidence and text-modulated evidence throughout the decoding process while maintaining competitive segmentation performance. Experimental results show that the sensitivity to text perturbation varies substantially across datasets. On BUSI and BTMRI, removing text causes catastrophic performance drops, indicating strong model reliance on textual input. On ISIC and Kvasir-SEG, text exerts relatively marginal influence. We further find that text affects predictions mainly through global semantic modulation rather than independent spatial localization, and that the specific semantic components driving text sensitivity differ across datasets. These findings provide a deeper understanding of modality interaction in multimodal medical image segmentation and offer practical insights for future model design.

1 Introduction

The paper asks how much text contributes to pixel-level medical segmentation, challenging the assumption that increasingly complex fusion architectures necessarily improve multimodal information use. Across datasets and models, fusion choice has little effect, while text sensitivity varies substantially by dataset.

  • Existing work treats clinical text as beneficial and emphasizes increasingly complex fusion architectures.
  • The study evaluates four imaging datasets, five pretrained medical VLMs, and four cross-modal fusion operators.
  • EDD combines evidential deep learning with deep supervision to decompose image and text-modulated evidence at each decoder layer without sacrificing accuracy.
  • Four topologically distinct fusion operators produce nearly identical segmentation performance, suggesting limited impact from fusion micro-structure under current pretrained representations.
  • Text sensitivity is dataset dependent: corruption severely degrades BTMRI and BUSI, but has relatively small impact on ISIC and Kvasir-SEG.
  • Text-modulated evidence is globally distributed, and different text components drive sensitivity across datasets.

2 Related Work

Prior medical VLM segmentation methods improve architecture and alignment, but generally treat text as inherently beneficial without quantifying its pixel-level contribution. Related multimodal research instead highlights modality gaps, imbalance, collapse, and uncertainty-aware modeling.

  • Text-guided segmentation frameworks use early encoder fusion or text-conditioned decoder attention and alignment, while MedCLIPSeg adds probabilistic adaptation.
  • These methods advance architecture design but do not quantitatively investigate how much the text branch contributes to pixel-level decisions.
  • Contrastive VLM studies associate modality gaps with mismatched pairs, temperature, and information imbalance between image and text embeddings.
  • Other work attributes modality collapse to noisy cross-modal entanglement and unaligned inter-modal gradients that hinder balanced convergence.
  • EDL-based medical methods support uncertainty-aware segmentation and multimodal reliability modeling, but existing multimodal EDL frameworks address different multimodal settings.

3 Methodology

The baseline processes image and text through separate encoders, then combines their features in a U-shaped decoder. The methodology evaluates pretrained encoders and alternative fusion operators within this framework, while EDD separately models modality evidence during decoding.

  • 3.3 Evidence Decoupling Decoder: EDD separately models image and text evidence streams throughout decoding to support fine-grained analysis of modality-specific representations.
  • 3.1 Problem Definition and Overall Framework: The framework encodes an image and its corresponding text with separate image and text encoders before U-shaped decoder segmentation.
  • 3.1 Problem Definition and Overall Framework: The evaluated encoders comprise five representative pretrained medical VLMs used as image-text feature extractors.
  • 3.2 Baseline Segmentation Network with Multimodal Features: At each decoder level, deeper features are upsampled and fused with encoder skip features to form intermediate visual features.
  • 3.2 Baseline Segmentation Network with Multimodal Features: A fusion module injects the text embedding into visual features, after which a residual convolutional block produces the current decoder output.
  • 3.2 Baseline Segmentation Network with Multimodal Features: The study integrates four typical fusion operators into the standard decoder to compare cross-modal interaction mechanisms.

Design Motivation and Modeling Considerations.

The EDD design treats intermediate decoder features as projectable into evidence and applies consistent supervision across resolutions. It decomposes each layer into image, text-modulated, and fused evidence streams.

  • Design Motivation and Modeling Considerations.: EDL represents outputs as non-negative evidence under a Dirichlet distribution, enabling explicit modality-level evidence decomposition.
  • Design Motivation and Modeling Considerations.: EDD assumes intermediate decoder features are semantically rich enough for projection into non-negative evidence through lightweight convolution and Softplus activation.
  • Design Motivation and Modeling Considerations.: Per-level evidence maps share the final output’s semantic target and differ only in spatial resolution, allowing a common supervision signal.
  • Design Motivation and Modeling Considerations.: The evidence branch divides each layer’s evidence into pure image evidence, text-modulated evidence, and fused evidence.

Evidence Decoupling Branch Design.

The evidence decoupling branch separates direct image evidence from text-modulated evidence at each decoder level. Text supplies global semantic intensity coefficients, while visual features provide the spatial response bases inherited by the text evidence.

  • The branch runs after each skip connection but before fusion, using unmodulated visual features as the evidence-extraction base.This placement avoids contaminating raw visual representations with text modulation.
  • The image evidence head directly predicts non-negative segmentation evidence from visual features alone.It reflects the vision encoder’s independent ability to discriminate the segmentation target.
  • Text evidence combines global semantic coefficients from the text embedding with spatial response bases derived from current-level visual features.An MLP maps the text embedding into class-dimensional semantic intensity coefficients, while visual features provide the spatial bases.
  • Channel-wise multiplication spatializes text semantics by preserving visual spatial structure while regulating its intensity with text.Thus, text evidence inherits spatial patterns from vision rather than forming an independent text-only map.
  • The text evidence map is text-modulated visual evidence, so the decomposition characterizes learned modality interaction rather than intrinsic text-to-pixel attribution.A global text vector lacks native spatial resolution, making visual spatialization necessary but interpretively constraining.
  • Image and text-modulated evidence streams are added without learned parameters to form total fused evidence.This makes the decomposition structurally transparent, while still reflecting learned evidence representation rather than ground-truth modality attribution.

Baseline Network Loss.

The baseline segmentation network is trained with a combined Dice and cross-entropy objective, balanced by a loss-weight parameter.

  • The standard segmentation network uses the combination of Dice loss and cross-entropy loss.The balance between the two terms is controlled by λ.

Evidence Network Joint Loss.

The evidence network trains with EDL losses that combine prediction fidelity and KL regularization across the final head and intermediate decoder levels. At inference, expected probabilities and epistemic uncertainty are derived from Dirichlet evidence.

  • EDL reformulates segmentation as evidence regression under a Dirichlet distribution, with concentration parameters α = e + 1 and total evidence S.The single-layer loss includes fidelity and KL regularization terms.
  • The fidelity term minimizes expected squared error under the Dirichlet distribution for the true class.Its components measure deviation from the ground-truth label and prediction variance.
  • KL regularization pushes evidence for non-true classes toward zero, discouraging high confidence across all classes.The regularized Dirichlet parameters remove the true-class evidence before the KL divergence is applied.
  • The single-layer EDL objective is L_EDL(e, Y) = L_err(e, Y) + λ_t · L_KL(e, Y).The annealing coefficient λ_t increases during training to avoid excessive early regularization.
  • The overall loss is a weighted sum of EDL losses at the final segmentation head and all intermediate decoder levels.The model uses four decoder levels, with weights decreasing as layer depth increases.
  • At inference, expected class probabilities are α_k/S, while low total evidence S indicates high epistemic uncertainty.Total evidence therefore supports both prediction and pixel-level uncertainty quantification.

4 Experiments

Experiments show that encoder choice and dataset strongly shape segmentation behavior, whereas fusion-module changes have limited effect. EDD enables evidence decomposition while generally preserving segmentation performance, and perturbation tests reveal dataset- and semantic-component-specific text sensitivity.

  • VLM Encoder Comparison: MedCLIP reaches 91.61% Dice on BTMRI, while PubMedCLIP reaches 78.40% and BioMedCLIP 87.63%; no encoder dominates every dataset.UniMed-CLIP leads on BUSI and ISIC, whereas MedCLIP leads on BTMRI and Kvasir-SEG.
  • Fusion Module Sensitivity Analysis: Fusion modules produce only marginal variation, with a maximum fluctuation ∆max of 2.75% for BioMedCLIP.Differences between VLM encoders, including roughly 4% between CLIP and MedCLIP on BTMRI, exceed differences across fusion strategies.
  • Evidence Decoupling Decoder Validation: EDD maintains segmentation performance close to the standard decoder for nearly all VLM-dataset pairs while enabling evidence decomposition.Most cases improve, including a 2.08% Dice gain for CLIP on BUSI and a 6.32% gain for PubMedCLIP on BTMRI.
  • Evidence Decoupling Visualization Analysis: Text evidence is globally distributed rather than independently spatial, while additive evidence remains nearly identical to image evidence, indicating image-dominated fused outputs.On BTMRI, MedCLIP text evidence responds to fine-grained structures absent from image evidence, complementing missing local visual patterns.
  • Text Perturbation Experiments: Text perturbation sensitivity varies by dataset: BTMRI and BUSI show severe degradation, while ISIC and Kvasir-SEG show weaker or moderate dependence.On BTMRI, MedCLIP falls from 91.61% to 40.76% Dice without text and to 14.73% with contradictory text; on ISIC it retains 89.76% without text, while Kvasir-SEG falls to 78.86% for MedCLIP and 64.78% for CLIP.
  • Text Perturbation Experiments: The semantic components driving text sensitivity differ across datasets: BTMRI depends more on category-level and lesion-existence cues, whereas BUSI depends more on location information.Removing location descriptors changes MedCLIP Dice from 91.61% to 90.88% on BTMRI but from 82.43% to 71.58% on BUSI.

5 Conclusion

Across five VLMs and four datasets, segmentation was largely insensitive to fusion-module choice, while text sensitivity varied substantially by dataset. EDD decomposed image and text-modulated evidence, revealing strong reliance on text for BTMRI and BUSI but relatively marginal influence for ISIC and Kvasir-SEG.

  • Fusion-module choice had limited impact, suggesting that current multimodal models are dominated by visual representations.
  • EDD decomposed image and text-modulated evidence while maintaining competitive segmentation performance.
  • Removing text caused severe degradation on BTMRI and BUSI but had relatively marginal influence on ISIC and Kvasir-SEG.
  • On BTMRI, category-level semantics rather than spatial cues primarily drove text sensitivity.
  • Perturbation results indicate model sensitivity rather than intrinsic text contribution, while EDD characterizes learned representations rather than ground-truth attribution.
Loading 2609.02663v1…