MedPlex: Deep Vision-Language Co-Adaptation for Clinically Grounded Medical Segmentation

Rafi Ibn Sultan; Hui Zhu; Chengyin Li; Dongxiao Zhu

Source-linked AI summary

MedPlex: Deep Vision-Language Co-Adaptation for Clinically Grounded Medical Segmentation

Rafi Ibn Sultan, Hui Zhu, Chengyin Li, Dongxiao Zhu

arXiv:2608.13690v1Aug 13, 2026cs.CVcs.AI

TL;DR

Medical image segmentation often underuses textual clinical knowledge, while existing text-guided methods typically integrate language too late for meaningful representation learning. MedPlex jointly adapts visual and textual representations throughout encoding with bidirectional fusion and concept alignment, consistently improving segmentation across CT and MR benchmarks, including free-text supervision.

Problem
Existing text-guided medical segmentation methods often integrate language too late to make it meaningful for visual representation learning.
Method
MedPlex jointly updates visual and textual representations through encoder-level BiFusion and class- and region-level concept alignment using clinical text.
Results
MedPlex consistently improves segmentation across CT and MR benchmarks, including free-text supervision, raising BTCV DSC from 73.54% to 79.65% without adaptation.
Takeaways & Limitations
Continuous vision-language co-adaptation and structured concepts support segmentation across diverse anatomical targets, modalities, and textual supervision settings.
Takeaways & Limitations
MedPlex uses volume- or anatomical-class-level textual supervision without explicitly modeling 2D slice-wise cues.

Abstract

from arXiv · show

Medical image segmentation is still largely treated as a vision-only problem, although clinical interpretation often relies on textual knowledge of anatomy, location, appearance, and surrounding context. Existing text-guided segmentation methods within the Vision-Language Model (VLM) paradigm often use language only as a late conditioning signal, limiting its influence on visual representation learning. We introduce MedPlex (Medical Plexus of Vision and Language), an end-to-end VLM framework that makes text guidance a continuous, clinically grounded component of segmentation learning. Through Bi-Fusion (Bidirectional Fusion), visual and textual representations evolve jointly across the encoding hierarchy. MedPlex further introduces class-level and region-level concept alignment to organize the shared representation at complementary granularities. Class-level alignment anchors each anatomical target to an aggregated clinical concept profile, while region-level alignment preserves individual concepts, such as shape, location, appearance, and texture, through class-specific visual evidence. In this way, language provides structured supervision throughout the encoder rather than serving only as a late-stage cue. MedPlex achieves state-of-the-art performance across CT and MR benchmarks for multi-organ, cardiac substructure, and tumor segmentation, including settings with real free-text clinical supervision. Code: https://github.com/rafiibnsultan/MedPlex.

Introduction

MedPlex addresses limitations in text-guided medical segmentation by making language a continuous, clinically grounded part of representation learning. Its bidirectional fusion and multi-granular concept alignment jointly organize visual and textual information across the encoder hierarchy.

Motivation: Purely visual segmentation models achieve strong performance but omit the anatomical knowledge, reports, notes, and descriptions used in clinical interpretation.
Limitations: Existing text-guided methods can inject anatomical context and disambiguate similar regions, yet often make language only a weak part of representation learning.
Limitations: Representation staleness arises when models inject text after substantial visual abstraction or broadcast fixed text representations, preventing continuous cross-modal interaction.
MedPlex: MedPlex uses BiFusion to progressively update visual and textual encoder streams, letting visual features query language while image evidence grounds textual representations.
MedPlex: MedPlex combines BiFusion with class-level and region-level concept alignment and supports standard backbones, structured descriptions, and real free-text clinical supervision.

2 Related Work

Prior language-guided segmentation work uses text to specify targets and resolve visual ambiguity, but vision-language interaction remains less developed in 3D medical segmentation. Existing approaches vary in fusion location, affected visual representations, and whether text adapts to image-specific evidence, motivating more structured concept alignment.

Textual descriptions help specify segmentation targets, disambiguate visually similar regions, and support open-vocabulary or referring segmentation.
Fusion at Vision Decoder: In 3D medical image segmentation, vision-language interaction remains less developed, with methods differing in fusion location, visual representations influenced, and text adaptation to image-specific evidence.
Biomedical and clinical text encoders provide domain priors, while global contrastive learning aligns image and text representations, but single global embeddings inadequately capture overlapping anatomical concepts.Relevant concepts include shape, location, appearance, and texture.
Structured alignment has matched image patches with report tokens or phrase groups in 2D chest X-ray pretraining, whereas 3D methods often align post hoc or mainly at decoder level.These approaches leave encoder representations less directly organized around fine-grained clinical concepts.
MedPlex applies class-level concept alignment and region-level concept alignment to structure clinical concepts in 3D medical image segmentation.

3 Method

MedPlex jointly learns 3D visual and textual representations through encoder-level bidirectional fusion, supporting real clinical reports or structured class-specific descriptions. It then exposes class-structured visual evidence and applies concept-grounded alignment across complementary granularities.

Framework: MedPlex combines a 3D encoder-decoder with a transformer text encoder, matching visual and textual representations stage by stage.Textual supervision can be provided as real free-text clinical reports or structured class-specific descriptions.
Bidirectional Fusion: BiFusion updates both modalities at every encoder stage, grounding text in image evidence before updating visual features with image-conditioned textual context.This coupled process lets cross-modal co-adaptation accumulate through the encoding hierarchy.
Class-Structured Representation: A class-response projection converts encoder features into one class-specific channel per anatomical class before concept alignment.The projection is supervised by a detached, downsampled decoder prediction, while final segmentation remains the decoder’s responsibility.
Concept Alignment: MedPlex aligns visual and textual representations at class and region levels, respectively capturing anatomical-class semantics and concept-wise visual evidence.Class-wise visual embeddings are obtained by average-pooling each class channel of the projected encoder representation.
Class-Level Alignment: For each anatomical class, clinical concepts are aggregated into an image-conditioned textual representation and aligned with class-wise visual embeddings in a shared normalized space.Bidirectional InfoNCE aligns each class with its clinical concept profile while separating it from other anatomical classes; Gaussian noise is added to both modalities during contrastive learning.

4 Experiments

Experiments evaluate MedPlex across diverse CT and MR segmentation benchmarks, including real free-text clinical supervision, and test its quantitative performance, concept alignment, fusion design, representation structure, and backbone robustness. Results show gains over vision-only and vision-language baselines, with structured concepts and bidirectional fusion contributing substantially to performance.

Datasets: MedPlex is evaluated on abdominal, cardiac, brain-tumor, and real clinical-text segmentation datasets spanning 3D CT and MR.The benchmarks include AMOS22, BTCV, MM-WHS, MSD-Brain, and the focal-abnormalities subset of ReXGroundingCT.
Main Segmentation Performance: On AMOS22, MedPlex improves DSC by nearly 2%, reduces HD95 from 8.32 mm to 6.52 mm, and increases NSD from 92.03% to 93.97% over the best vision-only result.It also surpasses the strongest vision-only baseline across all reported metrics on MM-WHS (CT).
Real Clinical-Text Supervision: On ReXGroundingCT, MedPlex achieves the best DSC and NSD among compared methods despite sparse masks and free-text reports describing focal abnormalities.This setting lacks clean class-level descriptions and may contain incomplete or weakly localized visual targets.
Textual Information and Ablations: Aggregating all concepts lowers DSC by 1.2 points, using only class names causes a 1.8-point drop, and removing textual input decreases DSC by 3.5 points.Consistent concept descriptions perform best, while ambiguous concepts reduce accuracy, supporting explicit concept structure rather than text alone.
Fusion and Alignment Ablations: BiFusion reaches 88.21 DSC and 93.97 NSD, exceeding matched unidirectional fusion at 85.98 DSC and 92.27 NSD and late fusion at 85.56 DSC and 91.66 NSD.The matched unidirectional strategy updates vision with text but does not update text using visual evidence; region-level alignment is most effective with class-level anatomical anchors.
Backbone Robustness: Replacing ClinicalBERT with BioBERT or PubMedBERT causes only minor performance changes, while U-Net and UNETR vision backbones retain consistent gains.These results suggest hierarchical fusion and concept-grounded alignment adapt different biomedical text encoders into a shared representation space.

5 Conclusion

MedPlex makes textual guidance a continuous, clinically grounded component of segmentation representation learning through BiFusion and concept-grounded alignment. It consistently improves segmentation across CT and MR benchmarks, including free-text report supervision, while currently lacking explicit modeling of 2D slice-wise cues.

5 Conclusion: MedPlex turns textual guidance into continuous, clinically grounded representation learning through BiFusion and concept-grounded alignment.The framework applies this approach across vision-language segmentation learning.
5 Conclusion: MedPlex consistently improves segmentation across CT and MR benchmarks, including settings with free-text report supervision.The improvements span diverse anatomical targets and textual settings.
5 Conclusion: MedPlex uses volume- or anatomical-class-level textual supervision without explicitly modeling 2D slice-wise cues.This is identified as a limitation and motivates future work.

A Appendix: Additional Technical Details and Benchmarks · A.1 Additional Method Details

MedPlex uses a Swin UNETR-based encoder-decoder with stage-wise visual-textual fusion and skip connections for 3D anatomical segmentation. Its training jointly optimizes segmentation, class-response supervision, and concept-grounded alignment with learned uncertainty-based weighting.

A.1 Additional Method Details: MedPlex integrates with standard encoder-decoder 3D segmentation backbones and uses Swin UNETR as its vision model in the main experiments.
A.1 Additional Method Details: The vision encoder has four Swin Transformer stages with MLP layers, normalization, residual connections, and downsampling between stages.At stage n, spatial resolution is approximately (H/2^n,W/2^n,D/2^n).
A.1 Additional Method Details: Multi-stage visual representations are used for stage-wise fusion with textual representations throughout the encoder.
A.1 Additional Method Details: The decoder receives BiFusion-updated representations through corresponding skip connections, combining earlier fine-grained detail with deeper semantic representations.The final prediction layer produces an N-class segmentation map, where N denotes the number of anatomical classes.
A.1 Additional Method Details: The appendix evaluates MedPlex with three domain-specific BERT encoders, including BioBERT and ClinicalBE.The supplied passage truncates the third encoder name after “ClinicalBE.”
A.1 Additional Method Details: The Dice loss uses the predicted segmentation map P and ground-truth mask G.
A.1 Additional Method Details: The cross-entropy term supplies voxel-wise class supervision and complements the overlap-based Dice objective.
A.1 Additional Method Details: MedPlex jointly optimizes segmentation, class-response supervision, and concept-grounded alignment using an uncertainty-weighted objective.The objective is L = 1/(2σ1^2)Lseg + 1/(2σ2^2)LClassEmbed + 1/(2σ3^2)Lalignment.

A.2 Controlled Textual Information

For datasets lacking report-level supervision, MedPlex generates structured, class-specific clinical descriptions with an LLM, covering observable anatomical concepts. Authors manually reviewed and secondarily quality-checked these descriptions before fixing them for consistent training.

Description generation: An LLM generates class-specific descriptions for each anatomical class when report-level supervision is unavailable.The descriptions are conditioned on the organ name and imaging modality.
Description generation: Each description organizes observable anatomy into shape, location, appearance/density, contour/symmetry, and internal texture.The prompt requests a visual description for each concept category in the relevant modality image.
Quality control: Authors manually reviewed every generated description and applied a secondary LLM-based quality pass.Review checked category adherence and excluded functional claims or clinical interpretation.
Quality control: The reviewed descriptions were fixed and used consistently throughout training.Representative class-specific structured descriptions are shown in Table 8.

A.3 Datasets and Preprocessing · A.3.1 Dataset Description

MedPlex is evaluated on diverse 3D medical imaging datasets spanning abdominal organs, cardiac substructures, brain tumors, and thoracic abnormalities. The datasets include fixed-anatomy segmentation with defined label mappings and free-text radiology findings grounded to voxel-level masks.

A.3.1 Dataset Description: The evaluation covers abdominal organs, cardiac substructures, brain tumors, and thoracic abnormalities across diverse 3D medical imaging datasets.These datasets support multiple clinically distinct segmentation settings.
A.3.1 Dataset Description: AMOS22 contains abdominal CT annotations for 15 organs, including the spleen, kidneys, liver, pancreas, bladder, and prostate/uterus.The passage lists all 15 annotated organs, including paired and composite anatomical targets.
A.3.1 Dataset Description: AMOS22 provides 200 training-validation images and 100 test images.The dataset is divided into training-validation and test image sets.
A.3.1 Dataset Description: Brain-tumor annotations derive tumor core, whole tumor, and enhancing tumor from label combinations.Labels 2 and 3 form TC, labels 1, 2, and 3 form WT, and label 3 forms ET.
A.3.1 Dataset Description: The brain-tumor dataset uses 484 publicly available cases split into 349 training, 39 validation, and 96 test cases.The split totals the stated 484 cases.
A.3.1 Dataset Description: ReXGroundingCT is a 3D chest CT dataset that grounds free-text radiology findings to voxel-level segmentation masks.It is derived from CT-RATE and uses the Focal Abnormalities subset containing nodules, consolidations, and focal opacities.
A.3.1 Dataset Description: The ReXGroundingCT experimental split contains 1,680 training volumes and 264 test volumes.The split is drawn from the Focal Abnormalities subset.

A.3.2 Dataset Preprocessing … A.6 Benchmark Models

The paper standardizes 3D preprocessing, reports experiment hyperparameters and clinically relevant segmentation metrics, and benchmarks MedPlex against vision-only and text-guided VLM methods. These benchmarks span CNN, transformer, hybrid, and multimodal segmentation approaches.

A.3.2 Dataset Preprocessing: MedPlex preprocesses all 3D datasets using consistent RAS orientation, dataset-specific resampling, applicable intensity clipping, modality-specific normalization, and foreground-guided 96 × 96 × 96 patch sampling.Training augmentation includes random flipping along all three spatial axes and random rotations.
A.4 Hyperparameters: Table 10 summarizes the key hyperparameters used in the experiments.The supplied passage refers to Table 10 for the experiment settings but does not enumerate their values.
A.5 Evaluation Metrics: Performance is evaluated with Dice Similarity Coefficient (DSC, %), 95% Hausdorff Distance (HD95, mm), and Normalized Surface Distance (NSD, %).Higher DSC reflects better overlap, lower HD95 reflects better boundary alignment, and higher NSD reflects better surface agreement within tolerance τ.
A.6 Benchmark Models: Vision-only comparisons include U-Net, UNet++, nnU-Net, STU-Net, MedNeXt, UNETR, Swin UNETR, SwinUNETR-V2, nnFormer, and MedFormer.These models represent CNN-based, transformer-based, hybrid, self-configuring, scalable, and locality- or long-range-dependency-oriented segmentation designs.
A.6 Benchmark Models: The transformer-based vision-only baselines use Vision Transformer or hierarchical shifted-window encoders, stage-wise convolutional improvements, long-range dependency modeling, or locality-sensitive attention.This summarizes UNETR, Swin UNETR, SwinUNETR-V2, nnFormer, and MedFormer.
A.6 Benchmark Models: Text-guided VLM comparisons include Universal-CLIP, MulModSeg, ZePT, and CAT under comparable prompt or text-input settings.Universal-CLIP uses Swin UNETR and the original text format; MulModSeg uses modality-conditioned embeddings with alternating CT/MR training.
A.6 Benchmark Models: ZePT receives one aggregated structured concept description per class to match its prompt format.ZePT is a zero-shot pan-tumor framework based on query disentanglement and self-prompting.
A.6 Benchmark Models: CAT coordinates anatomical and textual prompts through a query-based dual-prompt design, using the same text-input setting as ZePT.The shared setting reflects their related prompt-based formulations.