PaperlayerView exact revision on arXiv

Source-linked AI summary

CoMLP: Cooperatively-Gated MLPs for Fine-Grained Cross-Modal Information Fusion in Medical Image Segmentation

Mingyuan Meng, Shuchang Ye, Mingjian Li, Zhenyu Zhao, Jinman Kim, Lei Bi

arXiv:2609.04781v1cs.CV

TL;DR

Medical segmentation needs to combine heterogeneous visual and textual information while retaining fine spatial detail, but cross-attention becomes costly on high-resolution features. CoMLP uses cooperative cross-gating with regional and dilated MLP interactions in a unified architecture for inter-image and vision-language fusion, achieving consistent improvements across five benchmarks and enabling effective high-resolution interaction with favorable memory scalability.

  • Problem

    Medical segmentation requires fine-grained fusion of complementary imaging and clinical information, while dense cross-attention becomes difficult as spatial resolution and token counts increase.

  • Method

    CoMLP combines cooperative cross-gating with regional and dilated MLP interactions, using one interaction primitive for inter-image and vision-language fusion.

  • Results

    Experiments on five medical segmentation benchmarks show consistent improvements over existing multi-modal and language-guided methods, with effective high-resolution interaction and favorable memory scalability.

  • Takeaways & Limitations

    MLP-based dense interaction is a promising alternative for fine-grained cross-modal information fusion in medical vision.

  • Takeaways & Limitations

    The reported efficiency advantage is tied to interaction granularity and memory scalability rather than simply lower FLOPs or fewer parameters.

Abstract

from arXiv · show

Multi-modal medical images and clinical reports provide complementary anatomical, functional, and semantic information for medical image segmentation. Effectively exploiting these heterogeneous sources requires fine-grained cross-modal information fusion that preserves subtle spatial details while capturing semantic dependencies across modalities. Existing fusion approaches frequently rely on cross-attention, whose computational burden increases rapidly with spatial resolution, making dense cross-modal interaction difficult on high-resolution feature maps, particularly for volumetric medical images. In this work, we propose CoMLP, a cooperatively-gated MLP module for fine-grained cross-modal information fusion in medical image segmentation. CoMLP models cross-modal dependencies through cooperative cross-gating, built upon complementary regional and dilated MLP interactions, to capture local and global cross-modal dependencies. We further develop a multi-source fusion architecture in which CoMLP performs both inter-image fusion across imaging modalities and vision-language fusion between visual features and textual reports, enabling heterogeneous information to be integrated without relying on dense cross-attention. Extensive experiments on five medical segmentation benchmarks, covering 2D/3D images, clinical reports, multiple imaging modalities, and diverse anatomical regions, demonstrate consistent improvements over state-of-the-art multi-modal and language-guided segmentation methods. Ablation studies further show that fine-grained interaction at high spatial resolutions and complementary local-global fusion are critical to the performance gains. These results demonstrate the potential of MLP-based interaction as an effective alternative for fine-grained cross-modal information fusion in medical image segmentation.

I. INTRODUCTION

Medical segmentation must fuse complementary imaging and clinical information while preserving fine spatial detail. CoMLP addresses this need with cooperative, local-global MLP interactions used across inter-image and vision-language fusion.

  • Motivation: Medical images and clinical reports provide complementary anatomical, functional, and semantic information for segmentation.Effective fusion must integrate heterogeneous representation spaces while preserving subtle lesion boundaries and anatomical structures.
  • Motivation: Accurate segmentation requires fine-scale local tissue and boundary information together with broader context for distinguishing visually similar regions.Inter-image fusion should retain modality-specific characteristics, while vision-language fusion should associate textual semantics with spatial visual representations.
  • Research gap: Cross-attention models cross-modal dependencies explicitly but becomes computationally and memory intensive as high-resolution token counts grow.The burden is especially pronounced when two high-resolution spatial feature maps introduce large sets of visual tokens.
  • Research gap: MLP-based models offer a potential alternative for long-range and fine-grained modeling, but their use as dense cross-modal interaction operators remains largely open.Prior vision-language uses primarily emphasize projection, compression, or alignment rather than explicit fine-grained cross-modal dependencies.
  • Proposed approach: CoMLP combines cooperative cross-gating with regional and dilated MLP interactions to capture local and global cross-modal dependencies without dense cross-attention.Cross-modal information participates in gating, while complementary regional and dilated interactions support fine spatial granularity.
  • Contribution: A unified CoMLP architecture performs both inter-image fusion and vision-language fusion across five diverse medical segmentation benchmarks.The benchmarks include 2D/3D images, clinical reports, multiple imaging modalities, and diverse anatomical regions, with consistent effectiveness over state-of-the-art methods.

B. Vision-Language Fusion for Medical Segmentation

Medical vision-language segmentation uses clinical language as complementary semantic guidance for visual representations. CoMLP extends MLP-based interaction to heterogeneous image-text fusion while addressing the spatial granularity required by segmentation.

  • Vision-language motivation: Clinical language supplies high-level semantic information that complements visual context in medical images.Language-guided segmentation incorporates textual descriptions into segmentation and has established commonly used vision-language benchmarks.
  • Existing fusion approaches: Cross-attention remains prevalent for associating vision-language representations, while alternatives include dynamic convolutional guidance and multi-modal visual inputs.These approaches reflect ongoing efforts to fuse textual information with visual representations for segmentation.
  • MLP-based fusion: MLP-based models can model long-range visual dependencies without self-attention and have been explored for vision-language feature projection and token mixing.Medical vision-language systems use MLP-based projectors to connect multi-level visual representations with language embeddings.
  • Unified architecture: The same CoMLP interaction primitive supports inter-image fusion between imaging modalities and vision-language fusion between visual features and clinical reports.This provides a common formulation for two heterogeneous cross-modal interaction processes.
  • CoMLP: CoMLP uses cooperative cross-gating, regional and dilated MLP interactions, and modality adaptation to operate across image and image-text representations.The mainstream representation is enhanced using complementary information from another modality.

A. CoMLP: Fine-Grained Cross-Modal Interaction

CoMLP replaces dense cross-modal attention with cooperative cross-gating, where cross-modal features help determine spatial modulation. Complementary regional and dilated MLP interactions support fine-grained local and global dependency modeling.

  • A. CoMLP: Fine-Grained Cross-Modal Interaction: CoMLP performs MLP-based interactions along complementary local and global spatial axes for dense high-resolution cross-modal modeling.This avoids pairwise attention between cross-modal features while preserving fine-grained interaction.
  • A. CoMLP: Fine-Grained Cross-Modal Interaction: The cooperative cross-gating unit projects and splits the mainstream representation before incorporating complementary cross-modal information into gate construction.The projected representation is divided into two equal-dimensional components, u and v.
  • A. CoMLP: Fine-Grained Cross-Modal Interaction: Unlike conventional cross-gating, CoMLP determines the spatial gate jointly from mainstream and cross-modal information.The resulting gate modulates the mainstream response through element-wise multiplication.
  • A. CoMLP: Fine-Grained Cross-Modal Interaction: Cross-modal features directly determine which spatial responses are emphasized or suppressed rather than serving only as appended features or global conditioning.This design makes cross-modal information part of spatial gate generation.
  • A. CoMLP: Fine-Grained Cross-Modal Interaction: CoMLP factorizes cross-modal interaction into regional and dilated branches to distinguish fine local dependencies from long-range contextual interactions.The factorization addresses the computational and memory demands of unrestricted global interaction on high-resolution features.

2) Local-Global Multi-Axis Interaction:

CoMLP uses regional and dilated interaction branches to model local and global dependencies along structured spatial axes. The same cooperative interaction principle is adapted for aligned image features and heterogeneous image-text representations.

  • 2) Local-Global Multi-Axis Interaction:: For d-dimensional features, spatial elements are partitioned into regions of size P = R^d and reorganized for structured interaction.The representation is arranged as F^r ∈ R^G×P×C, where P is the number of elements per local region.
  • 2) Local-Global Multi-Axis Interaction:: The regional branch performs cooperative gating along each intra-region axis to capture local dependencies among neighboring modality-specific features.These interactions target tissue appearance, lesion texture, and boundary information.
  • 2) Local-Global Multi-Axis Interaction:: The dilated branch gates corresponding positions across regions along the G-dimensional axis to model long-range global dependencies.It provides contextual awareness for spatially distant but semantically correlated cross-modal cues.
  • 2) Local-Global Multi-Axis Interaction:: Regional and dilated outputs are restored, concatenated, and projected into a cross-modally enhanced representation.The two branches therefore provide complementary local and global enhancement.
  • 2) Local-Global Multi-Axis Interaction:: CoMLP avoids dense pairwise attention matrices and further reduces interaction burden through structured regional and dilated factorization.This design targets computationally favorable dependency modeling for high-resolution features.
  • 2) Local-Global Multi-Axis Interaction:: In inter-image fusion, complementary modality features share spatial coordinates and are processed through regional and dilated interaction branches.The complementary modality acts as Fcross while the other representation is enhanced.
  • 2) Local-Global Multi-Axis Interaction:: The same interaction principle jointly determines spatial modulation for mainstream and complementary representations, without requiring a shared semantic space.Only lightweight modality-specific adaptation is used before the common MLP interaction operator.

B. CoMLP-Based Multi-Source Fusion Architecture

The multi-source architecture combines inter-image fusion in an encoder with vision-language fusion in a decoder. CoMLP begins image interaction at the highest resolution and continues fusion across multiple scales.

  • B. CoMLP-Based Multi-Source Fusion Architecture: The architecture contains an inter-image fusion encoder and a vision-language fusion decoder.The encoder integrates complementary imaging modalities, while the decoder infuses report-derived semantics into visual representations.
  • B. CoMLP-Based Multi-Source Fusion Architecture: Two modality-specific encoder branches preserve individual imaging characteristics while progressively exchanging complementary information.Five successive HPVL blocks and intervening max-pooling operations construct a multi-level multimodal feature hierarchy.
  • B. CoMLP-Based Multi-Source Fusion Architecture: Each HPVL block runs intra-modal learning and inter-image interaction in parallel, combining residual convolutional processing with gMLP spatial mixing and CoMLP.This design preserves modality-specific local characteristics while learning inter-modal interactions.
  • B. CoMLP-Based Multi-Source Fusion Architecture: CoMLP interaction starts at the highest-resolution feature level F1 and continues through the encoder hierarchy.This enables complementary information to interact before progressive downsampling and supports multiscale inter-image fusion.
  • B. CoMLP-Based Multi-Source Fusion Architecture: The decoder uses five TSG blocks to combine upsampled decoder features with corresponding multimodal encoder skip features.Bilinear or trilinear interpolation is used for 2D or 3D feature upsampling, respectively.

2) Vision-Language Fusion Decoder:

The decoder progressively injects report-derived semantics into visual features across multiple spatial scales. The network then produces a segmentation mask and is trained with Dice and binary cross-entropy losses.

  • 2) Vision-Language Fusion Decoder:: Report-derived semantics interact with visual features progressively through successive TSG blocks at multiple spatial scales.Text is not introduced only at the bottleneck or prediction head.
  • 2) Vision-Language Fusion Decoder:: After the final TSG block, a convolutional segmentation head followed by Sigmoid activation generates the predicted mask.
  • 2) Vision-Language Fusion Decoder:: The unified architecture uses the encoder for fine-grained imaging-modality dependencies and the decoder for clinical-semantic associations with multiscale visual representations.This realizes inter-image and vision-language fusion within one segmentation architecture.
  • C. Learning Objective: The network is trained end-to-end using a combination of Dice loss and binary cross-entropy loss.These are used as the overall segmentation training objectives.
  • C. Learning Objective: The overall training objective includes a weighting factor λ for the cross-entropy term, empirically set to 1.

IV. EXPERIMENTAL SETUP

The experiments span five datasets covering 2D/3D images, multiple imaging modalities, clinical reports, and diverse anatomical regions, using standardized preprocessing and training procedures.

  • Datasets: Five datasets evaluate 2D/3D images, multiple imaging modalities, clinical reports, and diverse anatomical regions.OPC and NPC support joint PET/CT and report fusion, while QaTa-COV19, MosMedData+, and Kvasir-SEG evaluate vision-language fusion with single-modality images.
  • Datasets: OPC contains paired PET/CT images, manual tumor and lymph-node annotations, and expert-written clinical reports.The reports describe lesion location, size, and radioactive tracer uptake.
  • Datasets: NPC provides paired PET/CT images, clinical reports, and segmentation labels, with 632/234 cases assigned to training/testing.The training set was further split into 600/32 cases for training/validation.
  • Preprocessing: For OPC and NPC, PET/CT volumes were resampled to 1×1×1 mm3 and cropped to 160×160×160 voxels.PET used SUV maps with Z-score normalization, while CT was clipped to [−1024, 1024] HU and normalized to [−1, 1].
  • Preprocessing: The three 2D datasets used resized 224×224 images normalized to [0, 1].Dataset-specific splits were followed for QaTa-COV19, MosMedData+, and Kvasir-SEG.
  • Implementation: Experiments used PyTorch, an NVIDIA A100 GPU, Adam optimization, modality-specific batch sizes, 20,000 iterations, and scheduled learning-rate reductions.The batch size was 32 for 2D images and 2 for 3D images.

C. Comparison Methods

CoMLP is compared with segmentation methods grouped by their supported information sources and fusion mechanisms across multi-modal and uni-modal benchmarks.

  • Comparison categories: Comparison methods are grouped into general segmentation, language-guided segmentation, and other fusion categories according to supported information sources and mechanisms.The comparisons distinguish channel concatenation, explicit interaction, and language-based fusion strategies.
  • Multi-modal comparisons: On OPC and NPC, general and language-guided methods receive concatenated PET/CT, whereas multi-modal methods and CoMLP explicitly model inter-image interactions.Methods developed for 2D images were extended to 3D operations for these comparisons.
  • Uni-modal comparisons: On QaTa-COV19, MosMedData+, and Kvasir-SEG, comparisons use the uni-modal setting with general, VLP-based, and language-guided methods.Published aggregate results were used when available for the three public 2D benchmarks.
  • Experimental design: The evaluation compares CoMLP with existing methods across all five datasets and separately investigates inter-image and vision-language fusion.Controlled ablations examine high-resolution interaction, cross-gating variants, cross-attention, and heterogeneous representation preservation.
  • Evaluation: Dice and mIoU measure spatial overlap, with higher values indicating better segmentation performance.Paired Wilcoxon signed-rank tests with Holm-Bonferroni correction were used for OPC and NPC.

V. RESULTS

Across the evaluated settings, explicit inter-image fusion and direct language incorporation improve performance, while CoMLP achieves the best results on both multi-modal and vision-language benchmarks.

  • Inter-image fusion: Explicit inter-image fusion consistently improves over channel-concatenation strategies on OPC and NPC.CoMLP without reports also outperforms existing multi-modal image segmentation methods.
  • Multi-source fusion: Textual semantics provide complementary value, but methods without explicit inter-image interaction remain constrained on the PET/CT benchmarks.CoMLP jointly performs explicit inter-image and vision-language fusion and achieves the best performance on both datasets.
  • Vision-language fusion: CoMLP consistently achieves the best performance across QaTa-COV19, MosMedData+, and Kvasir-SEG.The comparison evaluates vision-language fusion independently from inter-image fusion.

B. Qualitative Comparison

Qualitative and ablation analyses link CoMLP’s gains to fine-grained high-resolution interaction, cooperative cross-gating, and complementary regional-dilated processing.

  • Qualitative comparison: CoMLP more reliably recovers small or localized lesions and reduces false-positive responses in visually confusing regions.These qualitative observations complement the reported quantitative improvements.
  • Inter-image fusion: Inter-image performance consistently decreases as the highest interaction resolution is reduced, with removing F1 causing the largest adjacent degradation.The results support interaction at fine spatial granularity before subtle information is damaged by downsampling.
  • Interaction operator: Under matched interaction granularity, cooperative cross-gating outperforms conventional cross-gating and cross-attention variants.Full-resolution interaction also improves conventional cross-gating, showing that granularity and gating design both matter.
  • Vision-language fusion: In vision-language fusion, CoMLP outperforms Global-CA and conventional cross-gating despite attention being computationally feasible for short textual sequences.Its advantage is attributed to content-adaptive gate generation and preservation of high-dimensional textual representations rather than avoiding attention cost.

E. Unified Interaction across Heterogeneous Sources

CoMLP serves as a shared interaction primitive for inter-image and vision-language fusion, with complementary regional and dilated interactions supporting different cross-modal dependencies. Its memory efficiency enables higher-resolution fusion while improving segmentation performance.

  • Unified fusion: Replacing MA-CA with CoMLP independently improves either inter-image or vision-language fusion, with the largest single-stage gain from inter-image replacement.Using MA-CA for both gives 79.55/65.72% average Dice/mIoU; replacing it for inter-image fusion gives 81.34/67.55%, while vision-language replacement gives 80.27/66.14%.
  • Complementary interactions: Removing either regional or dilated interaction degrades performance, while the regional branch matters more for inter-image fusion and the dilated branch more for vision-language fusion.These branches provide complementary local and broader cross-modal information across the two fusion settings.
  • Efficiency: At matched F2-F5 resolution, CoMLP reduces training/inference memory from 38/7.5 GB to 24/5.3 GB and computation from 0.98 to 0.90 TFLOPs versus MA-CA.Average Dice/mIoU also increases from 78.01/64.06% to 78.58/64.78%.
  • Efficiency: Full-resolution F1 interaction remains within the hardware budget at 28/5.9 GB training/inference memory and raises average Dice/mIoU to 80.51/66.97%.This configuration uses 1.14 TFLOPs and 30.6 M parameters, emphasizing that CoMLP's main advantage is finer accessible interaction rather than minimal computation.

H. Hyperparameter Analysis

CoMLP's effectiveness depends on interaction granularity and complementary local-global modeling. Performance is generally stable across region sizes, while high-resolution interaction and modality-specific branch preferences remain important.

  • Region size: Validation performance varies smoothly as region size R changes from 4 to 16, indicating general robustness to this hyperparameter.The best results use R = 8 on the 3D OPC and NPC datasets and R = 12 on the three 2D datasets.
  • Region size: An intermediate region size balances regional context against sparsity in the dilated branch.Small R limits local coverage, whereas large R increases the spatial interval between dilated interactions.
  • Fusion granularity: Removing interaction from progressively higher-resolution levels consistently degrades segmentation, with the largest decrease after removing full-resolution interaction.The efficiency analysis links this sensitivity to memory scalability and the preservation of subtle visual information before downsampling.
  • Branch complementarity: CoMLP's regional and dilated branches adapt to different modalities: inter-image fusion is more sensitive to regional removal, whereas vision-language fusion is more affected by dilated removal.This reflects stronger local correspondence needs for aligned imaging modalities and broader contextual needs for report-derived semantics.
  • Limitations: The simultaneous inter-image and vision-language evaluation is mainly limited to PET/CT with clinical reports, leaving other modality combinations and noisier reporting styles for future study.The region size is also manually specified despite relatively stable performance over a reasonable range.
  • Unified architecture: The unified architecture applies the same CoMLP interaction primitive to both inter-image and vision-language fusion within a multi-source design.Experiments across five benchmarks report consistent improvements and favorable memory scalability for high-resolution interaction.
Loading 2609.04781v1…