PaperlayerView exact revision on arXiv

Source-linked AI summary

Parameter-Efficient Multimodal Brain Tumor Segmentation With Stage-Wise Shared Adapters

Milad Rostamian, Mohammadreza Mohammadi, Ender Konukoglu, Mohsen Soryani

arXiv:2610.18571v1cs.CV

TL;DR

Multimodal fusion can increase model complexity when features are exchanged across modality-specific encoders. BWSA uses a shared adapter path between encoder stages, and its gated variant learns stream weights; both improve average region-wise Dice over unmodified backbones on BraTS 2023 online validation.

  • Problem

    Intermediate attention or gating can model cross-modal dependencies, but modality-specific branches and dense attention can increase parameter and computational costs.

    Source: §I · p. 1

  • Method

    BWSA combines compressed features from modality encoders in one shared path between stages, and TG-BWSA learns separate nonnegative weights for original and adapted features at each modality and stage.

    Source: §I · pp. 1–2

  • Results

    On BraTS 2023 online validation, BWSA raises average region-wise Dice by 1.3–7.6 points and TG-BWSA by 2.8–7.7 points versus unmodified backbones; both reduce average HD95.

    Source: §I · p. 1, §VI · p. 10

  • Takeaways & Limitations

    Fixed blending and learned weights have metric- and backbone-dependent trade-offs, and learned weights do not improve every metric.

    Source: §V · p. 10

    The evaluation uses BraTS 2020 and BraTS 2023 without an external cohort, and online-score variation across seeds was not measured.

    Source: LIMITATIONS AND FUTURE WORK · p. 10

Abstract

from arXiv · show

Different MRI modalities reveal different parts of a brain tumor. Segmentation models combine information from these modalities to outline tumor regions. Many models use a separate encoder for each modality. However, exchanging features among these encoders at several stages can increase model complexity. We introduce the Block-Wise Shared Adapter (BWSA), which combines compressed features from the modality encoders in a shared path and returns adapted features to each stream. In its fixed-blending form, BWSA adds the adapted features to the original features. Its trainable-gated variant (TG-BWSA) instead learns separate weights for the original and adapted features at each modality and encoder stage. We evaluate both variants on BraTS 2020 and BraTS 2023 with four segmentation backbones. On the official BraTS 2023 online validation set, BWSA increases average region-wise Dice by 1.3--7.6 percentage points relative to the corresponding unmodified backbones. TG-BWSA increases it by 2.8--7.7 points. Both variants add approximately 0.85--3.73% parameters and reduce average HD95 relative to the unmodified backbones.

I. INTRODUCTION

Fusion strategies trade off interaction timing, preservation of modality-specific streams, and model cost. BWSA shares an adapter path between encoder stages, while TG-BWSA learns fixed per-modality, per-stage blending weights; both are evaluated across backbones and BraTS datasets.

I. INTRODUCTION

  • Early fusion permits interaction throughout the network but loses separate modality streams, whereas late fusion preserves them but delays interaction until prediction.Intermediate attention or gating can add parameter and computational costs.Source: §I · p. 1
  • BWSA uses fixed unit weights to blend adapted and original features, while TG-BWSA learns two nonnegative weights per modality and encoder stage.Both use one shared path rather than a full adapter for every modality.Source: §I · p. 1
  • On BraTS 2023 online validation, BWSA raises average Dice by 1.3–7.6 points and TG-BWSA by 2.8–7.7 points versus unmodified backbones.The gated adapter adds 0.85–3.73% parameters relative to each unmodified backbone.Source: §I · pp. 1–2
  • The study evaluates fixed and learned shared-adapter weights and compares the shared adapter with separate modality-specific adapters.It tests whether a shared adapter can exchange features among four MRI encoders with a small parameter increase.Source: §I · pp. 1–2
  • Missing-modality experiments cover all 14 incomplete input configurations and report the complete-input case separately.Source: §I · p. 2

II. RELATED WORK

Related work spans early input fusion, interactions among separate modality streams, methods for modality weighting and missing inputs, and adapters used for multimodal fusion. BWSA connects four separate modality streams through a shared adapter path at each encoder stage.

A. MULTIMODAL FUSION METHODS

  • Swin UNETR combines sequences as input channels, while other approaches retain separate encoders and merge modality features through high-level fusion or hybrid attention.NestedFormer, mmFormer, and F2Net use attention, inter-modal Transformers, or shared and modality-specific representations to exchange information.Source: A. MULTIMODAL FUSION METHODS · p. 2

B. MODALITY WEIGHTING AND MISSING INPUTS

  • RFNet and A2FSeg derive modality weighting from input features, while missing-input methods include available-feature averaging, disentanglement with gated fusion, and fusion calibration.Qiu et al. report that adding a modality can harm predictions in some regions.Source: B. MODALITY WEIGHTING AND MISSING INPUTS · p. 2

C. ADAPTERS FOR MULTIMODAL FUSION

  • Prior adapters support medical-image adaptation or multimodal fusion between two pretrained SAM encoders or through dual-stream feature alignment.BWSA instead connects four separate modality streams through one shared adapter path at each encoder stage.Source: C. ADAPTERS FOR MULTIMODAL FUSION · p. 2

III. BLOCK-WISE SHARED ADAPTER FRAMEWORK

  • BWSA processes compressed features from all modalities through a shared path between encoder blocks, then adds adapted features to each modality’s original features.TG-BWSA learns separate nonnegative scalar weights for the original and adapted features at each modality and stage.Source: §III · p. 2

A. ARCHITECTURAL FRAMEWORK

Each MRI modality has its own encoder, while a shared decoder receives modality features through skip connections. Between encoder blocks, the adapter processes compressed features from all modalities through shared recalibration and refinement steps.

A. ARCHITECTURAL FRAMEWORK

  • Each modality is processed by its own encoder, and updated modality features form skip inputs to a shared decoder.The adapter changes connections between encoder blocks without changing those blocks’ internal operations.Source: A. ARCHITECTURAL FRAMEWORK · p. 3, Figure 2 · p. 3

B. BWSA MODULE AND GATED EXTENSION

W1 GAP(X ′(l)

The adapter refines compressed multimodal features through a shared recalibration and spatial-refinement path, then returns modality-specific adapted features. BWSA blends these with original features using unit weights, while TG-BWSA learns separate nonnegative gates for each modality and stage.

Channel Recalibration

  • The channel-recalibration operation uses channel weights derived with global average pooling, fully connected layers, and a sigmoid.The resulting weights are applied by channel-wise multiplication.Source: W1 GAP(X ′(l) · p. 3

3) Spatial Refinement and Expansion

  • A shared recalibration and refinement path processes the multimodal features, while the final expansion back to the original channel dimension is modality-specific.Depthwise separable convolution performs spatial refinement before expansion.Source: 3) Spatial Refinement and Expansion · p. 3

4) Residual Blending and Trainable Gates

  • BWSA combines each modality's original and adapted features with unit weights, whereas TG-BWSA learns two scalar gates per modality and stage.The gates are constrained to be nonnegative through Softplus.Source: 4) Residual Blending and Trainable Gates · p. 3
  • TG-BWSA adds 2ML trainable scalar weights beyond BWSA, or 32 weights for four modalities and four adapter stages.Both variants retain the same compression, recalibration, refinement, and expansion layers.Source: 4) Residual Blending and Trainable Gates · p. 3

TG-BWSA Overview

  • Figure 3 depicts TG-BWSA as compression followed by shared recalibration and refinement, then gated redistribution.Its second panel shows CMCR operating on the concatenated multimodal latent tensor.Source: Figure 3 · p. 4

A. DATASETS AND EVALUATION METRICS

The experiments use BraTS 2020 and BraTS 2023 with subject-level training and validation splits, and evaluate local and official online validation. Four backbones span 2D and 3D architectures, with a separate pretrained-encoder protocol for MEFCT.

A. DATASETS AND EVALUATION METRICS

  • BraTS 2020 has 369 subjects and BraTS 2023 has 1,251; both provide four co-registered MRI modalities and voxel-wise tumor annotations.Subjects were split once into 80% training and 20% local validation; BraTS 2023 also used an official online validation set of 219 subjects.Source: A. DATASETS AND EVALUATION METRICS · p. 4
  • Local-validation Dice is averaged over three runs, while 2D slice-based Dice is not directly comparable with 3D or official online Dice.BraTS 2023 online validation reports region-wise Dice and HD95 for ET, TC, and WT.Source: A. DATASETS AND EVALUATION METRICS · p. 4

B. IMPLEMENTATION DETAILS

  • BWSA variants were tested with two 3D backbones, RFNet and mmFormer, and two 2D backbones, MEUNet and MEFCT.For adapter comparisons, each backbone's decoder and loss function were retained.Source: B. IMPLEMENTATION DETAILS · p. 4
  • MEFCT encoders were pretrained individually on unlabeled BraTS 2023 training images, with held-out local-validation subjects excluded from pretraining and supervised optimization.Pretraining combined masked-patch reconstruction and contrastive instance discrimination; subsequent fine-tuning froze earlier encoder layers while unfreezing bottleneck blocks in Phase 2.Source: B. IMPLEMENTATION DETAILS · p. 5
  • The MEFCT LoRA and TG-BWSA comparison controls training protocol but not added parameter count or placement.LoRA adds approximately 1.3 × 10^5 trainable parameters and TG-BWSA approximately 9.0 × 10^5.Source: B. IMPLEMENTATION DETAILS · p. 5

C. RESULTS

On BraTS 2023 online validation, adapter variants improve region-wise Dice over unmodified backbones, while local, lesion-wise, and qualitative results provide complementary comparisons. The reported gains vary by backbone and evaluation setting.

Local Validation Results

  • MEUNet+TG-BWSA exceeds F2Net by 0.0230 Dice on BraTS 2020 and 0.0192 on BraTS 2023; on BraTS 2023, MEFCT+TG-BWSA exceeds LoRA by 0.0036 Dice.The MEFCT comparison uses the same training protocol, but parameter count and module placement differ.Source: 1) Local Validation Results · p. 5, Table 7 · p. 6

Region-wise Online Validation

  • TG-BWSA raises average Dice over the unmodified backbone by 0.0769 for mmFormer, 0.0283 for MEUNet, and 0.0450 for MEFCT, while reducing average HD95 for all three.The added effect of learned weights varies by region and backbone.Source: 2) Region-wise Online Validation · p. 6, Table 8 · p. 7

Lesion-wise Online Validation

  • TG-BWSA has lesion-wise Dice 0.852 and HD95 20.61 mm, versus 0.848 and 23.33 mm for Zheng et al.; the three highest-scoring listed teams have Dice of 0.876–0.888.Those teams used different training and inference pipelines, so their scores do not isolate the adapter’s effect.Source: 3) Lesion-wise Online Validation · p. 6, Table 9 · p. 7

Qualitative Analysis

  • In sampled local-validation cases, adapter variants recover some undersegmented tumor extent; 3D examples also show fewer isolated false positives and corrected over- or undersegmentation.Several displayed TG-BWSA boundaries more closely follow ground truth in 2D examples.Source: 4) Qualitative Analysis · pp. 6–7

Region-wise Online Validation

  • RFNet’s average Dice rises from 0.8314 without an adapter to 0.8639 with BWSA and 0.8654 with TG-BWSA; average HD95 is 28.80, 14.37, and 16.34 mm, respectively.Fixed-weight BWSA gives RFNet lower average HD95 than TG-BWSA.Source: 2) Region-wise Online Validation · p. 5, Table 8 · p. 7

D. PARAMETER OVERHEAD AND INFERENCE COST

The reported parameter overhead compares each TG-BWSA model with its unmodified backbone, while inference costs are measured for RFNet under one specified input and hardware setup.

Parameter Overhead

  • TG-BWSA’s reported parameter overhead is relative to the unmodified backbone, not the cost of gating alone; BWSA has nearly the same parameter count as TG-BWSA.BWSA uses the same adapter layers and lacks only the 2ML scalar gates.Source: 1) Parameter Overhead · p. 7, Table 10 · p. 7

Inference Cost

  • For RFNet, TG-BWSA increases latency by 2.4% (37 ms), peak memory by 8.2%, and GFLOPs by 4.1%.These measurements cover one forward pass with an 803 input on an NVIDIA A6000 GPU.Source: 2) Inference Cost · p. 7, Table 11 · p. 7
  • The inference-cost measurements apply only to the reported input size, RFNet implementation, and NVIDIA A6000 hardware configuration.They are not presented as measurements across backbones or hardware setups.Source: 2) Inference Cost · p. 7

E. ABLATION STUDIES AND TRAINING ANALYSIS

Ablations compare adapter components and gate settings, test incomplete modality inputs, and examine training loss and learned gate coefficients. The findings show differences among variants and settings without making training-curve behavior a validation result.

Component-wise Evaluation

  • With RFNet, TG-BWSA scores 0.8982 mean Dice versus 0.8955 for fixed-weight BWSA and 0.8947 for PMA; removing the original-feature path lowers mean Dice to 0.8944.PMA uses four independent adapters with similar total parameter count but does not combine modality features within an adapter.Source: 1) Component-wise evaluation of BWSA… · pp. 7–8, Table 12 · p. 8

Gate Initialization and Regularization

  • Across five tested gate initialization and regularization settings, WT Dice ranges from 0.9263 to 0.9293, TC from 0.8964 to 0.9007, and ET from 0.8598 to 0.8647.The authors describe sensitivity to these five tested settings as limited.Source: 2) Gate initialization and… · p. 8, Table 13 · p. 8

Performance with Missing Modalities

  • Across 14 incomplete modality configurations, both BWSA forms improve mean WT, TC, and ET Dice over RFNet; fixed weights score higher on TC, while gates score slightly higher on WT and ET.Fixed versus gated Dice is 0.796 vs. 0.779 for TC, 0.834 vs. 0.825 for WT, and 0.629 vs. 0.627 for ET; the complete-input case is separate.Source: 3) Performance with Missing Modalities · pp. 8, 10, Table 14 · p. 10

Training Loss and Gate Analysis

  • TG-BWSA’s training-loss curve is lower over most plotted epochs, but training loss alone does not establish convergence rate or validation performance.The dashed line marks region-fusion loss activation near epoch 100.Source: Figure 5 · p. 8, 4) Training Loss and Gate Analysis · p. 10
  • Learned gate coefficients differ by modality and stage and remain fixed across test inputs; FLAIR Stage 4 α rises from near 0.5 at initialization to 0.9220.For T1ce at Stage 2, final α and β are close: 0.7056 and 0.7016; coefficients do not link an individual gate to a particular tumor region.Source: Figure 6 · p. 9, 4) Training Loss and Gate Analysis · p. 10

V. DISCUSSION

Both adapter variants improve mean Dice over unmodified models across four backbones, but fixed and learned weights trade off by metric. The discussion also notes limits in test-time adaptation, resource measurements, and evaluation scope.

V. DISCUSSION

  • On BraTS 2023 online validation, TG-BWSA has higher average Dice on all four backbones, but fixed-weight RFNet has lower average HD95 (14.37 vs. 16.34 mm).For a given backbone, the authors recommend choosing between fixed blending and learned weights based on validation performance for the metric of interest.Source: §V · p. 10
  • Gate coefficients are fixed after training and cannot adjust to the reliability of an individual test image.Source: §V · p. 10
  • In the RFNet ablation, TG-BWSA has higher mean Dice than PMA at a similar parameter count (0.8982 vs. 0.8947); resource measurements cover only RFNet on an NVIDIA A6000.Source: §V · p. 10
  • For lesion-wise online metrics, RFNet+TG-BWSA has higher average Dice (0.852 vs. 0.848) and lower average HD95 (20.61 vs. 23.40 mm) than RFNet+BWSA, but remains below the three highest-scoring teams (0.876–0.888).Because submissions use different training and inference pipelines, these scores provide benchmark context rather than an isolated architecture comparison.Source: §V · p. 10
  • Evaluation lacks an external cohort; missing modalities are simulated by zeroing complete volumes without modality dropout training, and online-score variation across seeds was not measured.Source: LIMITATIONS AND FUTURE WORK · p. 10
  • The LoRA comparison changes parameter count and module placement, so it does not measure efficiency per parameter.Source: LIMITATIONS AND FUTURE WORK · p. 10

VI. CONCLUSION

BWSA and TG-BWSA improve BraTS online validation results with modest parameter overhead, and the choice between them should follow validation performance on the target metric. The ethics statement reports that the secondary analysis used deidentified public data and required no additional approval or consent.

VI. CONCLUSION

  • On official BraTS 2023 online validation, BWSA raises average region-wise Dice by 1.3–7.6 percentage points and TG-BWSA by 2.8–7.7 points versus unmodified backbones; both reduce average HD95.Both add approximately 0.85–3.73% parameters relative to the unmodified backbones.Source: §VI · p. 10
  • BWSA lets separate modality encoders combine features through one shared adapter between encoder stages.Source: §VI · p. 10
  • The choice between fixed blending and learned weights should follow validation results for the selected backbone and evaluation metric.Source: §VI · p. 10

ETHICS STATEMENT

  • The secondary analysis used publicly available, deidentified BraTS 2020 and BraTS 2023 data, with no additional institutional review board approval or informed consent required.Source: ETHICS STATEMENT · pp. 10–11

Related papers

Loading 2610.18571v1…