PaperlayerView exact revision on arXiv

Source-linked AI summary

Latent Correlation Representation Learning for Brain Tumor Segmentation with Missing MRI Modalities

Tongxue Zhou, Stéphane Canu, Pierre Vera, Su Ruan

arXiv:2104.06231v2eess.IVcs.CV

TL;DR

Brain tumor segmentation needs complementary MRI modalities, yet clinical data often omit some sequences. The paper proposes a multi-encoder network that models latent cross-modality correlations and fuses them for segmentation, reporting robust missing-modality performance and better results than state-of-the-art methods when modalities are gradually missing.

  • Problem

    Clinical MRI modalities can be incomplete, while robust segmentation requires a shared representation that accepts different modality subsets.

    Source: §I · p. 2

  • Method

    A multi-encoder network estimates modality-specific parameters, transforms individual features into latent multi-source correlation representations, and fuses them for reconstruction and segmentation.

    Source: A. Modeling the latent multi-source… · p. 3, §V · p. 9

  • Results

    The method achieves better results than state-of-the-art approaches when modalities are gradually missing, although it does not surpass the best method with full modalities.

    Source: §V · p. 10

  • Takeaways & Limitations

    Modeling correlations between modalities supports brain tumor segmentation when one or more MRI modalities are unavailable.

    Source: §V · p. 10

Abstract

from arXiv · show

Magnetic Resonance Imaging (MRI) is a widely used imaging technique to assess brain tumor. Accurately segmenting brain tumor from MR images is the key to clinical diagnostics and treatment planning. In addition, multi-modal MR images can provide complementary information for accurate brain tumor segmentation. However, it's common to miss some imaging modalities in clinical practice. In this paper, we present a novel brain tumor segmentation algorithm with missing modalities. Since it exists a strong correlation between multi-modalities, a correlation model is proposed to specially represent the latent multi-source correlation. Thanks to the obtained correlation representation, the segmentation becomes more robust in the case of missing modality. First, the individual representation produced by each encoder is used to estimate the modality independent parameter. Then, the correlation model transforms all the individual representations to the latent multi-source correlation representations. Finally, the correlation representations across modalities are fused via attention mechanism into a shared representation to emphasize the most important features for segmentation. We evaluate our model on BraTS 2018 and BraTS 2019 dataset, it outperforms the current state-of-the-art methods and produces robust results when one or more modalities are missing.

I. INTRODUCTION

Brain tumor segmentation benefits from complementary MRI modalities, but clinical scans often lack one or more modalities. The paper addresses this gap with a latent-correlation and attention-based network for robust missing-modality segmentation.

Motivation

  • FLAIR, T1c, T1, and T2 provide complementary information about whole tumor, tumor core, and enhancing tumor regions.FLAIR and T2 highlight whole tumor with peritumoral edema, while T1 and T1c highlight tumor core; T1c also shows enhancing tumor core.Source: §I · p. 1

Challenge

  • Segmentation remains difficult because anatomy, glioma appearance, intensity ranges, and tumor contrast vary across patients.Low contrast can make tumor contours fuzzy, especially when modalities are missing.Source: §I · p. 1

Related approaches

  • The network targets missing-modality segmentation because prior approaches commonly require complete modality sets or have limitations in latent-feature fusion.Source: §I · pp. 1–2
  • Existing missing-modality strategies train on all modality subsets, synthesize absent modalities, or fuse available modalities, each with stated complexity or quality constraints.Subset training is time-consuming; synthesis adds a network and makes segmentation dependent on synthesis quality.Source: §I · p. 2

Contribution

  • The proposed method learns latent correlations between modality features and uses attention-based fusion to emphasize informative representations for segmentation.The authors identify the correlation model and attention-based fusion as core contributions for missing-modality segmentation.Source: §I · pp. 2–3

Challenge

  • Missing-modality segmentation must learn a shared latent representation that accepts any subset of modalities and still produces robust segmentation.Source: §I · p. 2

II. METHOD

The method encodes each available MRI modality separately, models latent cross-modality correlations, and fuses the resulting representations for reconstruction and segmentation. Its correlation model uses modality-specific parameters to describe missing modalities from available inputs.

Network architecture

  • Individual encoders extract modality-specific representations while allowing the network to operate with available modalities.The multi-encoder design also avoids false adaptation between modalities.Source: §II · p. 3
  • The architecture projects encoded modalities through the Correlation Model and Fusion block, then decodes the fused representation into reconstructed modalities and a segmentation result.Source: Figure 2 · p. 3, §II · p. 3

Correlation Model

  • The Correlation Model contains parameter-estimation and linear-correlation modules that transform individual features into latent multi-source correlation representations.Source: A. Modeling the latent multi-source… · p. 3
  • The parameter-estimation module maps each modality-specific representation to unique parameters Γ_i = {α_i, β_i, γ_i, δ_i}.It uses two fully connected layers and LeakyReLU to estimate the modality-independent parameters.Source: A. Modeling the latent multi-source… · p. 3
  • The linear correlation expression combines representations from other modalities using learned parameters to construct each modality’s correlation representation.Equation (1) applies weighted terms from the other modalities, with i ≠ j ≠ k ≠ m.Source: A. Modeling the latent multi-source… · p. 3

Missing-modality handling

B. Fusion

The fusion block combines modality correlation representations using channel and spatial attention, producing a fused representation that emphasizes informative features. This addresses limitations of equal-weight mean and maximum fusion.

  • The correlation model transforms modality-specific representations into latent multi-source correlation representations before fusion.Source: Figure 4 · p. 4, Figure 5 · p. 4
  • Mean fusion weights every representation equally, which can lose valuable latent information.Source: B. Fusion · p. 4
  • Maximum fusion selects the most discriminative representation but may ignore contributions from other representations.Source: B. Fusion · p. 4
  • Correlation representations are concatenated before channel and spatial attention recalibrate their feature importance.Source: B. Fusion · p. 4, Figure 5 · p. 4
  • Channel attention applies learned weights to the representation channels, producing the channel-attention representation Fc.Source: B. Fusion · p. 4
  • Spatial attention weights feature representations at different spatial locations, producing Fs before fusion.Source: B. Fusion · p. 4
  • The final fused representation Ff is obtained by adding the channel- and spatial-attention representations.Source: B. Fusion · p. 4
  • The proposed attention fusion can adapt to multimodal fusion and is reported as superior to simple mean and maximum fusion.Source: B. Fusion · p. 4

C. Network Architecture and Learning Process

The network uses encoder–decoder architecture with modality-specific feature extraction, skip connections, and residual dilated convolutions. These components support multiscale information and spatial-detail preservation.

D. The choices of loss function

Training combines segmentation and reconstruction objectives. Dice loss measures prediction overlap, while MAE matches reconstructed modalities to their inputs, with equal component weighting in the experiment.

III. EXPERIMENTAL SETUP

Experiments use BraTS 2018 and BraTS 2019, whose cases contain four MRI modalities and expert-labeled ground truth. The data were standardized and additionally processed as 3D volumes for training.

B. Implementation details

The network uses modality-specific encoders, correlation transformation, fusion, and decoding, while training uses standard optimization and evaluation procedures.

  • The model is implemented in Keras on a single Nvidia Quadro P5000 GPU with 16G memory.Source: B. Implementation details · p. 5
  • Training uses Adam with an initial learning rate of 5e-4, factor-0.5 decay after 10-epoch patience, and early stopping after 50 unimproved epochs.Source: B. Implementation details · p. 5
  • The dataset is randomly split into 80% training and 20% testing, with results obtained through an online evaluation platform.Source: B. Implementation details · p. 5
  • The evaluation uses Dice Score, Sensitivity, and Hausdorff Distance to quantify segmentation accuracy.Dice measures overlap, Sensitivity measures correctly identified actual positives, and Hausdorff Distance indicates the largest boundary error.Source: C. Evaluation metrics · p. 5
  • Each modality is encoded separately, transformed by CM into dependent representations, fused, and decoded into the final segmentation.Source: Figure 6 · p. 6

IV. EXPERIMENT RESULTS

Experiments evaluate component contributions and compare the method with existing approaches on BraTS 2018 and 2019, showing competitive full-modality performance.

3) Comparison with the state-of-the-art methods on missing modalities:

The correlation model is evaluated for segmentation with missing modalities and is associated with more gradual performance degradation than comparison methods.

  • Across missing-modality settings, the method achieves the best results in most cases, covering 13 of 15 evaluated comparisons.Source: 3) Comparison with the… · p. 7
  • Compared with HeMIS, the method’s Dice score drops gradually as modalities are removed, whereas HeMIS shows a more severe performance decline.Source: 3) Comparison with the… · p. 7
  • Compared with WoCM, CM improves average Dice Score by 5.8%, 15.3%, and 15.6% for whole, core, and enhancing tumor, respectively.Source: 3) Comparison with the… · p. 7
  • Missing FLAIR sharply decreases Dice score across regions, while missing T1c severely decreases Dice score for tumor core and enhancing tumor.Source: 3) Comparison with the… · p. 7
  • The authors report promising results on both complete and missing-modality datasets, while noting that limited hardware restricted the model to 8 initial filters.Source: 3) Comparison with the… · p. 7

B. Qualitative analysis

Qualitative and tabulated analyses examine component integration and missing-modality behavior, with progressively improved segmentations and gradual degradation as modalities disappear.

  • The qualitative analysis visualizes full-modality results in Fig. 7 and Fig. 8 and missing-modality results in Fig. 9.Source: B. Qualitative analysis · pp. 7, 9
  • Tables I and II evaluate the proposed method on BraTS 2018 and BraTS 2019 training sets, marking improvements and best scores by tumor region.Source: Table I · p. 8, Table II · p. 8
  • Segmentation results improve progressively as the proposed strategies are integrated, and the complete model produces results reported as nearly matching ground truth.Source: B. Qualitative analysis · p. 9
  • With increasing missing modalities, the robust model’s segmentation results degrade slightly rather than suddenly.Source: B. Qualitative analysis · p. 9
  • Using only FLAIR yields good whole-tumor segmentation, while FLAIR plus T1c produces a competitive result relative to ground truth.Source: B. Qualitative analysis · p. 9

C. Visualization of the attention based fusion block

The attention-based fusion block makes tumor-related features more identifiable by emphasizing tumor locations and modality-specific feature strength before fusion.

  • Spatial attention emphasizes the positions of tumor regions in the independent feature representations.The tumor characteristics are initially not obvious in F1, F2, F3, and F4, but become more localized after spatial attention.Source: C. Visualization of the attention… · p. 9
  • Channel attention highlights the differing contributions of MRI modalities to tumor representation.FLAIR focuses on the whole tumor region, whereas T1c highlights tumor-core regions.Source: C. Visualization of the attention… · p. 9
  • Combining spatial- and channel-attention representations produces fused features in which the relevant tumor regions are clearly identified.Source: C. Visualization of the attention… · p. 9

V. DISCUSSION AND CONCLUSION

The proposed multi-encoder network combines latent cross-modality correlation modeling with attention-based fusion for segmentation when MRI modalities are missing. Ablations support the contribution of its components, and evaluations show better results than state-of-the-art methods as modalities are progressively removed, although full-modality performance was not best; the authors identify more complex correlation modeling as future work.

V. DISCUSSION AND CONCLUSION

  • The network uses encoders, a correlation model, an attention-based fusion block, and a decoder to support segmentation with missing modalities.The correlation model discovers latent correlations among four modality-specific feature representations, while attention selectively emphasizes channel- and spatial-level information.Source: §V · p. 9
  • Ablation experiments show that the fusion block, reconstruction decoder, and correlation model progressively refine segmentation until the best results are achieved.Source: §V · p. 9
  • Better quantitative and qualitative results than state-of-the-art methods are reported when modalities are gradually missing on BraTS 2018 and BraTS 2019.Source: §V · p. 10
  • The method remains competitive but does not surpass the best method when all modalities are available.Source: §V · p. 10
  • The authors plan to investigate more complex models for multi-source correlation representation and missing-data adaptation.Source: §V · p. 10

Related papers

Loading 2104.06231v2…