PaperlayerView exact revision on arXiv

Source-linked AI summary

Latent Correlation Representation Learning for Brain Tumor Segmentation with Missing MRI Modalities

Tongxue Zhou, Stéphane Canu, Pierre Vera, Su Ruan

arXiv:2104.06231v2eess.IVcs.CV

TL;DR

Brain tumor segmentation needs complementary MRI modalities, yet clinical data often omit some sequences. The paper proposes a multi-encoder network that models latent cross-modality correlations and fuses them for segmentation, reporting robust missing-modality performance and better results than state-of-the-art methods when modalities are gradually missing.

  • Problem

    Clinical MRI modalities can be incomplete, while robust segmentation requires a shared representation that accepts different modality subsets.

  • Method

    A multi-encoder network estimates modality-specific parameters, transforms individual features into latent multi-source correlation representations, and fuses them for reconstruction and segmentation.

  • Results

    The method achieves better results than state-of-the-art approaches when modalities are gradually missing, although it does not surpass the best method with full modalities.

  • Takeaways & Limitations

    Modeling correlations between modalities supports brain tumor segmentation when one or more MRI modalities are unavailable.

Abstract

from arXiv · show

Magnetic Resonance Imaging (MRI) is a widely used imaging technique to assess brain tumor. Accurately segmenting brain tumor from MR images is the key to clinical diagnostics and treatment planning. In addition, multi-modal MR images can provide complementary information for accurate brain tumor segmentation. However, it's common to miss some imaging modalities in clinical practice. In this paper, we present a novel brain tumor segmentation algorithm with missing modalities. Since it exists a strong correlation between multi-modalities, a correlation model is proposed to specially represent the latent multi-source correlation. Thanks to the obtained correlation representation, the segmentation becomes more robust in the case of missing modality. First, the individual representation produced by each encoder is used to estimate the modality independent parameter. Then, the correlation model transforms all the individual representations to the latent multi-source correlation representations. Finally, the correlation representations across modalities are fused via attention mechanism into a shared representation to emphasize the most important features for segmentation. We evaluate our model on BraTS 2018 and BraTS 2019 dataset, it outperforms the current state-of-the-art methods and produces robust results when one or more modalities are missing.

I. INTRODUCTION

Brain tumor segmentation benefits from complementary MRI modalities, but clinical scans often lack one or more modalities. The paper addresses this gap with a latent-correlation and attention-based network for robust missing-modality segmentation.

  • Motivation: FLAIR, T1c, T1, and T2 provide complementary information about whole tumor, tumor core, and enhancing tumor regions.FLAIR and T2 highlight whole tumor with peritumoral edema, while T1 and T1c highlight tumor core; T1c also shows enhancing tumor core.
  • Challenge: Segmentation remains difficult because anatomy, glioma appearance, intensity ranges, and tumor contrast vary across patients.Low contrast can make tumor contours fuzzy, especially when modalities are missing.
  • Related approaches: The network targets missing-modality segmentation because prior approaches commonly require complete modality sets or have limitations in latent-feature fusion.
  • Related approaches: Existing missing-modality strategies train on all modality subsets, synthesize absent modalities, or fuse available modalities, each with stated complexity or quality constraints.Subset training is time-consuming; synthesis adds a network and makes segmentation dependent on synthesis quality.
  • Contribution: The proposed method learns latent correlations between modality features and uses attention-based fusion to emphasize informative representations for segmentation.The authors identify the correlation model and attention-based fusion as core contributions for missing-modality segmentation.
  • Challenge: Missing-modality segmentation must learn a shared latent representation that accepts any subset of modalities and still produces robust segmentation.

II. METHOD

The method encodes each available MRI modality separately, models latent cross-modality correlations, and fuses the resulting representations for reconstruction and segmentation. Its correlation model uses modality-specific parameters to describe missing modalities from available inputs.

  • Network architecture: Individual encoders extract modality-specific representations while allowing the network to operate with available modalities.The multi-encoder design also avoids false adaptation between modalities.
  • Network architecture: The architecture projects encoded modalities through the Correlation Model and Fusion block, then decodes the fused representation into reconstructed modalities and a segmentation result.
  • Correlation Model: The Correlation Model contains parameter-estimation and linear-correlation modules that transform individual features into latent multi-source correlation representations.
  • Correlation Model: The parameter-estimation module maps each modality-specific representation to unique parameters Γ_i = {α_i, β_i, γ_i, δ_i}.It uses two fully connected layers and LeakyReLU to estimate the modality-independent parameters.
  • Correlation Model: The linear correlation expression combines representations from other modalities using learned parameters to construct each modality’s correlation representation.Equation (1) applies weighted terms from the other modalities, with i ≠ j ≠ k ≠ m.
  • Missing-modality handling: Available modalities can describe a missing modality through the correlation model, supporting segmentation when imaging data are incomplete.

B. Fusion

The fusion block combines modality correlation representations using channel and spatial attention, producing a fused representation that emphasizes informative features. This addresses limitations of equal-weight mean and maximum fusion.

  • The correlation model transforms modality-specific representations into latent multi-source correlation representations before fusion.
  • Mean fusion weights every representation equally, which can lose valuable latent information.
  • Maximum fusion selects the most discriminative representation but may ignore contributions from other representations.
  • Correlation representations are concatenated before channel and spatial attention recalibrate their feature importance.
  • Channel attention applies learned weights to the representation channels, producing the channel-attention representation Fc.
  • Spatial attention weights feature representations at different spatial locations, producing Fs before fusion.
  • The final fused representation Ff is obtained by adding the channel- and spatial-attention representations.
  • The proposed attention fusion can adapt to multimodal fusion and is reported as superior to simple mean and maximum fusion.

C. Network Architecture and Learning Process

The network uses encoder–decoder architecture with modality-specific feature extraction, skip connections, and residual dilated convolutions. These components support multiscale information and spatial-detail preservation.

  • The encoder obtains modality-specific representations through convolutional blocks, residual dilated blocks, and skip connections.
  • Stride-2 convolutions replace pooling to help maintain spatial information during encoding.
  • Residual dilated convolutions with rates 2 and 4 are used in both encoder and decoder paths for multiscale features.
  • Decoder levels upsample, apply convolution, and reduce the number of features by a factor of 2.

D. The choices of loss function

Training combines segmentation and reconstruction objectives. Dice loss measures prediction overlap, while MAE matches reconstructed modalities to their inputs, with equal component weighting in the experiment.

  • The overall loss function combines multiple objective components with trade-off parameters.
  • The trade-off parameter λ weights each loss component and is set to 1 in the experiment.
  • Dice loss evaluates the overlap between predicted segmentation results and ground truth.
  • Reconstruction uses Mean Absolute Error to match each reconstructed image to its input modality.
  • The loss notation defines N as examples, C as classes, p_ij as predicted probabilities, g_ij as ground-truth probabilities, and ϵ as a stabilizing constant.

III. EXPERIMENTAL SETUP

Experiments use BraTS 2018 and BraTS 2019, whose cases contain four MRI modalities and expert-labeled ground truth. The data were standardized and additionally processed as 3D volumes for training.

  • BraTS 2018 contains 285 training cases and 66 validation cases with hidden ground truth.
  • BraTS 2019 contains 335 training cases and 125 validation cases with hidden ground truth.
  • Each dataset case includes T1, T1c, T2, and FLAIR MRI modalities.
  • Organizers co-registered, resampled, and skull-stripped the images, with ground truth manually labeled by experts.
  • Additional preprocessing crops and resizes volumes from 155 × 240 × 240 to 128 × 128 × 128 and applies N4ITK distortion correction.

B. Implementation details

The network uses modality-specific encoders, correlation transformation, fusion, and decoding, while training uses standard optimization and evaluation procedures.

  • The model is implemented in Keras on a single Nvidia Quadro P5000 GPU with 16G memory.
  • Training uses Adam with an initial learning rate of 5e-4, factor-0.5 decay after 10-epoch patience, and early stopping after 50 unimproved epochs.
  • The dataset is randomly split into 80% training and 20% testing, with results obtained through an online evaluation platform.
  • The evaluation uses Dice Score, Sensitivity, and Hausdorff Distance to quantify segmentation accuracy.Dice measures overlap, Sensitivity measures correctly identified actual positives, and Hausdorff Distance indicates the largest boundary error.
  • Each modality is encoded separately, transformed by CM into dependent representations, fused, and decoded into the final segmentation.

IV. EXPERIMENT RESULTS

Experiments evaluate component contributions and compare the method with existing approaches on BraTS 2018 and 2019, showing competitive full-modality performance.

  • Experiments assess component importance, full-modality state-of-the-art comparisons, and robustness when modalities are missing.
  • The ablation studies evaluate the fusion block, reconstruction decoders, and CM against a baseline on BraTS 2018 and 2019.
  • 77.4 average Dice Score, 9.6 average Hausdorff Distance, and 71.7 average Sensitivity are reported for the BraTS 2019 baseline.
  • On BraTS 2018, CM improves the baseline by 2.6% average Dice Score, 20.2% average Hausdorff Distance, and 3.0% average Sensitivity.
  • On BraTS 2019, CM improves the baseline by 2.5% average Dice Score and 27.1% average Hausdorff Distance.
  • On BraTS 2018 validation data, the method achieves the second-best Dice Score across tumor regions, second-best Hausdorff Distance on tumor core, and best average Sensitivity.
  • On BraTS 2019 validation data, the method surpasses three compared methods in average Dice Score, obtains the best Hausdorff Distance, and ranks second in Sensitivity for two regions.

3) Comparison with the state-of-the-art methods on missing modalities:

The correlation model is evaluated for segmentation with missing modalities and is associated with more gradual performance degradation than comparison methods.

  • Across missing-modality settings, the method achieves the best results in most cases, covering 13 of 15 evaluated comparisons.
  • Compared with HeMIS, the method’s Dice score drops gradually as modalities are removed, whereas HeMIS shows a more severe performance decline.
  • Compared with WoCM, CM improves average Dice Score by 5.8%, 15.3%, and 15.6% for whole, core, and enhancing tumor, respectively.
  • Missing FLAIR sharply decreases Dice score across regions, while missing T1c severely decreases Dice score for tumor core and enhancing tumor.
  • The authors report promising results on both complete and missing-modality datasets, while noting that limited hardware restricted the model to 8 initial filters.

B. Qualitative analysis

Qualitative and tabulated analyses examine component integration and missing-modality behavior, with progressively improved segmentations and gradual degradation as modalities disappear.

  • The qualitative analysis visualizes full-modality results in Fig. 7 and Fig. 8 and missing-modality results in Fig. 9.
  • Tables I and II evaluate the proposed method on BraTS 2018 and BraTS 2019 training sets, marking improvements and best scores by tumor region.
  • Segmentation results improve progressively as the proposed strategies are integrated, and the complete model produces results reported as nearly matching ground truth.
  • With increasing missing modalities, the robust model’s segmentation results degrade slightly rather than suddenly.
  • Using only FLAIR yields good whole-tumor segmentation, while FLAIR plus T1c produces a competitive result relative to ground truth.

C. Visualization of the attention based fusion block

The attention-based fusion block makes tumor-related features more identifiable by emphasizing tumor locations and modality-specific feature strength before fusion.

  • Spatial attention emphasizes the positions of tumor regions in the independent feature representations.The tumor characteristics are initially not obvious in F1, F2, F3, and F4, but become more localized after spatial attention.
  • Channel attention highlights the differing contributions of MRI modalities to tumor representation.FLAIR focuses on the whole tumor region, whereas T1c highlights tumor-core regions.
  • Combining spatial- and channel-attention representations produces fused features in which the relevant tumor regions are clearly identified.

V. DISCUSSION AND CONCLUSION

The proposed multi-encoder network combines latent cross-modality correlation modeling with attention-based fusion for segmentation when MRI modalities are missing. Ablations support the contribution of its components, and evaluations show better results than state-of-the-art methods as modalities are progressively removed, although full-modality performance was not best; the authors identify more complex correlation modeling as future work.

  • V. DISCUSSION AND CONCLUSION: The network uses encoders, a correlation model, an attention-based fusion block, and a decoder to support segmentation with missing modalities.The correlation model discovers latent correlations among four modality-specific feature representations, while attention selectively emphasizes channel- and spatial-level information.
  • V. DISCUSSION AND CONCLUSION: Ablation experiments show that the fusion block, reconstruction decoder, and correlation model progressively refine segmentation until the best results are achieved.
  • V. DISCUSSION AND CONCLUSION: Better quantitative and qualitative results than state-of-the-art methods are reported when modalities are gradually missing on BraTS 2018 and BraTS 2019.
  • V. DISCUSSION AND CONCLUSION: The method remains competitive but does not surpass the best method when all modalities are available.
  • V. DISCUSSION AND CONCLUSION: The authors plan to investigate more complex models for multi-source correlation representation and missing-data adaptation.
Loading 2104.06231v2…