PaperlayerView exact revision on arXiv

Source-linked AI summary

A review: Deep learning for medical image segmentation using multi-modality fusion

Tongxue Zhou, Su Ruan, Stéphane Canu

arXiv:2004.10664v2eess.IVcs.CVcs.LGstat.ML

TL;DR

Multi-modal medical image segmentation seeks to combine complementary imaging information, but limited labeled data and complex modality relationships complicate effective fusion. This review synthesizes deep learning architectures and fusion strategies, finding that later fusion can outperform earlier fusion when the fusion method is effective, while identifying data scarcity and fusion design as continuing challenges.

  • Problem

    Medical image segmentation needs accurate use of complementary modalities, yet limited labeled data and differing modality properties complicate effective multi-modal fusion.

  • Method

    The paper reviews deep learning-based multi-modal segmentation through data preparation, network architectures, fusion strategies, post-processing, and reported field problems.

  • Results

    Later fusion can achieve better performance than input-level fusion because it gives greater attention to learning relationships among modalities, when the fusion method is effective.

  • Takeaways & Limitations

    Future work should improve how multi-modal networks combine modalities, exploit their latent relationships, and integrate multi-information into segmentation.

  • Takeaways & Limitations

    Medical image segmentation is constrained by scarce expert-labeled data, which can cause over-fitting in complex neural networks.

Abstract

from arXiv · show

Multi-modality is widely used in medical imaging, because it can provide multiinformation about a target (tumor, organ or tissue). Segmentation using multimodality consists of fusing multi-information to improve the segmentation. Recently, deep learning-based approaches have presented the state-of-the-art performance in image classification, segmentation, object detection and tracking tasks. Due to their self-learning and generalization ability over large amounts of data, deep learning recently has also gained great interest in multi-modal medical image segmentation. In this paper, we give an overview of deep learning-based approaches for multi-modal medical image segmentation task. Firstly, we introduce the general principle of deep learning and multi-modal medical image segmentation. Secondly, we present different deep learning network architectures, then analyze their fusion strategies and compare their results. The earlier fusion is commonly used, since it's simple and it focuses on the subsequent segmentation network architecture. However, the later fusion gives more attention on fusion strategy to learn the complex relationship between different modalities. In general, compared to the earlier fusion, the later fusion can give more accurate result if the fusion method is effective enough. We also discuss some common problems in medical image segmentation. Finally, we summarize and provide some perspectives on the future research.

1 Introduction

Deep learning-based multi-modal medical image segmentation has grown as researchers seek to combine complementary imaging information for more accurate analysis. This review focuses specifically on fusion strategies, network architectures, field challenges, and future directions.

  • Motivation: Medical image segmentation supports diagnosis, monitoring, and treatment by labeling unhealthy tissue and anatomical structures.The task generally involves detecting areas of interest and delineating different structures.
  • Deep learning trend: Deep learning has been applied to segmentation across brain, lung, pancreas, prostate, and multi-organ imaging, with reported performance superior to traditional methods.The review places these applications within the broader success of deep learning in classification, segmentation, detection, and tracking.
  • Motivation: Multi-modal imaging combines complementary information from modalities such as CT, MRI, and PET to improve feature representation and network discrimination.CT, MRI, and PET provide different structural, soft-tissue, and functional information.
  • Review scope: Existing medical-imaging reviews cover deep learning broadly, but this paper concentrates on multi-modal image fusion methods for segmentation.Its scope includes fusion strategies rather than deep learning concepts alone.
  • Review scope: The paper covers deep learning principles, data preparation, fusion-based segmentation architectures, common problems, and future perspectives.The review is organized around preprocessing, network design, fusion strategies, field challenges, and concluding perspectives.

2 Deep learning based methods

This section introduces deep learning and frames multi-modal medical image segmentation as a pipeline involving network design, fusion strategies, datasets, and evaluation metrics.

  • 2.1 Deep learning: Deep learning uses multiple nonlinear processing layers to extract hierarchical features from large amounts of data.CNNs are identified as the most successful deep learning approach for image segmentation in the supplied discussion.
  • 2.1 Deep learning: CNNs combine convolution, pooling, activation, and fully connected layers, with weights updated through back-propagation after loss computation.Convolution performs feature extraction, pooling reduces spatial size, and activation functions introduce nonlinearity.
  • 2.2 Multi-modal medical image segmentation: Because target tissues vary in size, shape, and location, segmentation remains challenging, motivating publicly available challenges for standardized comparison.Examples include BraTS, ISLES, MRBrainS, CHAOS, NeoBrainS, Iseg-2017, and IVDM3Seg.
  • 2.2 Multi-modal medical image segmentation: The proposed deep learning pipeline contains data preparation, network architecture, fusion strategy, and post-processing stages.Fusion is categorized by network location into input-level, layer-level, and decision-level strategies; post-processing can refine results using morphological techniques or conditional random fields.

3 Data processing

Data processing prepares multi-modal medical images for segmentation through dimension selection, pre-processing, augmentation, and post-processing. These steps address computational cost, modality variation, limited labels, overfitting, and segmentation artefacts.

  • Data processing: The section identifies data dimension selection, image pre-processing, augmentation, and post-processing as important components of deep learning segmentation.Together, these operations prepare inputs and refine outputs around the segmentation network.
  • Data dimension selection: Medical image models may process full 3D volumes, which exploit spatial context but have expensive computational cost, or process images slice by slice.Some approaches instead use smaller 3D patches to reduce the burden of full-volume processing.
  • Image pre-processing: Pre-processing reduces variation among modalities caused by differences in intensity, contrast, and noise before network training.Typical techniques include image registration, bias field correction, and intensity normalization.
  • Data augmentation: Limited expert-labeled data makes overfitting a concern when training large neural networks, so data augmentation creates additional transformed images.Reported transformations include rotation, translation, scaling, flipping, distortion, and adding noise.
  • Post-processing: Post-processing refines segmentation outputs by removing small incorrect fragments and retaining the largest connected volume when appropriate.Morphological techniques are preferred for this purpose, and processing can be adapted to the detected region's structure.

4 Multi-modal segmentation networks

The review categorizes multi-modal segmentation networks by where modalities are fused: input, layers, or decision. Input-level fusion is common and supports varied sequential architectures, while layer-level fusion uses separate modality pathways and feature integration to model cross-modal relationships.

  • Fusion strategy taxonomy: Multi-modal segmentation networks are categorized into input-level, layer-level, and decision-level fusion according to their fusion strategies.This categorization organizes the reviewed architectures by where multi-modal information is combined.
  • Input-level fusion network: Input-level fusion concatenates modalities channel by channel as multi-channel inputs before training the segmentation network.Most reviewed networks directly integrate modalities in the original input space.
  • Input-level fusion network: Input-level fusion preserves original image information and supports multi-task, multi-view, multi-scale, and GAN-based segmentation architectures.These sequential architectures are used to exploit feature representations from multi-modal images.
  • Layer-level fusion: Layer-level fusion learns modality-specific feature representations separately, fuses them within network layers, and feeds the fused result to the decision layer.Dense connections are commonly used to capture complex relationships between modalities and reduce over-fitting risk on smaller training sets.
  • Layer-level fusion: A dual-pathway layer-level network processes T2 and Flair separately from T1 and T1c before fusing features for four-class tumor segmentation.The architecture uses distinct pathways to learn complementary representations from the MRI modalities.

5 Common problems

Medical image segmentation commonly faces data scarcity, over-fitting, and severe class imbalance. The review describes data-, algorithm-, and loss-level strategies to address these problems.

  • 5.1 Over-fitting: Limited labeled data can cause over-fitting, so reducing model complexity or augmenting training data is necessary.Augmentation can use scaling, rotation, translation, brightness variation, elastic deformation, flipping, and mirroring.
  • 5.2 Class imbalance: Class imbalance can bias segmentation toward large regions, making loss-function selection especially important.In BraTS 2017, background examples overwhelmingly outnumber tumor-region examples.
  • 5.2 Class imbalance: Patch sampling mitigates imbalance by preferentially centering training patches on lesions or healthy voxels.One balanced strategy samples patches with 50% probability centered on lesions or healthy voxels; lesion-centered sampling adds random offsets to reduce location bias.
  • 5.2 Class imbalance: Algorithm-level approaches include two-phase training, multi-task segmentation, and ensemble learning to improve handling of imbalanced classes.Two-phase training first constructs an equiprobable patch dataset, then retrains only the output layer to calibrate probabilities.
  • 5.2 Class imbalance: Loss functions address imbalance through class weighting, overlap-based objectives, or emphasis on difficult examples.Weighted cross-entropy incorporates label weights, Dice loss measures prediction–ground-truth overlap, and focal loss down-weights easy examples.
  • 5.2 Class imbalance: Generalized Dice loss is reported as more robust than Dice and cross-entropy losses across different label-imbalance rates in 2D and 3D segmentation.Its class-rebalancing properties motivate its use for unbalanced tasks.

6 Discussion and conclusion

The review compares deep learning fusion strategies for multi-modal medical image segmentation and identifies architecture, data, and evaluation challenges that shape future work.

  • Dataset and benchmark landscape: Public multi-modal segmentation datasets are rare, with BraTS the most commonly used dataset and methods summarized from 2013 onward.The review notes that deep learning methods were applied to BraTS from 2013, while publicly available multi-modal segmentation datasets remain limited.
  • Network architecture and fusion: The strongest reported BraTS method combines input-level fusion, a CNN encoder-decoder, and a variational autoencoder branch for reconstruction and encoder guidance.The VAE branch also provides regularization, while the authors report that additional augmentation, post-processing, or network depth did not further improve performance.
  • Fusion strategies: Input-level fusion is simple and preserves modality features, whereas layer-level and decision-level strategies model modality information differently and may improve performance under different trade-offs.Layer-level fusion can capture complex relationships through dense connections; decision-level fusion uses separate modality networks but requires more memory and computation.
  • Evaluation caveat: Comparing fusion strategies by performance is difficult because their reported results were obtained on different data, so each strategy retains distinct advantages and disadvantages.The review cautions against treating the apparent superiority of one strategy as a controlled comparison.
  • Future directions: Future research should improve how modalities are combined, model their latent relationships, and integrate multi-information efficiently into segmentation networks.The review frames these design questions as open topics for multi-modal medical image segmentation.
  • Data and training challenges: Limited data, class imbalance, and difficult optimization can cause overfitting or unstable segmentation, motivating simpler architectures, resampling, tailored losses, augmentation, and faster convolution methods.The review also identifies public code release as a practice that can accelerate research.
Loading 2004.10664v2…