Source-linked AI summary
Multi-branch Convolutional Neural Network for Multiple Sclerosis Lesion Segmentation
Shahab Aslani, Michael Dayan, Loredana Storelli, Massimo Filippi, Vittorio Murino, Maria A Rocca, Diego Sona
TL;DR
The paper addresses automated MS lesion segmentation from multimodal brain MRI while avoiding limitations associated with 3D and patch-based approaches. It uses a whole-brain, slice-based multi-branch 2D CNN with modality-specific processing, multiscale fusion, and orthogonal-plane reconstruction. The method achieved stronger segmentation performance than single-branch alternatives on reported measures, while remaining somewhat slow and requiring further validation for clinical use.
Problem
Automated MS lesion segmentation is needed because manual delineation is time-consuming and variable, while existing 3D and patch-based approaches have dataset-size or contextual limitations.
Method
The method is a whole-brain slice-based multi-branch 2D CNN with modality-specific branches, multiscale feature fusion and upsampling, and multi-plane reconstruction.
Results
DSC=0.7649 versus 0.6844 and 0.6980 for cited single-branch methods, while the proposed method showed the best trade-off between LTPR and LFPR.
Takeaways & Limitations
The approach provides a robust lesion-location trade-off and reports top performance across two datasets for automated MS lesion segmentation.
Takeaways & Limitations
Segmentation takes approximately 90 seconds per 3D image, and clinical readiness requires validation on healthy subjects and longitudinal data.
Abstract
from arXiv · showhide
In this paper, we present an automated approach for segmenting multiple sclerosis (MS) lesions from multi-modal brain magnetic resonance images. Our method is based on a deep end-to-end 2D convolutional neural network (CNN) for slice-based segmentation of 3D volumetric data. The proposed CNN includes a multi-branch downsampling path, which enables the network to encode information from multiple modalities separately. Multi-scale feature fusion blocks are proposed to combine feature maps from different modalities at different stages of the network. Then, multi-scale feature upsampling blocks are introduced to upsize combined feature maps to leverage information from lesion shape and location. We trained and tested the proposed model using orthogonal plane orientations of each 3D modality to exploit the contextual information in all directions. The proposed pipeline is evaluated on two different datasets: a private dataset including 37 MS patients and a publicly available dataset known as the ISBI 2015 longitudinal MS lesion segmentation challenge dataset, consisting of 14 MS patients. Considering the ISBI challenge, at the time of submission, our method was amongst the top performing solutions. On the private dataset, using the same array of performance metrics as in the ISBI challenge, the proposed approach shows high improvements in MS lesion segmentation compared with other publicly available tools.
1. Introduction
MS lesion segmentation is clinically important but manual delineation is slow and variable, motivating automated methods. This paper proposes a whole-brain, slice-based multi-branch 2D CNN that combines modality-specific and multiscale information across orthogonal views.
- Motivation: Manual MS lesion segmentation is time-consuming, tedious, and prone to intra- and inter-observer variability, motivating automated approaches.
- Related approaches: 3D CNN segmentation can overfit small biomedical datasets because it fits many parameters, while 3D cross-hair convolution still uses more parameters than standard 2D convolution.
- Proposed approach: The proposed method is a whole-brain slice-based 2D encoder-decoder CNN designed to retain global structure while avoiding the overfitting of 3D methods and the limited context of patch-based methods.
- Proposed approach: Separate modality-specific branches, multilevel feature fusion and upsampling, and orthogonal-plane processing are used to capture modality-specific and contextual lesion information.
- Evaluation: The study evaluates model variants and reports top performance on two datasets for automated MS lesion segmentation.
2. Material
The study evaluates the method on the public ISBI 2015 longitudinal challenge dataset and an in-house NRU dataset of MS patients. The datasets include multimodal MRI and expert lesion annotations, with rater agreement assessed for the NRU data.
- Datasets: The evaluation uses the public ISBI 2015 Longitudinal MS Lesion Segmentation Challenge dataset and an in-house NRU dataset.
- ISBI dataset: The ISBI dataset contains 19 subjects split into 5 training and 14 test subjects, with longitudinal multimodal MRI and manually generated lesion masks.
- NRU dataset: The NRU dataset comprises 37 MS patients with varied clinical phenotypes and multimodal 3D FLAIR, T2w, and T1w acquisitions.
- Reference annotations: Two experienced readers independently delineated NRU lesions, achieving a mean DSC of 0.87 agreement.
- Ethics: The study received ethics approval and written informed consent from all participants.
3. Method
The method preprocesses multimodal 3D MRI into zero-padded 2D slices across three orthogonal orientations, then segments them with a multi-branch encoder-decoder and reconstructs a 3D lesion volume by majority voting.
- Data preprocessing: Each modality is registered and intensity-normalized, then converted into zero-padded 218×218 slices across axial, coronal, and sagittal orientations.The preprocessing produces equal-sized 2D inputs from each modality and orientation.
- Network architecture: The network uses separate parallel ResNets for FLAIR, T1w, and T2w to extract modality-specific multi-resolution features without weight sharing.The downsampling path organizes representations into five resolution levels.
- Network architecture: MMFF blocks combine same-resolution features across modalities, while MSFU blocks fuse multi-resolution features and progressively upsample them toward the input resolution.MMFF uses convolution and concatenation for multimodal fusion; MSFU combines upsampled low-resolution maps with higher-resolution maps.
- Training: The model is trained end-to-end on slices from all three views using soft Dice loss, Adam optimization, subject-level validation, and validation-based model selection.Training uses batches containing random slices from different orthogonal views and a maximum of 1000 epochs.
- Data availability: The NRU dataset is private and cannot be publicly released because of confidentiality, although the code is available by contacting the authors.
4. Experiments
The experiments compare the proposed model with established segmentation tools and evaluate it across ISBI and NRU datasets using multiple lesion-level, voxel-level, volume, and surface metrics.
- Evaluation metrics: The evaluation uses DSC, PPV, LTPR, LFPR, VD, SD, HD, and an overall evaluation score.These measures assess overlap, lesion detection and false positives, volume difference, surface distance, and aggregate performance.
- ISBI experiments: The first ISBI experiment uses five subjects with available ground truth in nested leave-one-subject-out cross-validation, evaluating annotations from two raters separately.Three subjects were used for training, one for validation, and one for testing.
- ISBI experiments: The official ISBI test experiment evaluates 14 subjects without available ground truth through the challenge web service after cross-validation on the five-subject training set.An ensemble of five trained models was applied to the official test set.
- NRU experiments: The NRU experiments use 37 subjects and cross-validation to test robustness, modality combinations, and comparisons with publicly available tools and a standard 2D U-Net.The modality experiment uses fixed training, validation, and test splits of 21, 7, and 9 subjects.
- Statistical analysis: Statistical differences between methods are assessed by sign-based resampling with 10 million samples and Holm correction across 28 comparisons.Seven metrics were assessed for each of four comparison methods.
- Model variants: The multi-branch model uses parallel, non-weight-sharing ResNets for separate modality combinations, while the single-branch model uses one ResNet.Two-branch and three-branch configurations were trained with corresponding modality combinations.
5. Results
The proposed method performed strongly on both ISBI and NRU evaluations, achieving a high official challenge score and leading several NRU metrics. Results also favored combining modalities, especially in the three-branch configuration.
- ISBI results: The method outperformed other methods in DSC and LFPR in the first ISBI experiment, while the highest LTPR came from another method.The experiment compared mean DSC, LTPR, and LFPR values.
- ISBI results: 92.12 overall evaluation score placed the method among the top-ranked published or reported solutions on the official ISBI test set.The score was reported by the ISBI challenge web service at submission time.
- NRU results: The method achieved the best NRU results for DSC, PPV, LFPR, VD, SD, and HD, while providing a good LTPR–LFPR trade-off.The comparison used mean values across the listed metrics.
- NRU results: Statistically significant performance differences were observed for most metrics and methods compared with the proposed model after Holm correction.The comparison is summarized by boxplots of seven evaluation metrics.
- Volume agreement: 0.9027 slope and 0.75 Pearson correlation coefficient indicated the strongest global agreement between estimated and ground-truth lesion volumes.The paper distinguishes volume agreement from lesion-overlap agreement measured by DSC.
- Modality analysis: The three-branch model using all modalities showed the best general performance among the modality configurations.Two-branch models including FLAIR generally outperformed single-modality single-branch models.
6. Discussion and Conclusions
The paper presents a multi-branch, multi-scale 2D CNN pipeline for multi-modal MS lesion segmentation and evaluates its design choices against alternative approaches. Results support separate modality branches, whole-brain slice processing, and multiscale fusion, while identifying runtime and lesion-type scope as remaining limitations.
- Method: Transfer learning initializes parallel ResNet50 downsampling branches with ImageNet weights before fine-tuning the complete network for MS lesion segmentation.The fusion and upsampling layers are randomly initialized from a Gaussian distribution.
- Modality processing: A separate branch for each MRI modality produced more accurate segmentations than single-branch modality stacking, with DSC=0.7649 versus 0.6844 and 0.6980.The single-branch comparisons correspond to the methods of Brosch et al. and Aslani et al., respectively.
- Modality processing: The multi-branch network achieved the best trade-off between lesion-wise true-positive and false-positive rates, despite some single-branch methods obtaining higher lesion-wise true-positive rates.The proposed method avoided the high lesion-wise false-positive rates associated with those single-branch results.
- Modality processing: FLAIR was the most important individual modality, with DSC>0.65, while using all modalities in the multi-branch network achieved top performance on most measures.The paper attributes FLAIR's usefulness to cerebrospinal-fluid signal suppression and stronger lesion-to-background contrast.
- Segmentation strategy: The whole-brain slice-based approach addresses patch-based methods' limited lesion spatial context and showed better performance on most measures, although another method had the highest DSC.The proposed method instead performed better on lesion-wise true-positive and false-positive rates and their trade-off.
- Limitations: The pipeline requires approximately 90 seconds per 3D image because whole-brain slicing, slice-level segmentation, and 3D reconstruction are sequential.The authors describe this as slightly slower than other CNN-based approaches on both ISBI and NRU datasets.
- Limitations: Cortical and subcortical lesion identification remains open, motivating future use of DIR sequences, tissue information, and multi-task segmentation.The proposed extensions target cortical, subcortical, and white-matter lesions alongside tissue classes.
Appendix A. Evaluation Protocols
Appendix A documents the training procedures used for the ISBI experiments and the nested cross-validation protocol used in the first NRU experiment.
- ISBI training: Tables A.1 and A.2 detail the training procedures for the first and second ISBI dataset experiments.
- NRU training: Table A.3 describes the nested 4-fold cross-validation training procedure for the first NRU dataset experiment.
Appendix B. Labels Aggregation
Appendix B evaluates alternative label-fusion methods in the first NRU experiment beyond majority voting.
- Label fusion: The first NRU experiment was repeated using averaging and STAPLE to aggregate outputs from three plane orientations and different models.These methods were tested as alternatives to the majority-vote framework.
Appendix C. Rater Evaluation on NRU Dataset
Appendix C evaluates the NRU model against ground truths supplied by individual raters rather than only the consensus mask.
- Rater evaluation: The first NRU experiment compared model performance using each expert's ground-truth mask to assess consistency across raters.The comparison was motivated by the high consensus between the two raters.