Efficient Multi-Scale 3D CNN with Fully Connected CRF for Accurate Brain Lesion Segmentation

Konstantinos Kamnitsas; Christian Ledig; Virginia F. J. Newcombe; Joanna P. Simpson; Andrew D. Kane; David K. Menon; Daniel Rueckert; Ben Glocker

Source-linked AI summary

Efficient Multi-Scale 3D CNN with Fully Connected CRF for Accurate Brain Lesion Segmentation

Konstantinos Kamnitsas, Christian Ledig, Virginia F. J. Newcombe, Joanna P. Simpson, Andrew D. Kane, David K. Menon, Daniel Rueckert, Ben Glocker

arXiv:1603.05959v3Mar 18, 2016cs.CVcs.AI

TL;DR

Accurate automatic brain-lesion segmentation remains difficult because lesions vary widely and can resemble healthy tissue, while manual assessment is labor-intensive and inconsistent. The paper introduces an efficient multi-scale 3D CNN with fully connected CRF post-processing and reports state-of-the-art or higher performance across traumatic brain injury, tumor, and stroke segmentation tasks.

Problem
Brain lesions vary in shape, size, location, and appearance, making accurate automatic segmentation difficult and limiting consistent quantitative assessment.
Method
The paper combines a dual-pathway, 11-layer 3D CNN with efficient dense training, multi-scale processing, and fully connected CRF post-processing.
Results
The system surpasses or matches state-of-the-art performance across traumatic brain injury, brain tumor, and stroke lesion segmentation tasks, including both BRATS 2015 and ISLES 2015.
Takeaways & Limitations
The generic pipeline supports application across different lesion segmentation tasks without major adaptations and is computationally efficient.
Takeaways & Limitations
Applying the pipeline to new tasks requires laborious CRF reconfiguration, and finding optimal parameters can be challenging.

Abstract

from arXiv · show

We propose a dual pathway, 11-layers deep, three-dimensional Convolutional Neural Network for the challenging task of brain lesion segmentation. The devised architecture is the result of an in-depth analysis of the limitations of current networks proposed for similar applications. To overcome the computational burden of processing 3D medical scans, we have devised an efficient and effective dense training scheme which joins the processing of adjacent image patches into one pass through the network while automatically adapting to the inherent class imbalance present in the data. Further, we analyze the development of deeper, thus more discriminative 3D CNNs. In order to incorporate both local and larger contextual information, we employ a dual pathway architecture that processes the input images at multiple scales simultaneously. For post-processing of the network's soft segmentation, we use a 3D fully connected Conditional Random Field which effectively removes false positives. Our pipeline is extensively evaluated on three challenging tasks of lesion segmentation in multi-channel MRI patient data with traumatic brain injuries, brain tumors, and ischemic stroke. We improve on the state-of-the-art for all three applications, with top ranking performance on the public benchmarks BRATS 2015 and ISLES 2015. Our method is computationally efficient, which allows its adoption in a variety of research and clinical settings. The source code of our implementation is made publicly available.

1. Introduction

Accurate lesion segmentation in multimodal 3D brain MRI is clinically valuable but difficult because lesions vary widely and manual delineation is impractical. The paper proposes an efficient, multi-scale 3D CNN with dense training and a 3D fully connected CRF to address these challenges.

Motivation: Lesion segmentation supports treatment planning, disease monitoring, outcome prediction, and quantitative analysis of neuropathologies.Quantitative imaging can provide clues about disease characteristics and effects on anatomical structures.
Challenges: Accurate segmentation is difficult because lesions vary in location, size, shape, frequency, and appearance across multimodal 3D images.Manual delineation is tedious, expensive, time-consuming, impractical for large studies, and introduces inter-observer variability.
Related work: 2D CNN approaches process 3D scans slice by slice, while fully 3D CNNs impose substantial parameter, memory, computational, and inference costs.Dense inference can greatly decrease inference times for 3D CNNs.
Contributions: The proposed approach uses an 11-layers deep, multi-scale 3D CNN with an efficient hybrid dense-training scheme that adapts to class imbalance.The paper also analyzes deeper, more discriminative, computationally efficient 3D CNNs using small kernels.
Contributions: A fully connected 3D CRF post-processes the CNN’s soft segmentation maps, handling arbitrarily large neighborhoods while preserving fast inference.The authors state that this is the first use of a fully connected CRF on medical data.

2. Method

The method combines a 3D CNN for soft voxel-wise segmentation with a fully connected 3D CRF for final hard labels. Its main CNN innovations are dense segment-based training, multi-scale dual pathways, and efficient deeper architectures.

Method overview: The pipeline uses a 3D CNN to produce soft segmentation maps, followed by a fully connected 3D CRF that regularizes outputs into hard labels.The CNN component contains the primary methodological contributions.
Dense training: Dense segment-based training feeds image segments larger than the CNN receptive field, producing posterior probabilities for multiple voxels in one pass.This exploits dense inference to address the memory and computation limits of patch-based 3D CNN training.
Dense training: Training segments are sampled with 50% probability centered on foreground or background voxels, automatically balancing class distributions.The sampling strategy targets class imbalance, which directly affects segmentation accuracy.
Efficient CNN design: Using 3^3 rather than 5^3 kernels reduces element-wise multiplications and trainable parameters by approximately 4.6×, enabling more efficient deeper variants.The smaller kernels are described as implicitly regularizing the deeper network variants.
Multi-scale architecture: A dual-pathway 3D CNN processes down-sampled and full-resolution images simultaneously, learning broader contextual features alongside detailed local appearance.The pathways are decoupled, allowing arbitrarily large context while avoiding large 3D-volume memory requirements.
CRF post-processing: The 3D fully connected CRF applies unary CNN likelihoods and pairwise interactions between all voxel pairs using Gaussian kernels over coordinates and multi-modal intensities.The implementation extends the original CRF formulation to 3D multi-modal scans.

3. Analysis of Network Architecture

Experiments on the TBI dataset analyze dense training, network depth, multi-scale context, and 3D processing choices for the proposed architecture. Dense training improves segmentation while deeper and multi-scale designs address limitations of shallower or single-scale models, and 3D processing outperforms the 2D variant.

Dense training: Dense training with larger lesion-background segments improves the balance between sensitivity and specificity, yielding better DSC than Peq.Performance increases for S-19 over Peq, while larger segments quickly balance Peq’s sensitivity and Puni’s specificity; performance levels off across a broad range, so remaining experiments use segments of size 25^3.
Network depth: The 9-layer Deep model fails by predicting only background, whereas revised initialization and Batch Normalization produce enhanced deep variants.The Deep model replaces each 5^3 convolution with two 3^3 layers, but training fails because signal propagation and weight-variance scaling are not preserved.
Multi-scale architecture: DeepMedic extends Deep+ with a second identical pathway, two feature-combination layers, and inputs down-sampled by a factor of three to capture multi-scale context.The resulting architecture has 11 layers and is evaluated against single-scale Deep+ and a wider single-scale model with comparable capacity.
Multi-scale architecture: DeepMedic outperforms the single-scale Deep+ model, and comparison with a similarly sized wide single-scale network attributes the gain to additional context rather than capacity.The wide single-scale model appears to suffer from over-fitting, while DeepMedic yields the best performance by capturing greater context.
3D versus 2D processing: 61.5% average DSC is achieved by the 2D variant on the validation fold, compared with 66.6% DSC for the 3D DeepMedic version.The decline indicates that processing 3D context remains important even when most acquired sequences have low resolution along one axis.

4. Evaluation on Clinical Data

The DeepMedic CNN, optionally paired with a fully connected CRF, is evaluated on three clinical lesion-segmentation tasks involving traumatic brain injuries, brain tumors, and ischemic stroke, with quantitative comparisons against the state of the art.

The proposed system comprises the DeepMedic CNN architecture, optionally coupled with a fully connected CRF.
Evaluation covers three challenging clinical lesion-segmentation tasks involving traumatic brain injuries, brain tumors, and ischemic stroke.
Each task is assessed quantitatively and compared with state-of-the-art methods.

4.1. Traumatic Brain Injuries

For traumatic brain injury lesion segmentation, the study evaluates a dual-pathway, 11-layer CNN with augmentation and fully connected CRF post-processing. The CNN outperforms a Random Forest baseline, while ensembling improves robustness across challenging lesion sizes.

Data: The TBI dataset comprises 66 moderate-to-severe patients imaged within the first week after injury.Imaging was performed using a 3-Tesla Siemens Magnetom TIM Trio at Addenbrooke’s Hospital.
Network configuration and training: Training uses sagittal reflection augmentation and channel-wise Gaussian intensity shifts to reduce sensitivity to absolute MRI intensities.For each channel, the intensity shift is sampled from N(0, 0.1) and scaled by the brain-mask intensity standard deviation.
Results: The CNN significantly outperforms a context-sensitive Random Forest baseline, although overall DSC values remain relatively low because TBI segmentation is difficult.The baseline uses 50 trees with maximum depth 30 and approximately balanced lesion/background sampling.
Results: A three-network ensemble averages predictions to reduce errors arising from different training-session local minima.The ensemble combines three similar networks because stochastic training produces differing errors across converged models.
Results: The network segments very small lesions, but performance is less robust because minor errors strongly affect DSC in these cases.Predicted lesion volumes also show no apparent bias toward lesion or background classes, including in difficult cases.

4.2. Brain Tumor Segmentation

The method is evaluated on BRATS 2015 brain-tumor MRI data across whole-tumor, core, and enhancing-tumor segmentation tasks. A multi-class DeepMedic variant, trained with data augmentation and tumor-focused sampling, is refined with a whole-tumor CRF that improves boundaries and removes isolated false positives, particularly on test data.

Dataset: BRATS 2015 provides 274 training cases and 110 test cases spanning high- and low-grade gliomas, with four MRI sequences and four tumor tissue classes.Evaluation distinguishes whole tumor, core, and enhancing tumor; volumes are skull-stripped, registered, and resampled to 1mm^3 resolution.
Network configuration and training: The network extends DeepMedic to five output maps for four tumor classes plus background and augments training with sagittal reflections.Unlike the TBI experiments, intensity perturbation and convolutional dropout are not used, and segments are sampled equally around whole tumor and healthy tissue.
CRF configuration: The CRF merges the four tumor probability maps into a whole-tumor map, refining tumor–background boundaries and removing isolated false positives.A single global multi-class CRF parameterization was considered difficult to make consistently beneficial across all classes.
Quantitative results: On BRATS training data, the CRF and three-network ensemble improve on DeepMedic, although the gains are fairly small because DeepMedic already performs strongly.The study also reports favorable performance relative to previous state-of-the-art submissions, including Bakas et al. and Pereira et al.
Quantitative results: On the 110-case BRATS test set, performance decreases relative to training data, but CRF gains are larger, consistent with stronger boundary and false-positive correction under dataset variation.The reported core-class DSC decreases for the CRF and ensemble were not statistically significant.

4.3. Ischemic Stroke Lesion Segmentation

On the ISLES-SISS 2015 ischemic stroke task, DeepMedic used a computationally reduced low-resolution pathway and a three-network ensemble coupled with a fully connected CRF. The submission ranked first among 14 testing-stage entries, while the authors attributed the advantage to 3D context, multi-scale field of view, and deeper-network representational power.

DeepMedic achieved the best results among all participants in the 2015 ISLES challenge for sub-acute ischemic stroke lesions.
The low-resolution pathway used relatively fewer feature maps, reducing computational cost without significantly influencing accuracy on generally small SISS lesions.
For challenge testing, an ensemble of three networks averaged predictions and was coupled with a fully connected CRF.
The submission ranked first among 14 testing-stage submissions, indicating superior performance on this challenging task.
The authors linked the method’s advantage to 3D context, DeepMedic’s large multi-scale field of view, and the representational power of deeper networks.

4.4. Implementation Details

The CNN is implemented in Theano with cuDNN support, requiring approximately one day per training session on an NVIDIA GTX Titan X while also supporting 3GB GPUs. Its fully connected CRF processes five-channel brain scans in under three minutes on a CPU, and dense whole-volume inference remains possible.

CNN implementation: The CNN uses Theano and cuDNN v5.0, with each training session requiring approximately one day on an NVIDIA GTX Titan X GPU.Models can also be trained on GPUs with only 3GB of memory.
Dense processing: Dense training on whole volumes is unsupported for the processed databases at this network size, but dense inference on a whole volume remains possible.The volume dimensions prevent dense whole-volume training, whereas whole-volume inference requires only a for…
CRF implementation: The 3D fully connected CRF extends the original Kr¨ahenb¨uhl and Koltun (2011) source code and processes a five-channel brain scan in under three minutes on a CPU.A GPU implementation could provide further speed-up, but was not considered necessary for this work.

5. Discussion and Conclusion

DeepMedic combines an efficient, adaptive training scheme with deeper multi-scale 3D CNN processing and fully connected CRF post-processing for lesion segmentation. The discussion highlights its broad applicability and performance, while identifying CRF configuration, interpretability, and cross-center data heterogeneity as challenges and future directions.

Contributions: DeepMedic surpasses state-of-the-art lesion-segmentation performance with computationally efficient training that partially alleviates class imbalance.The architecture uses smaller 3D kernels to support a deeper, more discriminative network without increasing computational cost or trainable parameters.
Multi-scale representations: Parallel convolutional pathways efficiently process large image context at multiple scales, combining localization cues with detailed patterns for lesion segmentation.The low-resolution pathway is assumed to localize large pathologies and difficult brain regions, reserving capacity for smaller lesions, fine structures, and ambiguous areas.
Applicability and performance: The generic system transfers to different lesion-segmentation tasks without major adaptations and achieved the highest reported accuracy on a severe-TBI patient cohort.The authors compare this performance with Rao et al. (2014), whose pipeline focused only on contusion segmentation, whereas DeepMedic was capable of segmenting every lesion type discussed.
Limitations and future improvements: CRF post-processing significantly improves performance across investigated tasks, but applying the pipeline to new tasks requires laborious reconfiguration and task-specific parameter selection.Because tumor substructures vary substantially in appearance, a global parameter set was difficult to identify, so the CRF was applied in a binary fashion.
Limitations and future work: Inference interpretability, non-Bayesian output probabilities, and performance drops across clinical centers motivate future work on Bayesian modeling and generative augmentation for acquisition heterogeneity.The authors attribute cross-dataset differences to scanner types and acquisition protocols and propose learning a generative model of data acquisition for augmentation.
Reproducibility: The entire system’s source code is publicly available to facilitate further research and provide a baseline for future evaluations.

Appendix A. Additional Details on Multi-Scale Processing

The proposed multi-scale architecture uses simple repetition to up-sample low-resolution features, which proved sufficient because subsequent hidden layers learned to combine the multi-scale representations. More elaborate learned upsampling schemes may benefit architectures with strides greater than one.

Multi-Scale Processing: Simple repetition up-sampling was sufficient for the proposed unary-stride architecture because subsequent hidden layers learned to combine multi-scale features.The method integrates parallel multi-scale pathways and repeats low-resolution features before later hidden layers combine them.

Appendix B. Additional Details on Network Configurations

The appendix specifies training details for the fully 3D CNNs and representative 2D configurations, including optimization, regularization, memory matching, and sampling calibration. Wider or deeper 2D variants did not improve performance, while uncalibrated sampling caused underperformance.

3D Networks: 3D CNNs use PReLU activations, RMSProp with Nesterov momentum m = 0.6, L1 = 10^-6 and L2 = 10^-4 regularization, and dense-training batches of 10.These settings apply to models discussed outside Sec. 3.5.
2D Networks: 2D configurations matched DeepMedic’s 3D memory requirements by adjusting width, depth, and batch size; wider or deeper variants did not improve performance.The appendix suggests the available filters and deep multi-scale field of view may already suffice for limited 2D information.
Sampling: Manual sampling calibration was required to achieve class balance comparable to image-segment-trained models; otherwise, the model underperformed.The note refers specifically to the sampling procedure used for the configurations.

Appendix C. Distribution of Tumor Classes Captured in Training

The proposed training scheme captures a tumor-centered and healthy-tissue-centered training distribution in which foreground-class proportions are closely preserved while healthy-tissue imbalance is alleviated.

Sampling strategy: The scheme extracts segments of size 253 centered on tumor and healthy tissue with equal probability.This sampling strategy produces the captured training distribution reported in Table C.1.
Distribution captured: Foreground-class relative distributions are closely preserved, while imbalance relative to healthy tissue is automatically alleviated.The comparison is between the real BRATS 2015 training distribution and the distribution captured by the proposed scheme.