PaperlayerView exact revision on arXiv

Source-linked AI summary

A deep learning model integrating FCNNs and CRFs for brain tumor segmentation

Xiaomei Zhao, Yihong Wu, Guidong Song, Zhenye Li, Yazhuo Zhang, Yong Fan

arXiv:1702.04528v3cs.CV

TL;DR

Brain tumor segmentation is important for diagnosis and treatment but remains difficult because gliomas have variable appearances and MRI data are challenging to interpret. The paper integrates FCNNs with CRF-RNNs, trains multi-view models, and fuses their predictions. It reports faster slice-by-slice segmentation, competitive performance using three MRI modalities, and first place on a multi-temporal evaluation.

  • Problem

    Accurate brain tumor segmentation is important for cancer diagnosis, treatment planning, and outcome evaluation, but glioma segmentation is challenging because tumors vary in appearance, location, shape, and size.

  • Method

    The method integrates FCNNs and CRF-RNNs in one network, trains them using patches and slices across three anatomical views, and combines predictions by voting.

  • Results

    The method ranked first on its multi-temporal evaluation and achieved competitive segmentation performance using Flair, T1c, and T2 scans.

  • Takeaways & Limitations

    The framework provides slice-by-slice brain tumor segmentation and supports competitive performance with three MRI modalities rather than four.

  • Takeaways & Limitations

    Because the method uses 2D CNNs, it does not take full advantage of the MRI data’s 3D information.

Abstract

from arXiv · show

Accurate and reliable brain tumor segmentation is a critical component in cancer diagnosis, treatment planning, and treatment outcome evaluation. Build upon successful deep learning techniques, a novel brain tumor segmentation method is developed by integrating fully convolutional neural networks (FCNNs) and Conditional Random Fields (CRFs) in a unified framework to obtain segmentation results with appearance and spatial consistency. We train a deep learning based segmentation model using 2D image patches and image slices in following steps: 1) training FCNNs using image patches; 2) training CRFs as Recurrent Neural Networks (CRF-RNN) using image slices with parameters of FCNNs fixed; and 3) fine-tuning the FCNNs and the CRF-RNN using image slices. Particularly, we train 3 segmentation models using 2D image patches and slices obtained in axial, coronal and sagittal views respectively, and combine them to segment brain tumors using a voting based fusion strategy. Our method could segment brain images slice-by-slice, much faster than those based on image patches. We have evaluated our method based on imaging data provided by the Multimodal Brain Tumor Image Segmentation Challenge (BRATS) 2013, BRATS 2015 and BRATS 2016. The experimental results have demonstrated that our method could build a segmentation model with Flair, T1c, and T2 scans and achieve competitive performance as those built with Flair, T1, T1c, and T2 scans.

1. Introduction

Brain tumor segmentation is clinically important but difficult because gliomas vary in appearance, location, shape, and size, while MRI intensity inhomogeneity complicates analysis. The paper addresses limitations of patch-based CNN methods by integrating FCNNs with CRF-RNNs for spatially and appearance-consistent segmentation.

  • Accurate brain tumor segmentation supports cancer diagnosis, treatment planning, and treatment outcome evaluation, while manual segmentation is laborious.
  • Glioma segmentation is challenging because tumors can resemble gliosis or stroke, occur throughout the brain, vary in shape and size, and invade surrounding tissue.
  • Existing methods include generative approaches using prior information and discriminative approaches that classify voxels from image features.
  • Deep learning methods commonly use 2D or 3D CNNs trained on image patches, but patch-based predictions typically assume voxel-label independence and lose spatial relationships.
  • CRFs can model appearance and spatial consistency and can be integrated with CNNs either as post-processing or as trainable neural-network components.
  • The proposed framework integrates FCNNs and CRF-RNNs, trains models across axial, coronal, and sagittal views, and fuses their outputs by voting.
  • The method segments images slice-by-slice faster than patch-based methods and achieves competitive performance using Flair, T1c, and T2 scans instead of four modalities.

2.1 Imaging data

The study evaluates brain tumor segmentation on BRATS 2013, 2015, and 2016 imaging datasets containing multimodal MRI and manually produced or competition-derived annotations.

  • BRATS 2013 provides clinical MRI data from 65 glioma patients, including 14 LGG and 51 HGG cases scanned at four centers.
  • Each BRATS 2013 patient has Flair, T1, T1c, and T2 scans, which were co-registered, resampled at 1 mm isotropic resolution, and skull stripped.
  • The BRATS 2015 dataset includes 220 HGG and 54 LGG training cases and 110 testing cases with unknown grades.
  • Testing cases provide imaging data without released ground truth for blind evaluation, while training annotations include manual labels and fused-method annotations in parts of BRATS 2015.
  • BRATS 2016 shares the BRATS 2015 training set and contains 191 testing cases with unknown grades.

2.2 Brain tumor segmentation methods based on FCNNs trained using image patches

Patch-based CNN segmentation converts image segmentation into patch classification and can provide many controllable training samples, but it is computationally inefficient at inference and loses relationships among patches. FCNNs instead segment complete slices directly, while CRF-RNN integration addresses consistency limitations.

  • Patch-based methods represent segmentation as classifying local image patches according to the label of each patch’s central voxel.
  • Training and testing extract many patches, classify them individually, and assemble their classifications into the final image segmentation.
  • FCNNs can segment a testing image slice by slice, providing improved computational efficiency despite patch-based training.
  • Patch sampling allows the number and location of training patches for each class to be controlled, helping avoid training-sample imbalance.
  • A limitation of patch-based segmentation is that relationships among image patches are typically lost.
  • Integrating CRF-RNNs with FCNNs is intended to overcome the loss of patch relationships in tumor segmentation.

2.3 The proposed brain tumor segmentation method

The method combines intensity normalization, FCNN–CRF-RNN segmentation, three-view fusion, and post-processing into a four-step brain tumor segmentation pipeline.

  • The pipeline comprises pre-processing, slice segmentation with integrated FCNNs and CRF-RNN, three-view fusion, and post-processing.
  • 2.3.2 A deep learning model integrating FCNNs and CRFs: The integrated network combines FCNNs, which predict tissue-label probabilities, with CRF-RNN, which produces a globally optimized slice segmentation.CRF-RNN implements fully connected CRFs as recurrent neural networks and uses FCNN predictions as inputs.
  • (1) FCNNs: FCNN training uses randomly extracted patches from axial, coronal, or sagittal slices with balanced class sampling, while testing proceeds slice by slice.Pooling captures large-scale image information with relatively few parameters, and unit strides support slice-wise testing.
  • (2) CRF-RNN: CRF-RNN estimates label probabilities by recurrently implementing mean-field inference over fully connected conditional random fields.Its unary term comes from FCNN outputs, while pairwise potentials encode relationships between pixels using intensity, spatial, and feature information.

3. Experiments

The experiments used BRATS 2013, 2015, and 2016 imaging data to evaluate tumor segmentation across defined regions and metrics.

  • Experiments were conducted on imaging data from the BRATS 2013, 2015, and 2016 datasets.
  • The models were implemented with Caffe and run on a shared server containing Tesla K80 GPUs and Intel E5-2620 CPUs, using one GPU and CPU concurrently.
  • For BRATS 2013 testing data, the trained model was evaluated through the BRATS evaluation website because ground-truth test segmentations were unavailable.
  • Segmentation performance was evaluated for complete tumor, core, and enhancing regions using Dice, PPV, and Sensitivity.Complete tumor comprises necrosis, edema, non-enhancing core, and enhancing core; the core region excludes edema.

3.1. Experiments on BRATS 2013 dataset

BRATS 2013 experiments examined CRFs, post-processing, training choices, preprocessing, scan inputs, and multi-view fusion. CRFs improved accuracy and consistency, while post-processing reduced false positives but could lower sensitivity.

  • Additional evaluations: The experiments compared models using four scans—Flair, T1, T1c, and T2—with models using Flair, T1c, and T2, and also assessed preprocessing and three-view fusion.
  • CRFs and post-processing: CRFs improved segmentation accuracy, while post-processing also improved results across the evaluated BRATS 2013 datasets.
  • CRFs and post-processing: FCNN+CRF and FCNN+post-process improved Dice and PPV for complete tumor, core, and enhancing regions, but reduced Sensitivity in several regions.CRFs improved enhancing-region Sensitivity, whereas the largest sensitivity degradation occurred for the complete tumor region after post-processing.
  • Qualitative results: Representative slices showed that FCNN+CRF improved spatial and appearance consistency, while additional post-processing reduced false positives.
  • Post-processing analysis: Among the six post-processing steps, step 3 played the most important role in tumor segmentation, although all steps might contribute.

3.1.3. Evaluating the impact of image patch size

Larger training patches and more training patches improved FCNN performance. CRF-RNN and post-processing reduced performance differences associated with patch size and patch count, although one inference remained for future confirmation.

  • Larger training patches provided more information and improved FCNN performance.The evaluated patch configurations ranged from 21*21*3 and 41*41*3 to 33*33*3 and 65*65*3.
  • CRF-RNN reduced performance differences caused by patch size by optimizing segmentation using information from the whole image slice.
  • Post-processing further reduced the performance differences associated with training patch size.
  • CRFs and post-processing narrowed performance differences caused by the number of training patches and possibly other training choices.The possibility concerning other training tricks was identified as an inference requiring future confirmation.

3.1.5. Performance Comparison between segmentation models built upon 4 and 3 imaging modalities

Models using three scans achieved similar performance to models using four scans, while robust-deviation normalization slightly improved segmentation. Multi-view fusion typically helped before post-processing, but its added benefit became insignificant afterward.

  • Imaging modalities: Models built with Flair, T1c, and T2 achieved competitive performance compared with models using Flair, T1, T1c, and T2.
  • Preprocessing: Robust-deviation intensity normalization slightly improved segmentation performance relative to standard-deviation normalization.
  • Multi-view fusion: Fusing segmentation results from axial, coronal, and sagittal views typically improved performance on both Challenge and Leaderboard datasets before post-processing.
  • Multi-view fusion: After post-processing, the improvement from multi-view fusion became insignificant.
  • Comparison and runtime: The method ranked second on the BRATS 2013 Challenge dataset and first on the Leaderboard dataset, with one view taking 2–4 minutes per subject.

3.2. Segmentation performance on the BRATS 2015

On BRATS 2015, the method was trained using HGG cases and evaluated on 110 testing cases, with multi-view fusion and larger training data associated with improved segmentation accuracy.

  • The BRATS 2015 training dataset contained 54 LGG and 220 HGG cases, while the testing dataset contained 110 cases.
  • Training: The models were trained using HGG cases, with 1000*5 patches extracted from each of 220 HGG cases for FCNN training.
  • Training: The CRF-RNN was initialized from the model trained on BRATS 2013, and the whole network was fine-tuned using BRATS 2013 training slices.
  • Evaluation: Evaluation on BRATS 2015 included results from models trained on both the BRATS 2013 and BRATS 2015 training datasets.
  • Results: Multi-view fusion improved segmentation accuracy, while the results indicated that a larger training dataset might improve segmentation.
  • Comparison: Direct comparison with methods participating in BRATS 2015 was not possible because the available testing cases differed from the earlier evaluation.

3.3. Segmentation performance on the BRATS 2016

On BRATS 2016, the method reused BRATS 2015 models but addressed partially pre-processed test images with manual tumor bounding boxes; it ranked first in multi-temporal evaluation.

  • Method: The method reused segmentation models trained on the BRATS 2015 training dataset because BRATS 2016 shared that training dataset.
  • Data and preprocessing: The BRATS 2016 testing dataset included unprocessed or partially pre-processed images, unlike the pre-processed BRATS 2015 training data.
  • Post-processing: Manual rectangular bounding boxes were placed around tumors to reduce false positives caused by incomplete pre-processing.
  • Results: Among 19 participating teams, the method ranked first on the multi-temporal evaluation.
  • Illustration: Figure 9 shows a partial skull-stripping case using Flair, T1c, and T2 images from left to right.
  • Results: Table 11 reports the method's ranking details across different BRATS 2016 evaluation items, including ties.

4. Discussions and Conclusion

The study integrates FCNNs and CRF-RNNs in a unified 2D framework, using multi-view fusion and preprocessing/post-processing strategies for brain tumor segmentation. It reports competitive performance across BRATS datasets, faster slice-by-slice segmentation, and limitations arising from incomplete use of 3D MRI information.

  • Method: The proposed network integrates FCNNs and CRF-RNNs to produce segmentation results with appearance and spatial consistency.CRF-RNN implements CRFs within one trainable deep network rather than as a separate FCNN post-processing step.
  • Method: Training uses patches for FCNNs, slices for CRF-RNN training with fixed FCNN parameters, and slice-based fine-tuning of the whole network.Three models are trained from axial, coronal, and sagittal views and combined through voting-based fusion.
  • Results: FCNN–CRF integration improved robustness to image patch size and training-patch count, while CRFs improved segmentation accuracy and consistency.The integrated approach is designed for slice-by-slice segmentation and is faster than patch-based segmentation methods.
  • Preprocessing and post-processing: The method combines N4ITK intensity normalization, small-region removal, and threshold-based label correction to improve tumor segmentation.The normalization is intended to make intensity values comparable across MRI scans.
  • Results: The method achieved promising performance on BRATS 2013 and BRATS 2015 testing datasets and ranked first in BRATS 2016 multi-temporal evaluation.It also achieved competitive performance using Flair, T1c, and T2 scans instead of four modalities.
  • Limitations and future work: Because the model is based on 2D CNNs, it cannot fully exploit the 3D information in MRI data.The authors report that 3D CRF post-processing improves performance and identify a fully 3D network as ongoing work.
Loading 1702.04528v3…