PaperlayerView exact revision on arXiv

Source-linked AI summary

3D Convolutional Neural Networks for Tumor Segmentation using Long-range 2D Context

Pawel Mlynarski, Hervé Delingette, Antonio Criminisi, Nicholas Ayache

arXiv:1807.08599v1cs.CVcs.LG

TL;DR

Manual tumor segmentation is time-consuming and variable, and CNN-based methods face computational and context-capture challenges. This paper combines long-range 2D features with 3D CNN processing, adds modality-specific subnetworks and voxelwise voting, and reports BRATS 2017 validation results. The 2D-3D approach outperformed the standard 3D model and achieved median Dice scores of 0.918 for whole tumor, 0.883 for tumor core, and 0.854 for contrast-enhancing core.

  • Problem

    Manual tumor segmentation is time-consuming and variable, while tumor intensity overlap and variation make accurate localization challenging.

    Source: §1 · pp. 2–3

  • Method

    The approach combines long-range 2D CNN features with 3D processing, uses modality-specific subnetworks, and merges model segmentations by voxelwise voting.

    Source: §1 · p. 4, §4 · pp. 25–26

  • Results

    The 2D-3D approach outperformed the standard 3D model and achieved median Dice scores of 0.918 (whole tumor), 0.883 (tumor core), and 0.854 (contrast-enhancing core).

    Source: §1 · p. 5

  • Takeaways & Limitations

    The method is described as adaptable to a variety of multiclass medical-image segmentation tasks.

    Source: §1 · p. 5

    The voxelwise merging strategy adds complexity, and users may prefer a single network architecture.

    Source: §4 · p. 27

Abstract

from arXiv · show

We present an efficient deep learning approach for the challenging task of tumor segmentation in multisequence MR images. In recent years, Convolutional Neural Networks (CNN) have achieved state-of-the-art performances in a large variety of recognition tasks in medical imaging. Because of the considerable computational cost of CNNs, large volumes such as MRI are typically processed by subvolumes, for instance slices (axial, coronal, sagittal) or small 3D patches. In this paper we introduce a CNN-based model which efficiently combines the advantages of the short-range 3D context and the long-range 2D context. To overcome the limitations of specific choices of neural network architectures, we also propose to merge outputs of several cascaded 2D-3D models by a voxelwise voting strategy. Furthermore, we propose a network architecture in which the different MR sequences are processed by separate subnetworks in order to be more robust to the problem of missing MR sequences. Finally, a simple and efficient algorithm for training large CNN models is introduced. We evaluate our method on the public benchmark of the BRATS 2017 challenge on the task of multiclass segmentation of malignant brain tumors. Our method achieves good performances and produces accurate segmentations with median Dice scores of 0.918 (whole tumor), 0.883 (tumor core) and 0.854 (enhancing core). Our approach can be naturally applied to various tasks involving segmentation of lesions or organs.

1. Introduction

Tumor segmentation is difficult because manual annotation is time-consuming and variable, while image intensities and tumor appearance complicate classification. The paper proposes CNN designs that combine long-range context with 3D processing, handle missing MR sequences, and merge model outputs; it reports BRATS 2017 validation results.

1. Introduction

  • Multisequence MRI depicts distinct tumor tissues, whose analysis supports assessment of tumor characteristics, including malignancy.The introduction notes that different tumor tissues may be treated with different therapies.Source: §1 · p. 2
  • Tumor variability and overlapping MR intensities make localization difficult, while manual segmentation is time-consuming and varies across experts.High-level contextual information is needed to identify tumor tissue.Source: §1 · pp. 2–3
  • Prior work includes atlas-based generative methods, voxelwise classifiers, and CNN segmentation approaches, including fully convolutional and 3D networks.Atlas methods may be affected by tumor-driven deformation, while tumor-growth models add complexity.Source: §1 · pp. 3–4
  • CNN-based segmentation faces high computational costs and difficulty capturing large 3D context; many methods also assume complete, correctly registered MR sequences.Those sequence conditions may not hold in practice.Source: §1 · p. 4
  • The proposed 2D-3D model feeds features from 2D CNNs into a 3D CNN to combine long-range context with 3D processing while addressing computational constraints.The introduction also proposes modality-specific subnetworks, voxelwise voting across models, and a stable training algorithm for large models.Source: §1 · pp. 4–5
  • 0.918 (whole tumor), 0.883 (tumor core), and 0.854 (contrast-enhancing core) were the median Dice scores reported after the 2D-3D approach outperformed the standard 3D model.The evaluation used the BRATS 2017 validation set and a public benchmark.Source: §1 · p. 5

2. Methods

The method combines long-range features from 2D slice networks with local 3D processing, then adds modality-specific learning, hierarchical voxelwise voting, and an efficient training scheme. These components address context, missing MR sequences, model variation, and training stability.

2.1. Spatial context and 3D models

  • 2D features from axial, coronal, and sagittal networks are concatenated with 3D patches, extending voxel classification beyond the local 3D receptive field to orthogonal planes.The 3D network processes 70x70x70 patches with a 45x45x45 receptive field, while the 2D features encode larger spatial context.Source: Figure 2 · p. 5, §2.1 · pp. 6, 8, 9
  • Batch-dependent voxel weights allocate target loss proportions to classes, addressing severe class imbalance in tumor segmentation.The target weights are chosen empirically; the non-tumor class receives a lower weight than under standard cross-entropy to limit oversegmentation.Source: §2.1 · pp. 10–11

2.2. 2D deep learning model

  • The 2D architecture combines modality-specific subnetworks with a joint all-sequence branch, allowing modality-specific training on images with missing sequences.Auxiliary classification layers train the subnetworks to extract sequence-specific information and are ignored at test time.Source: §2.2 · pp. 11–12

2.3. Ensembles of neural networks

  • Voxelwise voting merges segmentations through successive thresholds for whole tumor, tumor core, and enhancing core, with thresholds controlling sensitivity-specificity trade-offs.The experiments use six models; lower thresholds favor sensitivity and higher thresholds favor specificity.Source: §2.3 · pp. 14–15

2.4. Training scheme

  • Training aggregates gradients across multiple batches, normalizes their magnitude, and adapts the learning rate to observed loss convergence.The authors report satisfactory convergence with this scheme.Source: §2.4 · pp. 16–17

3. Results

On BRATS 2017, the method ranked among the top three for tumor core and enhancing core, while its whole-tumor mean Dice nearly matched the leading score. Experiments also found benefits from adding 2D features to 3D models, modality-specific pretraining, and voxelwise ensembling.

3.1. Data

  • BRATS 2017 supplied multisequence scans from 19 imaging centers, with four MR sequences per patient; its training set had annotations, while the 46-patient validation set did not.The benchmark images had been skull-stripped, registered to a common template, and interpolated to 1 mm3 resolution.Source: §3.1 · pp. 17–18

3.2. Test setting

  • Experiments compared two 2D architectures and three 2D-3D variants, including local testing on 50 held-out training patients and public evaluation on the validation set.The local comparison trained on the remaining 235 patients; the 2D-3D models used 2D features at different network stages or in a separate stream.Source: §3.2 · pp. 18–20

3.3. Results

  • Adding 2D-network features improved 3D-model performance over both standard 3D counterparts and the 2D networks that produced those features.The qualitative comparison attributes the advantage to richer information from distant voxels, particularly for tumor core and enhancing core; the axial 2D model also scored well without postprocessing.Source: §3.3 · pp. 20, 21, 23, Table 3 · p. 21
  • With MR sequences missing for some training cases, pretraining modality-specific subnetworks improved all three tumor-subregion scores, with statistically significant gains.The experiment made all four sequences available for only 20% of patients, allowing each subnetwork to train on cases where its sequence was present.Source: §3.2 · p. 19, Table 2 · p. 20, §3.3 · pp. 21–22
  • Qualitative outputs showed plane-parallel discontinuities and skull-stripping-border false positives in 2D segmentations, while standard 3D outputs were more spatially consistent but had limited distant-voxel information.The added 2D features supplied distant-context information, with visible differences for tumor core and enhancing core.Source: §3.3 · pp. 22–23
  • Voxelwise ensembling further improved accuracy for every tumor subregion, significantly for whole tumor and tumor core.The ensemble combined six trained 2D-3D models.Source: §3.2 · p. 18, §3.3 · p. 23
  • Among 55 teams evaluating all validation-set patients, our method achieved top-three performance for core and enhancing-core tumor subregions and a whole-tumor mean Dice of 0.9, almost equal to the best-scoring method.Source: §3.3 · pp. 23–24, Table 5 · p. 23
  • Median Dice scores were 0.918 for whole tumor, 0.883 for tumor core, and 0.854 for enhancing core.Whole-tumor scores were generally stable, while subregion scores varied more and medians exceeded means because of outliers.Source: Table 6 · p. 23, §3.3 · p. 25

4. Discussion and conclusion

The paper presents a multiclass tumor-segmentation system combining long-range context, modality-specific subnetworks, and voxelwise voting. Its discussion describes the benefits and implementation trade-offs of these components and identifies weakly or semi-supervised learning as future work.

4. Discussion and conclusion

  • The 2D-3D model uses long-range features from three orthogonal directions as additional input to a 3D CNN, improving performance over both standard 3D and feature-producing 2D models.The authors report that importing a few features does not considerably increase computation or memory load, though it adds reading operations.Source: §4 · p. 25
  • The 2D-3D approach is more complex than a pure 3D approach because it requires two-step processing, although extracted features can be read as additional input channels.The additional implementation element is extracting and saving outputs from trained 2D networks.Source: §4 · p. 26
  • The 2D-3D approach increases receptive-field size while remaining computationally efficient, making it particularly relevant to very large 3D scans.Compared with pure 3D processing, it requires two stages: 2D feature extraction followed by 3D processing.Source: §4 · p. 26
  • Modality-specific subnetworks improved segmentation when training databases lacked MR sequences and offer a way to train on images with a given sequence before training the main network on complete images.The paper also notes that 2D networks are easier to apply clinically when the number of acquired slices varies.Source: §4 · p. 26
  • Voxelwise voting merges multiclass outputs from several models, takes label semantics into account, and lets users control sensitivity to tumor subregions.The authors note that this merging strategy adds complexity and that users may prefer a single network architecture.Source: §4 · pp. 26–27
  • The methodological contributions can be included separately or jointly in CNN-based systems for specific segmentation problems.The contributions introduce normalized hyperparameters, including loss weights and model-merging thresholds, that control the importance of different elements and tumor-subclass sensitivity.Source: §4 · p. 27
  • Weakly or semi-supervised learning is proposed as future work because accurate manual tumor segmentations require substantial labor and expertise.Source: §4 · p. 27

Related papers

Loading 1807.08599v1…