Source-linked AI summary
Vox2Vox: 3D-GAN for Brain Tumour Segmentation
Marco Domenico Cirillo, David Abramian, Anders Eklund
TL;DR
Glioma borders are difficult to delineate because the tumours infiltrate brain tissue and involve surrounding edema. Vox2Vox applies a 3D Pix2Pix GAN to brain MRI, and its ensemble reports median testing Dice scores of 91.42%, 90.59%, and 83.65% for whole tumour, tumour core, and enhancing tumour.
Problem
Gliomas' diffuse, infiltrative growth and peritumoural edema make tumour borders arduous to define visually, including during analysis and surgery.
Source: §1 · p. 1Method
Vox2Vox adapts paired-image Pix2Pix into a 3D GAN that segments gliomas from multi-modal MRI.
Source: §1.1 · p. 2Results
Testing-set median Dice scores were 91.42% for whole tumour, 90.59% for tumour core, and 83.65% for enhancing tumour after ensembling.
Source: §4 · p. 9Takeaways & Limitations
The authors propose that Vox2Vox could combine with a 3D noise-to-image GAN to generate realistic MR volumes paired with segmentation targets.
Source: §4 · p. 9The generator uses one convolution per encoder and decoder block to reduce GPU memory; whether two convolutions improve segmentation remains future work.
Source: §2.3 · p. 4
Abstract
from arXiv · showhide
Gliomas are the most common primary brain malignancies, with different degrees of aggressiveness, variable prognosis and various heterogeneous histological sub-regions, i.e., peritumoral edema, necrotic core, enhancing and non-enhancing tumour core. Although brain tumours can easily be detected using multi-modal MRI, accurate tumor segmentation is a challenging task. Hence, using the data provided by the BraTS Challenge 2020, we propose a 3D volume-to-volume Generative Adversarial Network for segmentation of brain tumours. The model, called Vox2Vox, generates realistic segmentation outputs from multi-channel 3D MR images, segmenting the whole, core and enhancing tumor with mean values of 87.20%, 81.14%, and 78.67% as dice scores and 6.44mm, 24.36mm, and 18.95mm for Hausdorff distance 95 percentile for the BraTS testing set after ensembling 10 Vox2Vox models obtained with a 10-fold cross-validation.
1 Introduction
Glioma infiltration and peritumoural edema make tumour borders difficult to define visually. The paper proposes Vox2Vox, a 3D Pix2Pix GAN for glioma segmentation, motivated by the possibility that realism-aware outputs may improve segmentation.
1 Introduction
- Diffuse, infiltrative growth and surrounding edema make glioma borders arduous to define visually, including during analysis and surgery.The introduction distinguishes low-grade and high-grade gliomas while describing this shared border-delineation challenge.Source: §1 · p. 1
1.1 Related Works
- Prior medical-imaging work applies GANs to segmentation, including a 2D GAN for brain tumours, while this paper targets 3D glioma segmentation.The cited examples also cover spinal structures, epithelial cells, and organs at risk.Source: §1.1 · p. 2
- Vox2Vox adapts paired-image Pix2Pix to 3D segmentation, with the hypothesis that penalizing unrealistic segmentations may yield better results than voxelwise metric optimization alone.The paper contrasts GAN realism penalties with conventional segmentation optimization using measures such as Dice or intersection over union.Source: §1.1 · p. 2
2 Method
Vox2Vox uses multi-modal BraTS MRI to train a 3D generator-discriminator segmentation model, combining generalized Dice supervision with adversarial feedback and a learned ensemble.
2.1 Data
- BraTS 2020 training data comprise four MRI modalities from 369 patients, with annotations approved by experienced neuroradiologists; validation data include 125 patients.Training volumes were manually segmented by one to four raters, while validation volumes had no released segmentation.Source: §2.1 · pp. 2–3
2.2 Image Pre-processing and Augmentation
- Inputs are normalized per channel and labels become four categorical channels; training uses 128 × 128 × 128 patches and random 3D geometric and intensity augmentation.Each epoch uses 23.5% of the training set, with flips, rotations, gamma changes, elastic deformation, or combinations applied probabilistically.Source: §2.2 · p. 3
2.3 Model Architecture
- The generator maps four-channel 3D MRI to four-class segmentation, using encoder, residual bottleneck, and decoder blocks; the discriminator follows a PatchGAN style.A single convolution per encoder and decoder block was chosen to reduce GPU memory, compared with two in a standard U-Net.Source: §2.3 · p. 4
2.4 Losses
- The generator loss combines discriminator realism feedback with α-weighted generalized Dice loss; α = 0 gives a pure GAN, while α → ∞ ignores the discriminator.The discriminator is trained to distinguish image-ground-truth pairs from image-predicted-segmentation pairs.Source: §2.4 · pp. 5–6
2.5 Optimization and Regularization
- Both networks use Adam with λ = 2 · 10^-4, β1 = 0.5, and β2 = 0.999; bottleneck dropout is 0.2, and the discriminator is described as a shape regularizer.The discriminator is intended to encourage spatial consistency and realistic-looking brain segmentations.Source: §2.5 · p. 6
2.6 Model Ensembling and Post-processing
- A one-layer 3D convolutional Ensembler combines the outputs of M cross-validation models, and cluster-size thresholding can remove small false-positive regions.The Ensembler returns four channels and is trained with generalized Dice loss; post-processing addresses spurious detections of classes absent from a subject.Source: §2.6 · p. 6
3 Results
On the training set, α=5 is reported as the best trade-off, although enhancing-tumour detection remains problematic. The final workflow ensembles ten models and applies ET post-processing; medians may better represent typical performance than means when poor predictions inflate variability.
3 Results
- The model is trained on 128 × 128 × 128 sub-volumes from 369 subjects and validated on 125 subjects, using batch size 4 for 200 epochs.The implementation uses Python 3.7, TensorFlow 2.1, and Keras.Source: §3 · p. 6
- The 166 test volumes are cropped to 160 × 192 × 128 so each axis is divisible by 16, allowing input to fully convolutional Vox2Vox.The passage attributes the divisibility requirement to the generator's and discriminator's depth of four.Source: §3 · p. 7
- α=5 gives the best trade-off in Table 1, though ET is not detected properly; its metrics are better than generator-only and α=0 pure-GAN settings.The table reports Dice and Hausdorff distance 95 percentile scores across the tumour classes for the training dataset.Source: Table 1 · p. 7, §3 · p. 7
- ET is the most problematic class, and only 27 training subjects (7.3%) lack ET voxels; post-processing is therefore focused on ET.The ten cross-validation models' outputs are combined, and ET voxels are reassigned to NCR/NET when fewer than 1000 voxels (1 cm3) are predicted.Source: §3 · pp. 7–8
- For WT and TC, training metrics decreased slightly from Table 1, while ET metrics increased slightly, attributed probably to augmentation and to ensembling with post-processing, respectively.The ten models are trained through 10-fold cross-validation with image augmentation, then their outputs are combined by the Ensembler CNN.Source: §3 · p. 8
- Poor predictions compromise mean values and contribute to large standard deviations, so the reported medians may better represent typical segmentation performance.Table 2 covers training, validation, and test sets with ensembling and ET post-processing.Source: Table 2 · p. 8, §3 · p. 9
4 Conclusions
Ensembling ten cross-validation models with ET post-processing produces high-quality segmentation outputs across training, validation, and test sets. The authors identify metric optimization, PatchGAN optimization, and augmentation-diverse ensembling as future improvements, and propose Vox2Vox as a possible tool for image augmentation.
4 Conclusions
- On the testing dataset, median Dice scores were 91.42%, 90.59%, and 83.65%, and median Hausdorff distance 95 percentiles were 3.16mm, 3.0mm, and 2.23mm for whole tumour, core tumour, and enhancing tumour, respectively.Source: §4 · p. 9
- Future work includes optimizing for BraTS metrics, optimizing the PatchGAN architecture, and ensembling Vox2Vox models trained with different augmentation techniques.The authors frame these as possible improvements for following BraTS challenges.Source: §4 · p. 9
- Vox2Vox could also support image augmentation by generating realistic segmentation outputs that a 3D noise-to-image GAN could translate into realistic MR volumes.The authors suggest this combination might enable fast batch generation of MR images with targets.Source: §4 · p. 9