Source-linked AI summary
Combining Noise-to-Image and Image-to-Image GANs: Brain MR Image Augmentation for Tumor Detection
Changhee Han, Leonardo Rundo, Ryosuke Araki, Yudai Nagano, Yujiro Furukawa, Giancarlo Mauri, Hideki Nakayama, Hideaki Hayashi
TL;DR
Small and fragmented medical-imaging datasets limit the annotated data available for computer-assisted diagnosis, and prior work had not combined noise-to-image with image-to-image GANs. The paper proposes a two-step pipeline that generates separate tumor and non-tumor brain MR images with PGGANs, then refines them with MUNIT or SimGAN. Combined with classic augmentation, the approach raises tumor-detection sensitivity from 93.67% to 97.48%.
Problem
Small and fragmented medical-imaging datasets motivate augmentation methods, but prior work had not reported combining noise-to-image and image-to-image GANs for further performance improvement.
Method
The paper uses PGGANs to generate separate realistic/diverse 256 × 256 tumor and non-tumor MR images, then refines them with MUNIT or SimGAN.
Results
97.48% sensitivity was achieved with MUNIT-based augmentation combined with classic augmentation, versus 93.67% for the best-performing classic augmentation.
Takeaways & Limitations
Combining classic augmentation with the two-step GAN-based approach significantly outperformed classic augmentation alone in tumor detection.
Abstract
from arXiv · showhide
Convolutional Neural Networks (CNNs) achieve excellent computer-assisted diagnosis with sufficient annotated training data. However, most medical imaging datasets are small and fragmented. In this context, Generative Adversarial Networks (GANs) can synthesize realistic/diverse additional training images to fill the data lack in the real image distribution; researchers have improved classification by augmenting data with noise-to-image (e.g., random noise samples to diverse pathological images) or image-to-image GANs (e.g., a benign image to a malignant one). Yet, no research has reported results combining noise-to-image and image-to-image GANs for further performance boost. Therefore, to maximize the DA effect with the GAN combinations, we propose a two-step GAN-based DA that generates and refines brain Magnetic Resonance (MR) images with/without tumors separately: (i) Progressive Growing of GANs (PGGANs), multi-stage noise-to-image GAN for high-resolution MR image generation, first generates realistic/diverse 256 X 256 images; (ii) Multimodal UNsupervised Image-to-image Translation (MUNIT) that combines GANs/Variational AutoEncoders or SimGAN that uses a DA-focused GAN loss, further refines the texture/shape of the PGGAN-generated images similarly to the real ones. We thoroughly investigate CNN-based tumor classification results, also considering the influence of pre-training on ImageNet and discarding weird-looking GAN-generated images. The results show that, when combined with classic DA, our two-step GAN-based DA can significantly outperform the classic DA alone, in tumor detection (i.e., boosting sensitivity 93.67% to 97.48%) and also in other medical imaging tasks.
I. INTRODUCTION
The paper addresses limited, fragmented medical imaging datasets by combining noise-to-image and image-to-image GANs in a two-step augmentation pipeline. This approach generates and refines whole brain MR images, improving tumor-detection sensitivity when combined with classic augmentation.
- Small, fragmented datasets limit computer-assisted diagnosis, while classic augmentation produces samples with distributions similar to the originals.
- The proposed pipeline first uses PGGANs to generate separate realistic/diverse 256 × 256 tumor and non-tumor MR images.
- MUNIT or SimGAN then refines the generated images’ texture and shape toward the real-image distribution.
- The study evaluates tumor classification while examining ImageNet pre-training, removal of weird-looking synthetic images, physician visual assessment, and t-SNE visualization.
- The approach boosts tumor-detection sensitivity from 93.67% with the best classic augmentation to 97.48% when combined with classic augmentation.
- The two-step GAN approach combines noise-to-image and image-to-image GANs, a combination presented as novel for medical data augmentation.
II. GENERATIVE ADVERSARIAL NETWORKS
The section contrasts GAN-based medical-image augmentation with VAEs and reviews prior uses of noise-to-image and image-to-image GANs. It positions the paper as combining both GAN categories while generating whole 256 × 256 medical images.
- VAEs are easier to train but often produce blurred samples because reconstruction with a single objective is imperfect.
- Prior medical-imaging studies used noise-to-image GANs for liver-lesion CT and chest cardiovascular-abnormality X-rays.
- Other studies used image-to-image GANs for breast-cancer mammography and bone-lesion X-rays, translating benign images into malignant ones and vice versa.
- The paper claims to be the first to combine noise-to-image and image-to-image GANs for medical data augmentation.
- It also claims to be the first medical GAN work generating whole 256 × 256 images rather than only small pathological regions.
III. MATERIALS AND METHODS
The study uses filtered BRATS 2016 T1c brain MR slices and trains PGGANs to generate separate 256 × 256 tumor and non-tumor images. PGGAN training uses progressive resolution growth and a Wasserstein loss with gradient penalty.
- Dataset: The dataset contains 240 × 240 contrast-enhanced T1-weighted brain MR images from 220 high-grade glioma cases in BRATS 2016.T1c is used because its high contrast supports tumor detection.
- Preprocessing: The preprocessing retains slices #30–#130 and excludes ambiguous labels, unclear tumor appearance, or images with missing brain parts.The exclusions address unreliable 2D annotations and incomplete skull-stripped images.
- Dataset: The training, validation, and test sets contain 154, 44, and 22 patients, respectively.The corresponding image counts are 4,679/3,750, 750/608, and 1,232/1,013 for tumor/non-tumor images.
- Preprocessing: Manual discarding is used for reliability when removing unsuitable images from the dataset.The authors note that this procedure could be automated but conduct it manually.
- PGGAN-based generation: PGGANs progressively grow the generator and discriminator from low resolution to synthesize realistic/diverse 256 × 256 tumor and non-tumor MR images separately.New layers model image details as training progresses.
- PGGAN-based generation: The PGGAN architecture adopts Wasserstein loss with gradient penalty, with generated samples defined as ỹ = G(z) from Gaussian input noise.The discriminator is constrained to 1-Lipschitz functions, and λgp weights the gradient penalty.
- Implementation: PGGAN training uses 100 epochs, batch size 16, Adam learning rate 1.0 × 10^-3, λgp = 10, and one critic iteration per generator iteration.Random cropping of 0–15 pixels is applied during training as data augmentation.
C. MUNIT/SIMGAN-BASED IMAGE REFINEMENT
The refinement stage transforms PGGAN-generated images toward the real image distribution using either MUNIT or SimGAN. The study selects refinement checkpoints by validation accuracy after combining refinement with classic augmentation and ImageNet pre-training.
- Refinement approach: MUNIT and SimGAN refine resized 224 × 224 PGGAN-generated tumor and non-tumor images separately to match the real image distribution.MUNIT combines auto-encoding and translation, whereas SimGAN is designed specifically for data augmentation.
- Experimental selection: The experiments randomly select 3,000 real and 3,000 PGGAN-generated images for each tumor and non-tumor refinement task.The selected MUNIT/SimGAN models are those with the highest validation accuracy among evaluated refinement steps.
- MUNIT: MUNIT is a stochastic image-to-image GAN that uses auto-encoding and translation to increase generated-image realism and diversity.It extends UNIT by representing continuous output distributions.
- MUNIT: MUNIT training jointly combines VAE, GAN, cycle-consistency, and domain-invariant perceptual streams through encoders, generators, discriminators, and VGG-based terms.The experiments replace the full MUNIT loss with the UNIT loss plus perceptual loss because style loss is unnecessary for the experiments.
- Implementation: MUNIT models are trained for 100,000 steps with batch size 1 and Adam learning rate 1.0 × 10^-4, while SimGAN uses 20,000 steps, batch size 10, and SGD learning rate 1.0 × 10^-4.SimGAN halves its learning rate at 15,000 steps and applies horizontal flipping during training.
- SimGAN: SimGAN uses self-regularization and local adversarial loss while updating its discriminator with a history of refined images.Its refiner loss adds realism through a discriminator and preserves tumor/non-tumor features through regularization.
D. TUMOR DETECTION USING RESNET-50
The study evaluates ResNet-50 tumor detection across ten data-augmentation setups, with and without ImageNet pre-training, using resized real and synthetic MR images.
- Real 240 × 240 images and PGGAN-generated 256 × 256 images are resized for ResNet-50 input.
- ResNet-50 detects brain tumors through binary classification of tumor and non-tumor MR images.
- The experiments compare ten augmentation setups, including real images, classic augmentation, PGGAN-based augmentation, and MUNIT- or SimGAN-refined variants.
- Each setup is evaluated both with and without ImageNet pre-training to examine its influence on classification performance.
- The no-pre-training and pre-training experiments use one-channel grayscale and three-channel color inputs, respectively.
E. CLINICAL VALIDATION USING VISUAL TURING TEST
The clinical validation uses an expert physician’s Visual Turing Test to assess whether real and synthetic MR images appear realistic and retain tumor-related features.
- Images are presented in random order to an expert physician without revealing their real/synthetic or tumor/non-tumor labels.
- The physician classifies each image along two dimensions: real versus synthetic and tumor versus non-tumor.
- The Visual Turing Test is used to probe human recognition of image attributes and relationships when evaluating GAN-generated medical images.
F. VISUALIZATION USING T-SNE
The t-SNE analysis visualizes how geometrically transformed and GAN-based images relate to real tumor and non-tumor MR images in a shared low-dimensional space.
- The analysis compares real images with geometrically transformed images and images generated by PGGANs, MUNIT, and SimGAN.
- Each comparison uses 300 real tumor, 300 real non-tumor, 300 transformed or GAN-based tumor, and 300 transformed or GAN-based non-tumor images.
- t-SNE reduces high-dimensional image data into a lower-dimensional representation while balancing local and global relationships.
- The visualization uses normalized pixel values, perplexity 100, 1,000 iterations, and a two-dimensional output space.
IV. RESULTS
The results section covers synthetic-image generation and refinement, physician-based evaluation, t-SNE visualization, and effects on tumor detection.
- The results examine PGGAN synthesis, MUNIT and SimGAN refinement, expert-physician evaluation, t-SNE visualization, and tumor-detection influence.
A. MR IMAGES GENERATED BY PGGANS
PGGANs generated realistic brain MR images in most cases, but a substantial minority lacked clear tumor features or contained unrealistic artifacts.
- About 75% of PGGAN-generated images captured T1c-specific texture and tumor appearance while retaining realistic brain MR structure.
- The remaining 25% lacked clear tumor/non-tumor features or contained hyper-intensity, gray contours, and odd artifacts.
B. MR IMAGES REFINED BY MUNIT/SIMGAN
MUNIT and SimGAN refined PGGAN outputs while preserving overall shape, and combining GAN-based augmentation with classic augmentation produced the strongest sensitivity results.
- B. MR IMAGES REFINED BY MUNIT/SIMGAN: MUNIT and SimGAN rendered texture and contours while maintaining the overall shape of PGGAN-generated images.
- B. MR IMAGES REFINED BY MUNIT/SIMGAN: Non-tumor images changed more remarkably than tumor images under both refinement methods.
- C. TUMOR DETECTION RESULTS: When combined with classic DA, each GAN-based DA outperformed either GAN-based DA or classic DA alone in sensitivity.The paper attributes this complementarity to GANs learning local tumor features and classic DA covering global geometric features.
- D. VISUAL TURING TEST RESULTS: The Visual Turing Test used an expert physician to classify real versus synthetic and tumor versus non-tumor images, with 50% accuracy indicating chance-level indistinguishability.
- C. TUMOR DETECTION RESULTS: 97.48% sensitivity was achieved by MUNIT-based two-step DA with classic DA, versus 93.67% for the best classic DA setup.This comparison was statistically significant against the other DA setups in sensitivity.
D. VISUAL TURING TEST RESULTS
Visual evaluation and t-SNE analysis indicated that synthetic images captured tumor features and expanded the image distribution, while refinement improved similarity to real data but could introduce slight artifacts.
- D. VISUAL TURING TEST RESULTS: An expert physician classified a few PGGAN images as real, whereas fewer GAN-refined images were classified as real because of slight refinement artifacts.
- D. VISUAL TURING TEST RESULTS: Mild and modest tumors were often recognized as non-tumor in both real and synthetic images because clinical diagnosis relies on a full 3D volume rather than one 2D slice.
- E. T-SNE RESULTS: MUNIT-refined images showed better tumor/non-tumor discrimination and greater similarity to real-image distributions than PGGAN- or SimGAN-based images.
- E. T-SNE RESULTS: GAN-based images, especially MUNIT-refined images, filled distribution regions underrepresented by real or geometrically transformed images with less tumor/non-tumor overlap.
- V. CONCLUSION: ImageNet pre-training generally improved classification, while discarding artifact-containing GAN images boosted performance only without pre-training.
- V. CONCLUSION: The authors identify object detection and segmentation as additional medical imaging tasks that may benefit from the two-step GAN-based augmentation approach.