PaperlayerView exact revision on arXiv

Source-linked AI summary

Synthesizing Diverse Lung Nodules Wherever Massively: 3D Multi-Conditional GAN-based CT Image Augmentation for Object Detection

Changhee Han, Yoshiro Kitamura, Akira Kudo, Akimichi Ichinose, Leonardo Rundo, Yujiro Furukawa, Kazuki Umemoto, Yuanzhong Li, Hideki Nakayama

arXiv:1906.04962v2cs.CVeess.IV

TL;DR

Limited and fragmented medical-image datasets constrain data augmentation, and no 3D conditional GAN-based augmentation approach addressed general bounding-box-based object detection. The paper proposes MCGAN to generate conditioned lung nodules for detector training, reporting higher sensitivity at fixed false-positive rates and nodules experts could not distinguish from real ones.

  • Problem

    Medical imaging datasets are often small or fragmented, and the paper identifies no 3D conditional GAN-based augmentation approach for general bounding-box-based object detection.

    Source: §1 · pp. 1–2

  • Method

    MCGAN generates realistic, diverse 32 × 32 × 32 lung nodules conditioned on position, size, and attenuation, using context and nodule discriminators.

    Source: §1 · p. 2, §3.1 · p. 4

  • Results

    MCGAN-generated nodules yield higher 3D CNN detection sensitivity across nodule sizes and attenuation levels at fixed false-positive rates, and expert physicians could not distinguish generated from real nodules.

    Source: §1 · p. 2

  • Takeaways & Limitations

    MCGAN offers a bounding-box-based 3D augmentation approach for object detection when annotated medical images are limited.

    Source: §1 · p. 2, §2 · p. 3

Abstract

from arXiv · show

Accurate Computer-Assisted Diagnosis, relying on large-scale annotated pathological images, can alleviate the risk of overlooking the diagnosis. Unfortunately, in medical imaging, most available datasets are small/fragmented. To tackle this, as a Data Augmentation (DA) method, 3D conditional Generative Adversarial Networks (GANs) can synthesize desired realistic/diverse 3D images as additional training data. However, no 3D conditional GAN-based DA approach exists for general bounding box-based 3D object detection, while it can locate disease areas with physicians' minimum annotation cost, unlike rigorous 3D segmentation. Moreover, since lesions vary in position/size/attenuation, further GAN-based DA performance requires multiple conditions. Therefore, we propose 3D Multi-Conditional GAN (MCGAN) to generate realistic/diverse 32 X 32 X 32 nodules placed naturally on lung Computed Tomography images to boost sensitivity in 3D object detection. Our MCGAN adopts two discriminators for conditioning: the context discriminator learns to classify real vs synthetic nodule/surrounding pairs with noise box-centered surroundings; the nodule discriminator attempts to classify real vs synthetic nodules with size/attenuation conditions. The results show that 3D Convolutional Neural Network-based detection can achieve higher sensitivity under any nodule size/attenuation at fixed False Positive rates and overcome the medical data paucity with the MCGAN-generated realistic nodules---even expert physicians fail to distinguish them from the real ones in Visual Turing Test.

1. Introduction

The paper addresses the lack of 3D conditional GAN augmentation for bounding-box-based object detection, despite limited medical data and the lower annotation cost of bounding boxes compared with 3D segmentation. It proposes MCGAN to generate conditioned lung nodules and reports improved detection sensitivity at fixed false-positive rates.

1. Introduction

  • Large-scale annotated pathological-image datasets support CNN-based diagnosis, but collecting and annotating diverse examples is laborious, and conventional one-to-one augmentation cannot sufficiently cover the real-image distribution.The paper motivates GAN augmentation as a way to generate new samples for training.Source: §1 · p. 1
  • No 3D conditional GAN-based augmentation approach existed for general bounding-box-based 3D object detection, although bounding boxes require less annotation effort than rigorous 3D segmentation.The paper identifies lesion variation in position, size, and attenuation as a reason to use multiple conditions.Source: §1 · p. 2
  • MCGAN-generated training images yield higher 3D CNN nodule-detection sensitivity across nodule sizes and attenuation levels at fixed false-positive rates.The introduction also reports that expert physicians could not distinguish the generated nodules from real ones.Source: §1 · p. 2
  • MCGAN generates realistic, diverse 32 × 32 × 32 nodules at desired positions, sizes, and attenuation levels on lung CT scans.Its context discriminator evaluates nodule/surrounding pairs, while its nodule discriminator evaluates nodules conditioned on size and attenuation.Source: §1 · p. 2

2. Generative Adversarial Networks

Prior work applied GANs to image generation and medical-image augmentation, but 3D medical approaches largely focused on segmentation or subvolume tasks. This work presents a 3D medical GAN augmentation approach that conditions nodule placement for bounding-box object detection.

2. Generative Adversarial Networks

  • GANs use a two-player minimax game and have been used for computer-vision classification, object detection, and segmentation augmentation.The paper also notes training difficulties, artifacts, and mode collapse, especially for high-resolution, 3D, or conditional generation.Source: §2 · p. 2
  • Earlier 3D medical GAN approaches generated CT images for segmentation or nodule subvolumes for binary classification, while most other 3D object detectors could not use generated nodules lacking position conditioning as additional training data.The cited related work includes 64 × 64 × 64 CT images of nodules and surroundings for 2D segmentation and 40 × 40 × 18 nodule subvolumes for 3D detection.Source: §2 · pp. 2–3
  • The paper presents 3D medical GAN-based augmentation with automatic bounding-box annotation, which requires less annotation effort than rigorous 3D segmentation.It generates 32 × 32 × 32 nodules at desired positions, sizes, and attenuation levels.Source: §2 · p. 3

3. Methods

The methods prepare conditioned lung-nodule examples from LIDC scans and train a 3D generator with complementary context and nodule discriminators. The study also sets up a comparison of GAN objectives with and without reconstruction ℓ1 loss.

3.1. 3D MCGAN-based Image Generation

  • The study uses LIDC scans selected by slice-thickness and in-plane-spacing criteria, then divides 745 scans into training, validation, and test sets; only training scans train MCGAN.The split is 632 training scans, 37 validation scans, and 76 test scans; scans with more than 400 slices are excluded.Source: §3.1 · p. 3
  • The generator receives 64 × 64 × 64 × 7 inputs combining a noise-filled nodule box, its surrounding volume, and six size/attenuation condition channels.Nodules are cropped or resized to 32 × 32 × 32 voxels, while the surrounding volume is retained at 64 × 64 × 64.Source: §3.1 · p. 3
  • A 3D U-Net-like generator uses four encoder convolutional layers, four decoder deconvolutional layers, and skip connections to capture nodule and context information.The method also applies dropout and batch normalization.Source: §3.1 · p. 4
  • The context discriminator classifies real versus synthetic nodule/surrounding pairs using LSGAN loss, while the nodule discriminator classifies conditioned nodules using WGAN-GP loss.The paper assigns the context loss a role in learning surrounding tissue and the nodule loss a role in generating realistic, diverse nodules without over-focusing on detail.Source: §3.1 · p. 4
  • The methods compare MCGAN objectives with and without reconstruction ℓ1 loss because that loss can make synthetic images resemble originals at the expense of diversity.The passage frames this comparison around the effect of ℓ1 loss during classifier training.Source: §3.1 · p. 4

G max D1,D2 LLSGAN(G, D1)

The section describes implementation and evaluation procedures for 3D MCGAN-based augmentation and 3D Faster RCNN detection. It also outlines physician classification and t-SNE analyses of generated nodules.

G max D1,D2 LLSGAN(G, D1)

  • Implementation details include 6,000,000 training steps for MCGAN, followed by resampling generated nodules and mapping them onto original CT scans.Training uses batch size 16 and learning rate 2.0 × 10^-4; test-time augmentation shifts and zooms nodules by up to 10%.Source: G max D1,D2 LLSGAN(G, D1) · p. 4

3.2. Lung Nodule Detection Using 3D Faster RCNN

  • The detector comparison evaluates real-only training against 1×, 2×, and 3× MCGAN augmentation, with and without ℓ1 loss, using FROC and CPM.CPM averages sensitivity at seven predefined false-positive rates per scan; testing selects the validation model under IoU threshold 0.25 and detection threshold 0.5.Source: §3.2 · p. 4, Table 1 · p. 5

3.3. Clinical Validation Using Visual Turing Test

  • Two physicians classify real and synthetic nodules in four tests, covering 32 × 32 × 32 nodules and 64 × 64 × 64 nodules with surroundings, each with or without ℓ1 loss.Each physician receives 50 real and 50 synthetic images in random order, with axial, coronal, and sagittal views.Source: §3.3 · p. 4

3.4. Visualization Using t-SNE

  • The t-SNE analysis maps 500 real nodules, 500 generated nodules, and 500 ℓ1-trained generated nodules into two dimensions.The method uses perplexity 100 for 1,000 iterations and normalizes images to [0, 1].Source: §3.4 · pp. 4–5

G max D1,D2 LLSGAN(G, D1)

  • Figure 3 contrasts 64 × 64 × 64 examples with surrounding tissue against the 32 × 32 × 32 nodules generated by MCGAN.The displayed image is a 2D axial view.Source: Figure 3 · p. 5

4. Results

MCGAN-generated nodules blend with surrounding lung tissue, and detector augmentation improves sensitivity at fixed false-positive rates. The results also report physicians’ judgments and t-SNE distributions, while showing that more augmentation or ℓ1 loss can reduce detection sensitivity.

4.1. Lung Nodules Generated by 3D MCGAN

  • Generated nodules blend naturally with surrounding vessels, soft tissues, and thoracic walls, while nodules trained without ℓ1 loss show slight shading differences from real nodules.The generated nodules are placed in noise-box regions at varied positions, sizes, and attenuation levels.Source: §4.1 · p. 5

4.2. Lung Nodule Detection Results

  • 1× MCGAN-based augmentation without ℓ1 loss outperforms real-only training across nodule sizes and attenuation, with average CPM improvement of 0.032.Detection is easier for larger, lower-attenuation nodules; adding more synthetic images tends to reduce sensitivity, and ℓ1 loss also reduces sensitivity while adding realism at the expense of diversity.Source: §4.2 · pp. 5–6, Figure 4 · p. 6, Figure 5 · p. 6

4.3. Visual Turing Test Results

  • Physicians fail to distinguish real from generated nodules without surrounding tissue, but more readily recognize synthetic nodules when surrounding tissue is included.The reported recognition difference is associated with slight shading differences between nodules and surroundings; the authors suggest the images might also train medical students and radiology trainees when images are scarce.Source: §4.3 · pp. 6–7

4.4. T-SNE Results

  • Generated nodules have a distribution similar to real nodules, but concentrate in left inner areas with fewer real examples, especially without ℓ1 loss.The authors interpret this as synthetic images partially filling regions of the real-image distribution uncovered by the original dataset.Source: §4.4 · p. 7, Figure 6 · p. 7

5. Conclusion

MCGAN-generated nodules boost 3D CNN detection sensitivity across nodule sizes and attenuations at fixed false-positive rates. The conclusion also identifies a real/synthetic training balance and several directions for future work.

5. Conclusion

  • Bounding box-based MCGAN generates diverse, CT-realistic nodules at desired positions, sizes, and attenuations that blend naturally with surrounding tissue.The authors state that these synthetic data boost sensitivity at fixed false-positive rates across nodule sizes and attenuations.Source: §5 · p. 7
  • MCGAN-generated nodules boost 3D CNN detection sensitivity across nodule sizes and attenuations at fixed false-positive rates.The authors attribute this to complementary discriminator losses and size/attenuation conditioning that cover image-distribution gaps in the original dataset.Source: §5 · p. 7
  • Over-sufficient synthetic images worsen detection results, while training without ℓ1 loss improves data-augmentation performance through increased diversity.The authors relate the first finding to real/synthetic image balance and partial coverage of the real-image distribution.Source: §5 · p. 7
  • The authors propose future comparisons with non-GAN augmentation, tests without size/attenuation conditioning, detection-oriented optimization, and physician-training use.They also suggest MCGAN could help reduce annotation work and address medical-data paucity beyond lung CT nodules.Source: §5 · p. 7

Related papers

Loading 1906.04962v2…