PaperlayerView exact revision on arXiv

Source-linked AI summary

DermGAN: Synthetic Generation of Clinical Skin Images with Pathology

Amirata Ghorbani, Vivek Natarajan, David Coz, Yuan Liu

arXiv:1911.08716v1cs.CVcs.LG

TL;DR

Medical deep learning needs large expert-annotated datasets, but annotation is costly and diverse, balanced clinical data are difficult to obtain. DermGAN adapts Pix2Pix to generate clinical skin images from condition and skin-color controls; the images were high-fidelity, and classifier augmentation was comparable overall while improving rare conditions.

  • Problem

    Medical deep learning needs large expert-annotated datasets, yet annotation is costly and clinical datasets can be difficult to collect in diverse, balanced form.

    Source: §1 · pp. 1–2

  • Method

    DermGAN adapts Pix2Pix to translate semantic maps specifying a skin condition, its size and location, and skin color into clinical skin images.

    Source: §1 · p. 2, §4 · p. 11

  • Results

    Generated images were high-fidelity in objective and qualitative tests; classifier augmentation was comparable to baseline overall and improved performance on rare skin conditions.

    Source: §4 · p. 11

  • Takeaways & Limitations

    The framework generates images that reflect specified skin-color and condition-size/location changes while maintaining characteristics of the skin condition.

    Source: §4 · p. 11

    Human Turing-test participant performance had a large standard deviation, which the authors hypothesize reflects differing experience with skin images.

    Source: §3 · p. 10

Abstract

from arXiv · show

Despite the recent success in applying supervised deep learning to medical imaging tasks, the problem of obtaining large and diverse expert-annotated datasets required for the development of high performant models remains particularly challenging. In this work, we explore the possibility of using Generative Adverserial Networks (GAN) to synthesize clinical images with skin condition. We propose DermGAN, an adaptation of the popular Pix2Pix architecture, to create synthetic images for a pre-specified skin condition while being able to vary its size, location and the underlying skin color. We demonstrate that the generated images are of high fidelity using objective GAN evaluation metrics. In a Human Turing test, we note that the synthetic images are not only visually similar to real images, but also embody the respective skin condition in dermatologists' eyes. Finally, when using the synthetic images as a data augmentation technique for training a skin condition classifier, we observe that the model performs comparably to the baseline model overall while improving on rare but malignant conditions.

1. Introduction

Medical deep learning depends on large expert-annotated datasets, but annotation is costly and diverse, balanced clinical datasets are difficult to obtain. DermGAN addresses the untried task of synthesizing clinical skin images, whose varied acquisition conditions make them challenging to model.

1. Introduction

  • Expert-level medical imaging models require large annotated datasets, but obtaining annotations is often prohibitively costly.The cited work reports that prior efforts required thousands or sometimes millions of training examples.Source: §1 · p. 1
  • Datasets can be biased and unbalanced, with few examples of rare conditions, and models trained on them tend to perform poorly on underrepresented populations or rare conditions.The cited passage links this imbalance to the way healthcare sites disproportionately serve certain demographics.Source: §1 · p. 2
  • Clinical images vary in scale, perspective, lighting, blur, skin color, body parts, and background, complicating synthesis compared with dermoscopic images.Figure 1 contrasts dermatoscopic samples with cropped clinical samples and shows uncropped teledermatology images with varying size, scale, and quality.Source: §1 · pp. 2–3, Figure 1 · p. 3
  • The authors report no prior work synthesizing clinical images with skin pathology and propose generating them from semantic maps specifying condition, size, location, and skin color.They report high-fidelity images and comparable overall classifier performance with improved performance on rare malignant classes when synthetic images augment training.Source: §1 · pp. 2–3

2. Methods

DermGAN uses teledermatology images and condition-region and skin-color annotations to learn image generation from semantic maps. Its Pix2Pix adaptations address condition-focused reconstruction, image detail, and checkerboard artifacts.

2. Methods

  • The teledermatology dataset comprised 9,897 cases and 49,920 images collected at 17 clinical sites in two U.S. states from 2010 to 2018, with dermatologist-aggregated labels for 26 common skin conditions and an “other” category.The dataset was unbalanced, especially for malignant conditions such as Melanoma, which had less than 200 examples.Source: §2 · p. 3
  • Each case includes an ROI bounding box and Fitzpatrick skin-color annotation, both determined by aggregated opinions of dermatologist-trained annotators.The Fitzpatrick scale ranges from Type I to Type VI.Source: §2 · p. 4
  • ROI annotations guide random crops around one or adjacent conditions, removing background and producing 40,000 256 × 256 training images and 24,000 evaluation images.Figure 2 illustrates crops that place the skin condition clearly in focus.Source: §2 · p. 4, Figure 2 · p. 4
  • The RGB semantic map encodes skin color in its R channel and condition ROI boxes in its G and B channels, defining the controls for image generation.The goal is to sample a corresponding image for arbitrary valid maps because a map can correspond to multiple images.Source: §2 · p. 4
  • DermGAN adapts Pix2Pix with a modified U-Net generator and fully convolutional discriminator, replacing deconvolution with resizing followed by convolution to reduce checkerboard artifacts.Its four losses are whole-image and condition-specific ℓ1 reconstruction, feature matching, and min-max GAN loss.Source: Figure 3 · p. 5, §2 · pp. 5–6
  • Condition-specific ℓ1 reconstruction emphasizes the ROI, while feature matching compares discriminator-layer activations to real-image statistics and supports more stable training.Adding a separate condition-specific discriminator loss did not improve results; feature matching uses the discriminator’s second-to-last convolutional layer.Source: §2 · p. 6

3. Experiments

Experiments varied skin color and condition size, evaluated image realism and condition relevance, and tested synthetic-image augmentation for classification. The augmented classifier remained comparable overall while improving on some malignant minority classes.

Synthetic images as data augmentation for skin condition classification

  • For augmentation, the authors focused on eight conditions comprising 17% of the dataset and randomly transformed validation-set semantic maps to generate varied examples.Transformations included translating bounding boxes and randomly selecting subsets when a map contained multiple boxes.Source: §3 · p. 6

Synthetic images with different skin colors

  • Changing the semantic map’s encoded skin color across six types changed the generated background and condition appearance; darker tones also produced darker Melanocytic nevi, mimicking real data.The authors report that surrounding visual changes with skin color were consistent with real-world occurrences.Source: §3 · pp. 7–8, Figure 5 · p. 8

Synthetic images with different sizes of the skin condition

  • Gradually increasing the semantic-map bounding-box size changed the generated condition’s visual appearance, consistent with real-world occurrences.This experiment used synthetically generated semantic maps, rather than maps randomly transformed from the validation set.Source: §3 · p. 7, Figure 6 · p. 9

GAN evaluation metrics

  • The authors evaluated GANs with FID on a 24,000-image test set; their passage explains why inception score is unsuitable for this dataset but gives no ablation values.FID compares generated and real image distributions in Inception-V3 Pool 3 activation space.Source: §3 · pp. 8–9, Table 1 · p. 10

Human Turing test

  • Dermatologists’ average recall was 0.61 for real images and 0.45 for synthetic images, although synthetic images were judged more representative for Melanocytic nevus, Melanoma, and SK/ISK.Across conditions, recall ranged from 0.3 to 1.00 for real images and 0.00 to 0.67 for synthetic images.Source: §3 · p. 10
  • In the Turing test, average true positive rate was 0.52 and false positive rate was 0.30; across conditions, TPR ranged 0.51–0.69 and FPR 0.37–0.50.TPR exceeded FPR for every condition; the authors interpret the high FPR as indicating high synthetic-image fidelity.Source: §3 · p. 9, Figure 7 · p. 10

Synthetic images as data augmentation for skin condition classification

  • Top-1 accuracy remained relatively unchanged (p = 0.56); Melanoma F1 rose from 0.148 to 0.282 and Basal cell carcinoma F1 from 0.428 to 0.458, while Melanocytic nevus F1 decreased 0.113.Performance on the other five classes was comparable between the models.Source: Figure 8 · p. 11, §3 · p. 11

4. Discussion and Conclusion

DermGAN adapts Pix2Pix to generate clinical skin-condition images for teledermatology, with controls for skin color, condition size, and location. The authors report high-fidelity images and comparable classifier performance to baseline, with improvement on rare skin conditions; they identify resolution and diversity as areas for further work.

4. Discussion and Conclusion

  • DermGAN adapts Pix2Pix to generate realistic clinical images for pre-specified skin conditions in a teledermatology setting.The framework varies skin color and condition size and location; generated images reflect these changes while maintaining condition characteristics.Source: §4 · p. 11
  • Synthetic-image augmentation yields classifier performance comparable to baseline, with improved performance on rare skin conditions.The authors say further work is needed to improve generated-image resolution and diversity for classification tasks.Source: §4 · p. 11
  • The authors identify improving image resolution and diversity, using techniques such as Grover et al. (2019) for classification, and exploring other clinical applications as future work.Source: §4 · p. 11

Related papers

Loading 1911.08716v1…