PaperlayerView exact revision on arXiv

Source-linked AI summary

Sketch guided and progressive growing GAN for realistic and editable ultrasound image synthesis

Jiamin Liang, Xin Yang, Yuhao Huang, Haoming Li, Shuangchi He, Xindi Hu, Zejian Chen, Wufeng Xue, Jun Cheng, Dong Ni

arXiv:2204.06929v3eess.IVcs.CVcs.LG

TL;DR

Limited annotated ultrasound data makes training sonographers and deep-learning systems difficult, especially for rare diseases. The paper proposes sketch-guided progressive GAN synthesis with editable texture and high-resolution generation. Experiments on three datasets report promising results in synthesizing realistic ultrasound images.

  • Problem

    Limited large-scale medical-image datasets and annotations constrain deep-learning applications, while rare diseases can leave novices with little real-data training.

    Source: §1 · p. 2

  • Method

    The spGAN overlays edge sketches on object masks and progressively grows image resolution, using feature loss to align high-level generated and real-image features.

    Source: §1 · p. 2, §4 · p. 13

  • Results

    Experiments on three large ultrasound datasets, including ablations, texture editing, user studies, and segmentation tests, demonstrate promising results in realistic image synthesis.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The study presents a useful tool for generating high-resolution, high-fidelity ultrasound images from label maps.

    Source: §4 · p. 13

    Generated high-resolution images can appear blurry and contain salt-and-pepper noise, especially in regions without annotated labels or sketches.

    Source: §2.7 · p. 7

Abstract

from arXiv · show

Ultrasound (US) imaging is widely used for anatomical structure inspection in clinical diagnosis. The training of new sonographers and deep learning based algorithms for US image analysis usually requires a large amount of data. However, obtaining and labeling large-scale US imaging data are not easy tasks, especially for diseases with low incidence. Realistic US image synthesis can alleviate this problem to a great extent. In this paper, we propose a generative adversarial network (GAN) based image synthesis framework. Our main contributions include: 1) we present the first work that can synthesize realistic B-mode US images with high-resolution and customized texture editing features; 2) to enhance structural details of generated images, we propose to introduce auxiliary sketch guidance into a conditional GAN. We superpose the edge sketch onto the object mask and use the composite mask as the network input; 3) to generate high-resolution US images, we adopt a progressive training strategy to gradually generate high-resolution images from low-resolution images. In addition, a feature loss is proposed to minimize the difference of high-level features between the generated and real images, which further improves the quality of generated images; 4) the proposed US image synthesis method is quite universal and can also be generalized to the US images of other anatomical structures besides the three ones tested in our study (lung, hip joint, and ovary); 5) extensive experiments on three large US image datasets are conducted to validate our method. Ablation studies, customized texture editing, user studies, and segmentation tests demonstrate promising results of our method in synthesizing realistic US images.

1. Introduction

Limited medical-image data and annotations constrain sonographer training and deep-learning analysis, while conventional augmentation adds little new information. The paper proposes sketch-guided progressive GAN synthesis to create high-resolution, editable ultrasound images.

1. Introduction

  • Medical imaging often lacks the large annotated datasets needed for deep learning, and novices may have little real-data training for rare diseases.Sonographers also rely heavily on experience to acquire and interpret diagnostic images.Source: §1 · p. 2
  • Affine augmentation increases sample size but adds little information, motivating methods that provide more variability for training on small datasets.The paper notes that translation, rotation, and scaling make only small content changes.Source: §1 · p. 2
  • Physics-based ultrasound synthesis can be computationally intensive and time-consuming at high resolution, and may depend on difficult-to-build models.The cited approaches model wave propagation or use pre-built models.Source: §1 · p. 2
  • Conditional GANs enable user-controlled synthesis, but generated images may be low-resolution or show checkerboard artifacts, while high-resolution training is difficult and memory-intensive.The paper describes auxiliary sketch or edge guidance as prior work for improving structural details.Source: §1 · p. 2
  • The proposed spGAN combines sketch guidance and progressive training to synthesize high-resolution B-mode ultrasound images with customized texture editing.It overlays an edge sketch on the object mask and uses the composite mask as input; progressive training moves from low to high resolution, with feature loss matching high-level generated and real-image features.Source: §1 · p. 2
  • Experiments on three large ultrasound datasets include ablations, texture editing, user studies, and segmentation comparisons, and the method is proposed for other anatomical structures too.The tested structures include lung, hip joint, and ovary.Source: §1 · p. 3

2. Materials and methods

The method synthesizes ultrasound images from annotated structures augmented with edge sketches, then progressively grows a GAN from 256×256 to 512×512. A feature loss further constrains high-level image features to improve texture fidelity.

2.1. Datasets

  • Three B-mode ultrasound datasets—lung, infant hip joint, and ovary—provide annotated structures for synthesis, with distinct diagnostic or anatomical characteristics.The datasets include corresponding segmentation maps annotated by experienced doctors.Source: §2.1 · p. 3

2.2. Overview of our proposed method

  • The framework combines composite labels, a 256×256 GAN warm-up, progressive high-resolution synthesis, fade-in blocks, and feature loss.The overview describes these components as a sequence for synthesizing high-fidelity, high-resolution ultrasound images.Source: §2.2 · p. 4

2.3. Auxiliary guidance

  • Edge sketches from background texture are superposed onto object masks, supplying auxiliary guidance for realistic background texture and customized editing.The binary annotated-structure mask preserves target-object areas while the sketch is added.Source: §2.3 · p. 4

2.4. Backbone structure of GAN

  • The 256×256 backbone uses composite labels as conditional inputs and provides the base architecture for subsequent high-resolution synthesis.The generator produces ultrasound images, while the discriminator receives the composite labels with real or generated images.Source: §2.4 · p. 4, Figure 2 · p. 5

2.5. Progressive growing scheme

  • Progressive growing begins with warm-up training at 256×256, then shares backbone weights while adding layers for high-resolution synthesis.New layers are faded in to avoid sudden shocks to already trained layers.Source: §2.5 · p. 6

2.6. Fade-in Blocks

  • Fade-in blocks use weighted side and main branches to transition between resolutions; FIB-D downsamples, while FIB-U upsamples.FIB-D is used in both generator and discriminator; FIB-U is used only in the generator.Source: §2.6 · p. 6, Figure 4 · p. 6

2.7. Feature loss

  • The feature loss compares high-level features of real and generated images, addressing blur and salt-and-pepper noise while retaining the L1 loss.The authors report that removing L1 led to poor performance because low-frequency restriction was lacking.Source: §2.7 · p. 7

3. Experiments and results

Experiments evaluate synthesis across COVID-19, hip joint, and ovary ultrasound datasets using qualitative comparisons and four image metrics. They also assess physician judgments, editable synthesis, segmentation augmentation, and parameter choices.

3.1. Implementation details

  • The evaluation covered three ultrasound datasets and used FID, KID, MS-SSIM, and LPIPS; training duration varied by dataset and synthesis phase.The reported implementation used PyTorch and a single NVIDIA GeForce RTX 2080 Ti GPU.Source: §3.1 · p. 7

3.2. Low-resolution image synthesis

  • At 256×256, sketch guidance improved every metric across datasets; feature loss further improved results except hip-joint MS-SSIM and LPIPS.Qualitatively, sketch guidance removed COVID-19 checkerboard patterns, while feature loss removed scattered dark dots.Source: §3.2 · pp. 7–8, Table 1 · p. 8, Figure 5 · p. 8

3.3. High-resolution image synthesis

  • spGAN v2 improved almost all metrics over interpolation across all three datasets, except hip-joint MS-SSIM, in the high-resolution synthesis comparison.The ablations found that removing any of spGAN’s three components degraded performance in most cases, with sketch guidance the most important component.Source: §3.3 · p. 8, Table 2 · p. 10

3.4. User studies

  • In physician studies, spGAN v2 had lower average classification accuracy than the other synthesis methods on every dataset except ovary, where it exceeded spGAN.Lower accuracy indicates more realistic synthesized images; ovarian ratings also varied between Doctor 2 and the other doctors.Source: §3.4 · p. 9, Table 3 · p. 11

3.5. Editable synthesis

  • Editing label maps produced examples including more severe COVID-19, developmental hip dislocation, and ovary images with changed ovary size or follicle count.The authors describe the platform as a tool for simulating meaningful ultrasound images for training sonographers and deep neural networks.Source: §3.5 · p. 9

3.6. Segmentation experiments

  • GAN-based augmentation generally improved segmentation with both 20% and full training data, while traditional augmentation’s gains depended on dataset and training-set size.The comparison used U-Net and DICE; Table 4 reports segmentation performance with and without augmentation.Source: §3.6 · p. 9, Table 4 · p. 11

3.7. Effect of parameters

  • Parameter choices varied by dataset: feature-loss weights were 10 for COVID-19 and hip joint and 5 for ovary; residual-block counts were 15, 15, and 10, respectively.For hip joint, n=20 improved MS-SSIM and LPIPS over n=15, but visual quality was worse; discriminator output sizes were 30×30, 120×120, and 30×30.Source: §3.7 · pp. 10, 12, 13

4. Conclusion

The proposed spGAN v2 generates high-resolution, high-fidelity ultrasound images from segmentation label maps, using sketch guidance, progressive growing, and feature loss. Experiments on COVID-19, hip joint, and ovary datasets demonstrate its effectiveness, and an editable platform supports creating varied images by modifying label maps.

4. Conclusion

  • Experiments on COVID-19, hip joint, and ovary datasets demonstrate the effectiveness of spGAN v2 for high-resolution, high-fidelity ultrasound image synthesis.The framework generates images from segmentation label maps.Source: §4 · p. 13
  • Sketch guidance supplies information for realistic background texture, while progressive growing and fade-in blocks support a smooth transition from low to high resolution.Feature loss is used to suppress noise and deblur generated images.Source: §4 · p. 13
  • The editable image synthesis platform creates varied ultrasound images by modifying segmentation label maps.The study describes the framework as a useful and convenient tool for generating high-resolution, high-fidelity images from label maps.Source: §4 · p. 13

Related papers

Loading 2204.06929v3…