PaperlayerView exact revision on arXiv

Source-linked AI summary

COVID-19 CT Image Synthesis with a Conditional Generative Adversarial Network

Yifan Jiang, Han Chen, Murray Loew, Hanseok Ko

arXiv:2007.14638v2eess.IVcs.CVcs.LG

TL;DR

Limited and labor-intensive COVID-19 CT collection makes it difficult to obtain the data needed for deep-learning medical-imaging models. The paper proposes a cGAN with global-local generation and multi-resolution discrimination to synthesize realistic CT images, and reports state-of-the-art image-quality performance with promising use in semantic segmentation and other AI applications.

  • Problem

    COVID-19 CT data are difficult to collect and label because infection-control requirements expose medical staff to risk and annotation is labor-intensive.

  • Method

    A cGAN uses a global-local generator and multi-resolution discriminator to synthesize COVID-19 CT images conditioned on semantic segmentation maps.

  • Results

    The method generates realistic COVID-19 CT images, achieves state-of-the-art image-quality performance against common synthesis approaches, and supports semantic-segmentation evaluation.

  • Takeaways & Limitations

    The synthetic images show potential for COVID-19 data augmentation and AI-based applications including semantic segmentation and diagnosis.

Abstract

from arXiv · show

Coronavirus disease 2019 (COVID-19) is an ongoing global pandemic that has spread rapidly since December 2019. Real-time reverse transcription polymerase chain reaction (rRT-PCR) and chest computed tomography (CT) imaging both play an important role in COVID-19 diagnosis. Chest CT imaging offers the benefits of quick reporting, a low cost, and high sensitivity for the detection of pulmonary infection. Recently, deep-learning-based computer vision methods have demonstrated great promise for use in medical imaging applications, including X-rays, magnetic resonance imaging, and CT imaging. However, training a deep-learning model requires large volumes of data, and medical staff faces a high risk when collecting COVID-19 CT data due to the high infectivity of the disease. Another issue is the lack of experts available for data labeling. In order to meet the data requirements for COVID-19 CT imaging, we propose a CT image synthesis approach based on a conditional generative adversarial network that can effectively generate high-quality and realistic COVID-19 CT images for use in deep-learning-based medical imaging tasks. Experimental results show that the proposed method outperforms other state-of-the-art image synthesis methods with the generated COVID-19 CT images and indicates promising for various machine learning applications including semantic segmentation and classification.

I. INTRODUCTION

COVID-19 CT data are valuable for diagnosis and AI-based imaging, but collection and annotation are difficult because infection-control demands and labeling workloads constrain available datasets. The paper proposes a cGAN with global-local generation and multi-resolution discrimination to synthesize realistic CT images for these needs.

  • Motivation: COVID-19 CT collection is difficult and time-consuming because infection-control procedures expose medical staff to risk and segmentation requires labor-intensive labeling.These constraints limit the availability of well-labeled data for deep-learning models.
  • Proposed approach: The proposed cGAN combines a global-local generator with a multi-resolution discriminator to generate CT images with stable global structure and diverse local details.The generator and discriminator use dual-network designs that learn global and local information separately while exchanging information.
  • Proposed approach: The global-local generator contains two generators that address different levels of CT information, while the multi-resolution discriminator uses full- and half-resolution inputs.The two generator components and two discriminator components are described as distinct contributors to global and local learning.
  • Proposed approach: The dynamic element-wise sum and dynamic feature matching mechanisms weight information or losses across global and local pathways during training.DESUM balances lung and lesion information, while DFM weights feature-matching losses from inputs at different resolutions.
  • Applications: The method is presented as a safe image-synthesis route with potential applications in COVID-19 data augmentation, radiologist training, and transfer to other medical-imaging domains.The stated application scope includes X-ray and MRI imaging in addition to CT.

II. RELATED WORKS

The paper builds on GAN-based conditional image synthesis and formulates COVID-19 CT synthesis as semantic-layout-conditioned image-to-image translation. Its pipeline trains a global-local generator and multi-resolution discriminator, then synthesizes lung images and restores the non-lung region.

  • Conditional image synthesis: COVID-19 CT synthesis is formulated as a semantic-layout-conditional image-to-image translation task using a cGAN.The semantic segmentation map provides the condition for generating the CT image.
  • Model design: The proposed cGAN uses global and local generators together with a multi-resolution discriminator to balance global structure and local CT details.This design is the paper’s core architectural response to the image-synthesis task.
  • Training and testing pipeline: During training, the segmentation map is passed to the generator, and real or synthesized CT images concatenated with the map are evaluated by the multi-resolution discriminator.Discriminator decisions produce losses used to update both networks.
  • Training and testing pipeline: At test time, only the generator is used to transform a data-augmented segmentation map into a synthesized lung CT image.The method then combines the synthesized lung with the non-lung area to form the final CT image.

A. Global-local generator

The global-local generator is a dual network that combines a global-information generator for high-level structure with a local-detail generator for texture and fine structures.

  • A. Global-local generator: The global-local generator contains G1 for global information and G2 for local detail, operating in a coarse-to-fine direction.G1 learns semantic labels and image structure, while G2 enhances texture and fine structures.
  • A. Global-local generator: The global-information and local-detail branches exchange information so the combined generator can learn both global structure and local details.The architecture is designed to coordinate the two levels of representation rather than synthesize them independently.
  • A. Global-local generator: The generator is trained through three procedures that separately establish the two branches before combining them through joint training.The cited overview identifies a three-step training process for the global-local generator.

1) Individual training for the global information generator:

Individual and joint training establish complementary global and local representations, while the multi-resolution discriminator assigns full- and half-resolution inputs to separate discriminators and dynamically matches their features.

  • 1) Individual training for the global information generator:: G1 processes a half-resolution 256×256 segmentation map through down-sampling, residual blocks, and up-sampling to reconstruct a half-resolution synthesized image.The residual blocks maintain 32 × 32 feature dimensions while expanding the receptive field.
  • 1) Individual training for the global information generator:: G2 uses a full-resolution 512×512 segmentation map and preserves that resolution to learn fine texture and low-level structure.This contrasts with G1’s half-resolution processing.
  • 1) Individual training for the global information generator:: Joint training feeds the two generators inputs at different resolutions and uses dynamic feature summation so G2 receives both global and local information.The joint stage connects G1 up-sampling features with G2 down-sampling features.
  • B. Multi-resolution discriminator: The multi-resolution discriminator uses full-resolution D1 for local detail and half-resolution D2 for global information, with dynamic feature matching weighting their losses.Both sub-discriminators make patch-wise decisions, and DFM improves communication between them.

C. Dynamic communication mechanism

The method uses dynamic communication to balance global lung structure against local lesion detail in both generator features and discriminator losses.

  • C. Dynamic communication mechanism: DESUM dynamically weights global and local generator features, emphasizing local detail for complex lesions while preserving global lung structure.The learned weighting network adjusts α during joint training based on the input.
  • C. Dynamic communication mechanism: DFM dynamically weights feature-matching losses from full- and half-resolution discriminators to improve communication between their complementary inputs.Its weighting parameter β is computed from an intermediate feature of the full-resolution discriminator.

D. Learning objective

The learning objective combines conditional adversarial training with dynamic feature matching so the generator produces realistic images while the discriminators communicate across resolutions.

  • D. Learning objective: The overall objective combines cGAN loss with dynamic feature-matching loss, weighted by λ.The cGAN term trains realism under the segmentation-map condition, while DFM links full- and half-resolution discriminator training.
  • D. Learning objective: The cGAN loss trains the generator to produce images that can fool the discriminator when conditioned on an input segmentation map.The formulation is based on pix2pix.
  • D. Learning objective: DFM loss dynamically weights feature differences across full- and half-resolution discriminators to overcome their communication problem.This permits synchronous training and helps learn details from inputs at different resolutions.

E. Testing process

At testing time, only the pretrained generator is used to synthesize the lung region from an augmented radiologist-created segmentation map, which is then combined with the real non-lung area.

  • E. Testing process: Testing uses only the pretrained generator, taking an augmented segmentation map as input to synthesize a CT image of the lung area.The segmentation maps can be augmented from radiologist-produced masks using standard image-editing software.
  • E. Testing process: The synthesized lung image is combined with the corresponding non-lung region from the real image to form a complete synthesized CT image.

IV. EXPERIMENTS

Experiments evaluate synthetic COVID-19 CT images using a public infected-slice dataset, image-quality metrics, and semantic-segmentation settings with varying synthetic-data proportions.

  • A. Experimental settings: The study uses 446 infection-containing slices selected from 829 lung CT slices of nine COVID-19 patients in the Radiopaedia dataset.The dataset includes CT images, lung masks, and infection masks labeling ground-glass opacity and consolidation.
  • A. Experimental settings: Image quality is evaluated with FID, PSNR, SSIM, and RMSE, while semantic segmentation is assessed using Dice, sensitivity, and specificity.
  • A. Experimental settings: Table III compares semantic-segmentation results when different proportions of real data are replaced by synthetic data from multiple image-synthesis methods.The listed alternatives include SEAN, SPADE, pix2pixHD, and pix2pix.
  • A. Experimental settings: Table IV evaluates semantic-segmentation results when synthetic data are added at different proportions.

B. Quantitative results

The proposed method is evaluated for image quality and semantic segmentation using synthetic COVID-19 CT images. It outperforms competing synthesis methods on image-quality metrics, while synthetic data maintains stable segmentation performance and improves results when added to real data.

  • 1) Image quality evaluation:: The proposed method outperforms other state-of-the-art methods on four image-quality metrics for both complete and lung-only synthetic CT images.The global-local generator and multi-resolution discriminator support global structure, local details, and a relatively high signal-to-noise ratio.
  • 2) Medical imaging semantic segmentation evaluation:: The semantic-segmentation evaluation trains Unet on mixtures of real and synthetic CT images and tests it on a fixed real CT dataset.The experiments separately assess replacing real images with synthetic data and adding synthetic images while keeping the real-image count fixed.
  • 2) Medical imaging semantic segmentation evaluation:: Unet segmentation performance remains stable when synthetic images replace up to 50% of real training data, indicating similarity between synthetic and real CT images.With 30% synthetic replacement, Unet achieves the best reported Spec for ground-glass opacity and Dice and Sen for consolidation.
  • 2) Medical imaging semantic segmentation evaluation:: Adding synthetic data to real training data yields the best semantic segmentation performance at a 40% synthetic-data proportion.The results indicate that the synthetic images are sufficiently diverse and realistic to improve dataset quality for deep-learning-based COVID-19 diagnosis.

C. Qualitative results

Qualitative comparisons show that the proposed method preserves lung structures, lesion textures, and boundaries more faithfully than SEAN and SPADE in representative cases. Additional patient examples illustrate realistic synthesis across different configurations of ground-glass opacity and consolidation.

  • C. Qualitative results: Compared with SEAN and SPADE, the proposed method preserves infection structure and texture while avoiding artifacts, omitted lung regions, and confused lesion boundaries.Figure 6 compares three cases involving consolidation, extensive ground-glass opacity, and both infection types.
  • C. Qualitative results: The proposed method synthesizes realistic CT images across cases with isolated or combined ground-glass opacity and consolidation.Patient examples include sharp consolidation, small consolidation within widespread ground-glass opacity, separated lesion types, and single-category infection.
  • C. Qualitative results: The ablation study evaluates dynamic element-wise summation and dynamic feature matching alongside generator and discriminator configurations.The supplied ablation-table caption identifies Table V as a study of the proposed model structure, while the accompanying passages report DESUM-related findings.

2) Dynamic feature matching process (DFM):

The proposed multi-resolution discriminator uses dynamic feature matching to stabilize training and improve synthesis performance. Its dual-resolution design and intermediate features support weighting information from inputs at different resolutions.

  • 2) Dynamic feature matching process (DFM):: Dynamic feature matching helps train the discriminator stably and improves performance across multiple metrics.The evaluation compares configurations without DFM, using an intermediate feature from D1, and using a fixed β value.
  • 3) Fine-tuning level optimization:: The dual generator and discriminator structure provides the strongest performance benefit by balancing performance and training efficiency.The authors report that suitable numbers of generators and discriminators also help avoid overfitting and improve stability.
  • V. CONCLUSION AND FUTURE STUDY: The synthesized images achieve state-of-the-art image quality and support semantic segmentation for COVID-19 diagnosis using AI models.The conclusion links image fidelity and segmentation evaluation to the proposed method’s potential use in diagnosis-oriented computer vision tasks.
Loading 2007.14638v2…