PaperlayerView exact revision on arXiv

Source-linked AI summary

Breaking the Dilemma of Medical Image-to-image Translation

Lingke Kong, Chenyu Lian, Detian Huang, Zhenjiang Li, Yanle Hu, Qichao Zhou

arXiv:2110.06465v2eess.IVcs.CV

TL;DR

Medical image translation faces a trade-off: Pix2Pix needs aligned pairs, while Cycle-consistency handles misaligned or unpaired data but may yield multiple solutions. RegGAN treats misaligned targets as noisy labels and jointly trains translation with registration. Across the reported evaluations, it outperformed Pix2Pix on aligned data and Cycle-consistency on misaligned or unpaired data, while improving several existing methods.

  • Problem

    Pix2Pix requires well-aligned paired images, whereas Cycle-consistency can produce multiple solutions and may not meet medical translation accuracy requirements.

  • Method

    RegGAN treats misaligned target images as noisy labels and adds a registration network that adaptively fits their displacement-noise distribution during joint training.

  • Results

    RegGAN outperformed Pix2Pix on aligned data and Cycle-consistency on misaligned or unpaired data, and adding registration significantly improved evaluated methods.

  • Takeaways & Limitations

    RegGAN provides an unsupervised medical image-translation mode that does not require well-aligned paired images and can be incorporated into existing methods.

  • Takeaways & Limitations

    RegGAN may not work well on natural images because their differences may be much greater than those in medical images, making noise unsuitable to model simply as deformation errors.

Abstract

from arXiv · show

Supervised Pix2Pix and unsupervised Cycle-consistency are two modes that dominate the field of medical image-to-image translation. However, neither modes are ideal. The Pix2Pix mode has excellent performance. But it requires paired and well pixel-wise aligned images, which may not always be achievable due to respiratory motion or anatomy change between times that paired images are acquired. The Cycle-consistency mode is less stringent with training data and works well on unpaired or misaligned images. But its performance may not be optimal. In order to break the dilemma of the existing modes, we propose a new unsupervised mode called RegGAN for medical image-to-image translation. It is based on the theory of "loss-correction". In RegGAN, the misaligned target images are considered as noisy labels and the generator is trained with an additional registration network to fit the misaligned noise distribution adaptively. The goal is to search for the common optimal solution to both image-to-image translation and registration tasks. We incorporated RegGAN into a few state-of-the-art image-to-image translation methods and demonstrated that RegGAN could be easily combined with these methods to improve their performances. Such as a simple CycleGAN in our mode surpasses latest NICEGAN even though using less network parameters. Based on our results, RegGAN outperformed both Pix2Pix on aligned data and Cycle-consistency on misaligned or unpaired data. RegGAN is insensitive to noises which makes it a better choice for a wide range of scenarios, especially for medical image-to-image translation tasks in which well pixel-wise aligned data are not available

1 Introduction

Pix2Pix performs well but requires well-aligned paired images, whereas Cycle-consistency accommodates misaligned or unpaired data but may produce multiple solutions. RegGAN reframes misaligned targets as noisy labels and combines translation with registration to address both limitations.

  • Existing modes: Pix2Pix minimizes pixel-level L1 loss and therefore requires well-aligned paired images with pixel-wise correspondence.Its paired-data requirement may be difficult to satisfy in real-world scenarios.
  • Existing modes: Cycle-consistency supports misaligned or unpaired images by using forward and reverse generators linked through a cycle-consistency assumption.It generally works better than Pix2Pix when images are misaligned or unpaired.
  • Existing modes: In medical image translation, Cycle-consistency may produce multiple solutions, while misalignment can propagate errors through Pix2Pix.These properties make it difficult to preserve anatomy accurately and meet medical translation requirements.
  • RegGAN: RegGAN treats misaligned target images as noisy labels and trains an additional registration network to fit their noise distribution adaptively.Its objective is to find a common optimum for image-to-image translation and registration.
  • RegGAN: RegGAN eliminates the requirement for well-aligned paired images and reportedly outperforms Pix2Pix on aligned data and Cycle-consistency on misaligned or unpaired data.The method can also be integrated into other approaches without changing their original network architectures.

2 Related Work

Prior work established GAN-based translation, Cycle-consistency, noisy-label learning, and deformable registration as relevant foundations. RegGAN differs by using registration to adaptively remove misalignment noise during joint image-translation training.

  • Image-to-image Translation: GANs train a generator and discriminator adversarially and have been applied to medical image analysis tasks including segmentation, registration, and dose calculation.Pix2Pix uses aligned paired data, while Cycle-consistency enables translation with misaligned images.
  • Image-to-image Translation: Cycle-consistency may produce multiple solutions and become sensitive to perturbation, limiting accuracy for medical image-to-image translation.This is its main limitation identified in the related-work discussion.
  • Learning From Noisy Labels: Noisy-label research includes transition-matrix estimation, robust losses, label correction, importance weighting, and meta-learning; RegGAN estimates noise using prior knowledge of its distribution.The paper characterizes its noise as displacement error.
  • Deformable Registration: Deformable registration methods include Demons, B-spline, elastic deformation, and deep-learning approaches such as Voxelmorph.The paper uses a CNN registration network based on U-Net to model deformation.
  • Position of RegGAN: RegGAN extends related registration work by demonstrating that joint registration can improve image-to-image translation through adaptive noise elimination.The authors present it as a new medical image-to-image translation mode rather than a registration-only method.

3 Methodology

RegGAN treats misaligned target images as noisy labels and corrects their influence through a learned registration network. Its objective combines correction, deformation smoothness, and adversarial losses to recover translation performance closer to clean-label training.

  • Theoretical Motivation: Misaligned target images are modeled as noisy labels, reframing image-to-image translation as supervised learning with unknown clean targets.The goal is to train on noisy pairs while approaching performance obtained from clean paired data.
  • Theoretical Motivation: Directly optimizing the generator against misaligned labels can fail because the generator cannot remove the noise influence.This motivates an explicit loss-correction mechanism rather than relying on the generator alone.
  • RegGAN: RegGAN models label noise with a transition function and uses a registration network after the generator to match displacement errors adaptively.The registration network estimates deformation fields, while resampling corrects the generated output toward the observed target.
  • RegGAN: The registration network uses a smoothness loss to regularize deformation fields, and adversarial loss is added to form the total training objective.The total loss is expressed as LCorr + LSmooth + LAdv.

4 Experiments

Experiments evaluate RegGAN across multiple image-to-image translation methods, noise levels, and data-alignment conditions. Registration consistently improves translation performance, while RegGAN remains robust to noise and performs best on unpaired data.

  • 4.2 Performances in Different Methods: RegGAN was evaluated with CycleGAN, MUNIT, UNIT, and NICEGAN using cycle-consistency, registration-augmented, non-cycle, and RegGAN modes.The proposed NC+R mode combines adversarial training with a registration network and correction loss.
  • 4.2 Performances in Different Methods: Adding registration significantly improved all evaluated methods, and simple CycleGAN with RegGAN surpassed NICEGAN in the cycle-consistency mode using fewer parameters.The reported improvements over NICEGAN were 0.01 for NMAE, 0.4 for PSNR, and 0.03 for SSIM.
  • 4.2 Performances in Different Methods: Registration-augmented combinations produced more realistic and accurate tumor-region and detail-level images than combinations without registration.Figure 2 compares representative outputs across four methods and four training modes.
  • 4.3 Performances in Different Noise Levels: RegGAN outperformed CycleGAN under all seven noise levels and showed more stable training, with results converging to similar values after multiple epochs.The evaluation used NMAE, PSNR, and SSIM across affine and non-affine noise settings.
  • 4.3 Performances in Different Noise Levels: Pix2Pix performance deteriorated rapidly as noise increased, whereas RegGAN exceeded Pix2Pix trained without noise at every tested noise level.Under Noise.0, the authors note that slight residual misalignment may explain why registration still helps.
  • 4.4 Performances on Unpaired Dataset: Under unpaired conditions, Pix2Pix performed worst, while RegGAN achieved the best performance despite a less dramatic improvement over CycleGAN than on paired data.The authors attribute the remaining challenge to difficulty fitting the unpaired noise distribution.

Conclusion

RegGAN was introduced as a medical image-to-image translation mode that outperformed Pix2Pix and Cycle-consistency across aligned, misaligned, and unpaired settings, while adapting to varied noise levels. Its performance may not transfer well to natural images.

  • RegGAN outperformed Pix2Pix and Cycle-consistency whether datasets were aligned or not.The authors report that RegGAN adapts from no noise to large-scale noise.
  • RegGAN may not work well on natural images because natural-image differences are much greater than those in medical images.The stated limitation is that noise may not be representable simply as deformation errors in natural images.

Broader Impact

Medical image-to-image translation can support diagnosis and treatment while reducing the need for separate equipment to obtain different image sequences. However, creating large collections of well-aligned paired images is impractical in many clinical scenarios.

  • Medical image-to-image translation aids diagnosis and treatment by converting between imaging sequences.Previously, obtaining different sequences required physicians to use different medical imaging equipment.
  • Large well-aligned paired datasets are not practical to create in most clinical scenarios.

A Training Details

The experiments used a standardized PyTorch training setup and compared several established architectures, including CycleGAN, MUNIT, UNIT, and NICEGAN.

  • All experiments used PyTorch on 64-bit Ubuntu Linux with 96GB RAM and a 24GB Nvidia Titan RTX GPU.Images were normalized to [-1, 1] and resampled to 256×256.
  • Training used Adam with learning rate 1e-4, β1=0.5, β2=0.999, batch size 1, weight decay 1e-4, 80 epochs, and over 640K iterations.
  • CycleGAN used two downsampling blocks, nine residual blocks, two up-sampling blocks, and four discriminator layers.
  • MUNIT decomposed image representations into content and style codes using two encoders and a decoder.
  • UNIT assumed shared latent coding across modalities and shared high-level encoder-decoder weights.
  • NICEGAN reused discriminators for encoding to derive a more compact architecture for unsupervised image-to-image translation.

B Theoretical Analysis

The theoretical analysis models misaligned target images as noisy labels generated by deformation fields and uses loss correction to recover the clean-data optimization objective.

  • The image-to-image translation objective is to optimize a generator G mapping source images X to target images Y.For aligned paired images, the loss is evaluated between each source image x and target image y.
  • In practice, noisy labels are modeled as deformed targets, ey = y ◦T, where T is a random deformation field producing pixel displacement.
  • An unbiased registration model can fit the deformation-noise distribution so the expected corrected loss equals the clean-data loss.
  • Under a smooth deformation field satisfying T ◦T^-1 ≡ I, the corrected-loss minimizer under noisy data equals the original-loss minimizer under clean data.
  • The proof substitutes the deformed-label relation ey = y ◦T into the corrected-loss expression.

C More Rusults

RegGAN outperforms Pix2Pix across the tested noise levels. Under the largest tested noise, Pix2Pix becomes ineffective while RegGAN maintains good performance.

  • RegGAN is superior to Pix2Pix at all tested noise levels.The comparison covers Noise.0 and Noise.5, the largest noise level in the experiments.
  • At Noise.0, RegGAN produces smaller errors and smoother texture details than Pix2Pix, although the overall difference is not significant.
  • At Noise.5, Pix2Pix is no longer effective, whereas RegGAN maintains good performance.Noise.5 is identified as the largest noise level in the experiments.
  • Figures 8 and 9 show eight randomly selected examples from the total test dataset for Noise.0 and Noise.5, respectively.
Loading 2110.06465v2…