PaperlayerView exact revision on arXiv

Source-linked AI summary

Unsupervised Medical Image Translation with Adversarial Diffusion Models

Muzaffer Özbey, Onat Dalmaz, Salman UH Dar, Hasan A Bedel, Şaban Özturk, Alper Güngör, Tolga Çukur

arXiv:2207.08208v3eess.IVcs.CV

TL;DR

Medical image translation can impute omitted modalities, but GAN-based one-shot mappings may limit sample fidelity and diversity. SynDiff uses conditional adversarial diffusion with large reverse steps and cycle-consistent diffusive/non-diffusive modules for unpaired learning, achieving superior results across reported MRI and MRI-CT tasks.

  • Problem

    Medical image translation is needed to synthesize missing target modalities, while GANs’ implicit distribution characterization can limit synthesized image quality and diversity.

  • Method

    SynDiff combines source-conditional diffusion, an adversarial projector for large-step reverse sampling, and cycle-consistent coupled modules for unpaired bilateral translation.

  • Results

    SynDiff achieves superior quality against competing GAN and diffusion models across multi-contrast MRI and MRI-CT translation, with highest performance in all reported MRI tasks except PD→T2.

  • Takeaways & Limitations

    SynDiff provides a high-fidelity approach for imputing missing scans and translating between native MRI contrasts and MRI-to-CT modalities.

Abstract

from arXiv · show

Imputation of missing images via source-to-target modality translation can improve diversity in medical imaging protocols. A pervasive approach for synthesizing target images involves one-shot mapping through generative adversarial networks (GAN). Yet, GAN models that implicitly characterize the image distribution can suffer from limited sample fidelity. Here, we propose a novel method based on adversarial diffusion modeling, SynDiff, for improved performance in medical image translation. To capture a direct correlate of the image distribution, SynDiff leverages a conditional diffusion process that progressively maps noise and source images onto the target image. For fast and accurate image sampling during inference, large diffusion steps are taken with adversarial projections in the reverse diffusion direction. To enable training on unpaired datasets, a cycle-consistent architecture is devised with coupled diffusive and non-diffusive modules that bilaterally translate between two modalities. Extensive assessments are reported on the utility of SynDiff against competing GAN and diffusion models in multi-contrast MRI and MRI-CT translation. Our demonstrations indicate that SynDiff offers quantitatively and qualitatively superior performance against competing baselines.

I. INTRODUCTION

Medical image translation addresses costly missing-modality acquisition but remains challenging because cross-modality tissue signals vary nonlinearly and GAN mappings can limit fidelity and diversity. SynDiff combines conditional adversarial diffusion, large-step sampling, and cycle-consistent coupled modules for unpaired translation.

  • Motivation: Multi-modal imaging improves anatomical and functional assessment, but economic and labor costs motivate synthesizing missing target modalities from acquired source images.The translation problem is ill-conditioned because tissue signals vary nonlinearly across modalities.
  • Prior approaches: GANs indirectly characterize target distributions, making them potentially vulnerable to premature convergence and mode collapse that can reduce synthesized image quality and diversity.These limitations may affect generalization in GAN-based image translation.
  • SynDiff: SynDiff uses conditional diffusion, with source images guiding reverse denoising, and a source-conditional adversarial projector for efficient and accurate large-step sampling.The projector captures complex reverse transition probabilities when large diffusion steps invalidate the Gaussian assumption.
  • SynDiff: SynDiff introduces a cycle-consistent architecture coupling diffusive and non-diffusive modules to enable unsupervised translation with unpaired source-target datasets.The method translates bilaterally between two modalities.
  • Scope: The paper presents SynDiff as an adversarial diffusion model for medical synthesis and demonstrates unsupervised translation in multi-contrast MRI and multimodal MRI-CT.It contrasts conditional source-guided translation with unconditional adversarial diffusion for natural-image generation.
  • Prior approaches: GANs perform one-shot source-to-target mapping with adversarial supervision, improving sensitivity to high-frequency tissue details over pixel-wise losses.GAN-based translation has been applied to paired and unpaired medical imaging tasks.

III. THEORY

Regular diffusion models gradually add noise to images and learn reverse denoising transitions, with training based on a variational bound related to data likelihood.

  • Regular diffusion maps images to isotropic Gaussian noise through repeated Gaussian-noise additions over T time steps.The forward process is a Markov chain, with β_t controlling noise variance.
  • Reverse diffusion forms a denoising Markov chain from x_T to x_0, with each step approximated by a Gaussian transition under large T and small β_t.The network estimates reverse-transition parameters such as µ and/or Σ.
  • Training minimizes a variational bound on log-likelihood over image samples spanning the diffusion time steps.The bound is expressed using expectations under q and a network-parameterized joint distribution p_θ.
  • A common parameterization omits Σ and focuses on estimating the reverse-process mean µ.The bound decomposition includes a KL-divergence term independent of θ that is omitted during optimization.
  • During inference, sampling starts from x_T drawn from N(0, I), then repeatedly derives µ from the predicted noise and samples x_t−1.The reverse steps proceed from t=T down to 1.

B. SynDiff

SynDiff accelerates conditional diffusion for modality translation by using large reverse steps and a source-conditional adversarial projector to model complex denoising transitions.

  • SynDiff uses conditional diffusion to translate between source and target modalities, unlike unconditional diffusion models designed to generate random images.The source image anatomically guides reverse diffusion.
  • Large diffusion steps improve sampling efficiency but eliminate the closed-form reverse transition and invalidate the usual normality assumption.The method addresses this issue with an adversarial projector for q(x_t−k|x_t, y).
  • A source-conditional adversarial projector captures the complex reverse transition probability for large k.The conditional generator receives (x_t, y), gradually denoises the target, and produces x_t−k estimates.
  • The discriminator distinguishes generated and true denoising samples conditioned on the current noisy image and diffusion time.Both generator and discriminator use temporal embeddings, while the discriminator includes a gradient penalty weighted by η.
  • The unknown conditional reverse distribution is evaluated by introducing x_0 and using conditional independence, Bayes’ rule, and forward transitions.This yields a Gaussian expression for q(x_t−k|x_t, x_0) with analytically specified parameters.
  • For network-parameterized denoising, the generator output provides a deterministic sample from p_θ(x_t−k|x_t, y).

2) Network Architecture:

SynDiff combines non-diffusive and diffusive modules in a bilateral cycle-consistent architecture, allowing unpaired training while using estimated source images to guide target synthesis.

  • The architecture bilaterally translates between modalities A and B using coupled non-diffusive and diffusive modules.Two generator-discriminator pairs support each translation direction.
  • The non-diffusive module estimates a source image paired with each target image when the training data are unpaired.It produces ỹ_B from x_A and ỹ_A from x_B using conditional generator-discriminator pairs.
  • The estimated source images serve as guidance for the diffusive module’s target-image synthesis.The module produces x̂_A given ỹ_B and x̂_B given ỹ_A.
  • Diffusive synthesis starts from Gaussian noise and generates target images through T/k reverse diffusion steps.Each step first produces a deterministic denoised estimate, then samples from the corresponding denoising distribution.
  • Cycle-consistent learning compares true target images with reconstructions generated through the translation modules.

3) Learning Procedures:

SynDiff jointly trains its diffusive and non-diffusive modules with cycle-consistency and adversarial objectives, while inference uses only the generator for the desired translation direction.

  • The non-diffusive module projects estimated source images into the target domain through its generators.
  • Cycle-consistency losses compare original images with reconstructions using L1 image differences from both modules.Separate weights λ_1φ and λ_1θ control the non-diffusive and diffusive consistency terms.
  • The diffusive and non-diffusive modules are trained jointly without pretraining procedures.Overall generator and discriminator losses combine their respective adversarial and cycle-consistency components.
  • During inference, only the diffusive generator corresponding to the desired source-to-target mapping is required.Inference begins with Gaussian noise and uses the acquired source image to guide reverse diffusion.
  • Inference uses T/k reverse diffusion steps, taking the noisy target sample from each step as the next step’s input.The final sample at time step 0 is the synthetic target image.

A. Datasets

SynDiff was evaluated on three medical-imaging datasets using subject-level training, validation, and test splits. IXI used registered source-target volumes for validation and testing, enabling quantitative assessment.

  • Three datasets were divided into training, validation, and test sets with no subject overlap.
  • Quantitative assessment required paired and registered source-target volumes in the validation and test sets, although training used unpaired images.
  • IXI: IXI contained T1-, T2-, and PD-weighted images from 40 healthy subjects, split into 25 training, 5 validation, and 10 test subjects.Validation and test T2 and PD volumes were registered to T1 volumes; 100 brain-tissue-containing axial cross-sections were selected per subject.

1) IXI Dataset:

The IXI experiments assessed MRI contrast translation on healthy-subject data and compared SynDiff with GAN and diffusion baselines. The visual and methodological setup included representative translation tasks and selected training configurations.

  • IXI Dataset: SynDiff was compared with non-attentional GAN, attentional GAN, and diffusion models for unsupervised translation on unpaired modalities.Hyperparameters were selected on the validation set to maximize performance while maintaining spatial acuity.
  • IXI Dataset: Representative IXI visualizations show source, synthesized, and ground-truth images for T1→T2 and T2→PD translation.Display windows were [0 0.65] for T1→T2 and [0 0.80] for T2→PD.
  • SynDiff: SynDiff used a fast diffusion configuration with T=1000, step size k=250, and T/k=4 diffusion steps.The configuration also used cycle-consistency and adversarial-loss weights of 0.5 and 1, respectively.

2) cGAN:

The compared cycle-consistent GAN baselines used distinct architectural assumptions and attention mechanisms for unsupervised translation. Their configurations included ResNet, shared-latent-space, shared-content-space, and attention-based designs.

  • UNIT: UNIT assumed a shared latent space between source and target modalities and used cyclic translators with ResNet backbones.The model contained two discriminators and domain-image encoders and generators with a shared latent space.
  • MUNIT: MUNIT assumed shared content but distinct style distributions across source and target modalities.Its architecture included content encoders, MLP style encoders, ResNet decoders, and corresponding discriminators.
  • AttGAN: AttGAN used two convolutional attention UNet generators and two patch discriminators for unsupervised translation.
  • SAGAN: SAGAN used ResNet generators with self-attention layers in the last two residual blocks and two patch discriminators.

7) DDPM:

SynDiff was evaluated against diffusion and GAN baselines across brain MRI and pelvic MRI-CT translation tasks. Across these experiments, it achieved strong quantitative and qualitative results, while comparisons also highlighted baseline-specific artifacts and detail losses.

  • MRI-CT Translation: Representative pelvic MRI-CT images show lower artifacts and more accurate anatomy near diagnostically relevant regions for SynDiff than for diffusion and GAN baselines.UNIT performed better for some tissue-boundary segments, while SynDiff was generally stronger across the image.
  • Multi-Contrast MRI Translation: In IXI, SynDiff achieved the highest performance in every MRI translation task except PD→T2, where cGAN performed similarly.Average improvements were 2.2dB PSNR and 2.5% SSIM over non-attentional GANs, and 5.7dB PSNR and 6.6% SSIM over regular diffusion models.
  • Baseline Comparisons: Regular diffusion models exhibited blurring, spatial warping, over-smoothing, or feature losses in the reported MRI and MRI-CT comparisons.
  • Multi-Contrast MRI Translation: In BRATS, SynDiff again achieved the highest synthesis performance in all tasks except comparable cGAN results for T1→FLAIR PSNR and FLAIR→T1.Average improvements over diffusion models were 4.2dB PSNR and 6.8% SSIM.
  • MRI-CT Translation: SynDiff achieved the highest performance in all pelvic MRI-CT tasks, with average gains of 2.1dB PSNR and 7.6% SSIM over non-attentional GANs.It also exceeded attentional GANs by 3.3dB PSNR and 14.4% SSIM, and diffusion models by 3.6dB PSNR and 7.2% SSIM.

C. Model Complexity

SynDiff offers a more favorable compromise between image fidelity and computational complexity than regular diffusion models. Ablation and variability studies examine sampling efficiency, module contributions, parameter robustness, and stochastic output consistency.

  • Model Complexity: SynDiff improves inference efficiency by more than two orders of magnitude over DDPM and UNIT-DDPM while retaining higher memory use than several GAN baselines.Its training time is relatively comparable to other diffusion models, and its memory demand is comparable to MUNIT.
  • Stochastic Outputs: Across stochastic methods, sample-to-sample variability remains below 0.02dB in PSNR and 0.07% in SSIM for nearly all methods.The evaluation synthesized 10 target images for each of 50 randomly selected test cross sections.
  • Ablation Studies: The adversarial projector remains competitive with the ℓ1 projector at T/k=1000 while using T/k=4 steps.These results support large-step reverse diffusion with adversarial projections for efficient and accurate sampling.
  • Ablation Studies: SynDiff achieves substantially higher performance than variants without adversarial loss, cycle-consistency loss, or the diffusive module.The comparison uses PSNR and SSIM reported as mean±std across the test set.
  • Robustness: SynDiff shows a degree of reliability against parameter variations and moderate degradation in source-image estimates.The study separately evaluates loss-term weights, diffusion steps, pretraining, and intentionally undertrained non-diffusive modules.

VI. DISCUSSION

The discussion positions SynDiff as a fast conditional diffusion approach for constrained medical image translation, while identifying training, information, and uncertainty boundaries. Its effectiveness depends on source-image information and the quality of non-diffusive source estimates.

  • Discussion: Adversarial learning serves two roles: accurate reverse diffusion with large steps and unsupervised training through the non-diffusive module.Validation performance progressed smoothly toward convergence, and pretraining did not provide a notable benefit.
  • Discussion: The diffusive module’s performance degrades under significant discrepancies between training-time source estimates and acquired source images at inference.Moderate discrepancies are tolerated, but a well-functioning non-diffusive module remains key for training.
  • Future Directions: Future work includes many-to-one conditioning, supervised adaptation for paired data, transformer backbones, and deeper assessment of translation variability and uncertainty.These extensions address insufficient single-source information, contextual sensitivity, and uncertainty characterization.
  • Discussion: SynDiff supports fast inference relative to regular diffusion models, but its training time is higher than GANs and moderately longer than regular diffusion models.The added adversarial components and losses account for the additional training cost.
  • Discussion: Translation success is bounded by information shared between source and target domains, making some mappings substantially more ill-posed than others.The discussion specifically contrasts CT-to-MRI with mappings where the source contains more target-relevant information.
Loading 2207.08208v3…