Source-linked AI summary
UNIT-DDPM: UNpaired Image Translation with Denoising Diffusion Probabilistic Models
Hiroshi Sasaki, Chris G. Willcocks, Toby P. Breckon
TL;DR
UNIT-DDPM targets unstable training and quality limitations in unpaired image-to-image translation. It uses a DDPM-based joint Markov-chain model with denoising score matching and conditioned Langevin-based MCMC sampling. The authors report stable training, high-quality outputs, and state-of-the-art FID performance across varied benchmark datasets, while higher-resolution models exhibit global-colour failures.
Problem
Unpaired image-to-image translation methods based on GANs require specific optimisation and architectural choices for stable training and can fail to cover all modes.
Method
UNIT-DDPM jointly models both image domains as a Markov chain using a conditioned denoising score matching objective and generates target images through conditioned Langevin-based denoising MCMC.
Results
The authors report stable training, high-quality outputs, and qualitative and quantitative outperformance of CycleGAN, UNIT, MUNIT, and DRIT++ across varied benchmark datasets.
Takeaways & Limitations
The method provides a non-adversarial approach to unpaired image-to-image translation with reported state-of-the-art FID performance across several public datasets.
Abstract
from arXiv · showhide
We propose a novel unpaired image-to-image translation method that uses denoising diffusion probabilistic models without requiring adversarial training. Our method, UNpaired Image Translation with Denoising Diffusion Probabilistic Models (UNIT-DDPM), trains a generative model to infer the joint distribution of images over both domains as a Markov chain by minimising a denoising score matching objective conditioned on the other domain. In particular, we update both domain translation models simultaneously, and we generate target domain images by a denoising Markov Chain Monte Carlo approach that is conditioned on the input source domain images, based on Langevin dynamics. Our approach provides stable model training for image-to-image translation and generates high-quality image outputs. This enables state-of-the-art Fréchet Inception Distance (FID) performance on several public datasets, including both colour and multispectral imagery, significantly outperforming the contemporary adversarial image-to-image translation methods.
1. Introduction
UNIT-DDPM addresses unstable adversarial training in unpaired image-to-image translation by using denoising diffusion probabilistic models. It models both domains jointly and uses conditioned Markov-chain sampling, with reported qualitative and quantitative gains across varied benchmark datasets.
- Realistic image synthesis supports applications including data augmentation, privacy protection, and reduced data-acquisition costs.
- GAN-based unpaired translation methods can require specific optimisation and architectural choices for stable training and may fail to cover all modes.
- UNIT-DDPM replaces adversarial networks with a DDPM backend to mitigate unstable training and improve generated-image quality.
- The method introduces a dual-domain Markov-chain model that approximates and interrelates the source and target data distributions.
- Training avoids adversarial objectives while generating realistic outputs that capture high-frequency variations across noise levels.
- Conditioned Markov Chain Monte Carlo sampling synthesises target-domain images from unpaired source-domain images.
- The approach reportedly outperforms CycleGAN, UNIT, MUNIT, and DRIT++ qualitatively and quantitatively across varied benchmark datasets, including Facade, Photos–Maps, Summer–Winter, and RGB–Thermal.
2. Related Work
Prior I2I work includes paired and unpaired mappings, with GAN-based methods addressing unpaired translation. DDPMs instead learn generation by progressively reversing noise, motivating their use for stable, domain-connected translation.
- I2I translation learns mappings between source and target image domains and is divided into paired and unpaired approaches.
- Paired methods learn from aligned source-target image pairs, while unpaired methods use independent image sets without paired examples.
- CycleGAN performs unpaired translation with GAN-based generators, discriminators, and a cycle-consistency loss.
- UNIT, MUNIT, and DRIT++ extend unpaired translation through shared-latent or disentangled representations for domain-specific attributes and shared content.
- DDPMs sequentially add noise and learn a reverse Markov process that progressively denoises white noise into images.
- UNIT-DDPM connects latent representations across domains, enabling target-domain samples to be progressively denoised from noise in relation to source images.
3. Methodology
UNIT-DDPM learns bidirectional, conditional reverse processes for source and target domains using empirical risk minimization and denoising objectives. At inference, it conditions progressive target-domain denoising on a noised source image rather than using translation functions directly.
- The method learns source-target translation by estimating joint distributions over both image domains and inferring target images from corresponding source images.
- Figure 2 separates the processing flow into model training at the top and image-translation inference at the bottom.
- During training, reverse processes in both domains are iteratively optimized, with parameters updated simultaneously using a conditional denoising objective.
- Cycle-consistency regularizes training so that the two domain translation models are bijective.
- Inference progressively synthesizes target images from Gaussian noise while conditioning on source images perturbed by the forward diffusion process.
- The source image is noised from t = T to a release timestep tr, then regenerated by the reverse process to produce the target-domain image.
4. Evaluation
UNIT-DDPM is evaluated against prior unpaired image-to-image translation methods on four public datasets, using both qualitative comparisons and FID. The evaluation also examines release-time sensitivity and higher-resolution generation.
- Datasets and comparison: The evaluation compares UNIT-DDPM with CycleGAN, UNIT, MUNIT, and DRIT++ on four public datasets with available ground-truth input-output pairs.The datasets are Facade, Photos–Maps, Summer–Winter, and RGB–Thermal.
- Datasets and comparison: The datasets contain two image domains and separate training and test sets, with images resized to 64 × 64 pixels.RGB–Thermal uses cropped aligned pedestrian-image pairs from the KAIST Multispectral Pedestrian Dataset.
- Implementation: The denoising models use a U-Net with timestep embeddings, batch normalization, ReLU activations, and no self-attention block to reduce computation.The domain translation functions use ResNet architectures with the same layer depth as the U-Net.
- Results: Qualitatively, UNIT-DDPM generates more realistic images than CycleGAN, UNIT, MUNIT, and DRIT++, while avoiding mode collapse and providing more stable training without adversarial training.Output examples are shown in Figure 5, and progressive reverse-process sampling is shown in Figure 6.
- Results: ∼20% average performance improvement over previous approaches is reported across Facade, Photos–Maps, Summer–Winter, and RGB–Thermal using FID comparisons.FID is computed between ground-truth and output images.
- Additional analysis: Changing the release time from tr = 1 to 900 produces no significant FID change, although slight dataset-dependent differences remain.The authors identify release-time tuning as dataset dependent and leave further analysis for future work.
- Limitations: At 256 × 256 resolution, the models produce wrongly coloured images across all pixels, suggesting difficulty learning global image information in the higher-dimensional image space.The authors suggest adding layers and attention to the U-Net as a possible future solution.
5. Conclusion
UNIT-DDPM achieves stable, high-quality unpaired image-to-image translation and state-of-the-art FID performance across colour and multispectral datasets, while retaining important limitations in resolution and sampling speed.
- UNIT-DDPM uses DDPM without adversarial training for unpaired image-to-image translation, jointly learning domain models through a conditioned denoising score matching objective.Target images are generated through conditioned denoising Markov Chain Monte Carlo based on Langevin dynamics.
- The method reports stable training and high-quality outputs, with state-of-the-art FID performance across several public colour and multispectral datasets.The reported evaluation significantly outperforms contemporary adversarial image-to-image translation methods.
- The supplied examples show 256 × 256-pixel outputs generated on the Facade dataset resized to that resolution.
- Performance is not uniformly positive at larger resolutions, requiring implementation changes to model large images more accurately.
- DDPM image generation has slow inference time, although modified Markovian processes or fewer learned-timestep schedules may accelerate sampling.