PaperlayerView exact revision on arXiv

Source-linked AI summary

BBDM: Image-to-image Translation with Brownian Bridge Diffusion Models

Bo Li, Kaitao Xue, Bin Liu, Yu-Kun Lai

arXiv:2205.07680v2cs.CVeess.IV

TL;DR

Image-to-image translation remains difficult because diffusion methods commonly rely on conditional generation across distinct domains. BBDM instead learns domain-to-domain translation through a Brownian Bridge diffusion process and achieves competitive results across several image-translation tasks.

  • Problem

    Existing diffusion-based image-translation methods condition reverse diffusion on a reference image, without clearly guaranteeing the desired conditional distribution.

  • Method

    BBDM directly maps paired images between domains with a Brownian Bridge process whose destination is the clean conditional input, operating in VQGAN latent space.

  • Results

    Quantitative and qualitative experiments show that BBDM achieves competitive performance across various image-to-image translation tasks.

  • Takeaways & Limitations

    BBDM provides a direct Brownian Bridge-based framework for image-to-image translation and produces promising results on several different tasks.

  • Takeaways & Limitations

    The desired conditional distribution is difficult to guarantee because it does not explicitly appear in the diffusion training objective.

Abstract

from arXiv · show

Image-to-image translation is an important and challenging problem in computer vision and image processing. Diffusion models (DM) have shown great potentials for high-quality image synthesis, and have gained competitive performance on the task of image-to-image translation. However, most of the existing diffusion models treat image-to-image translation as conditional generation processes, and suffer heavily from the gap between distinct domains. In this paper, a novel image-to-image translation method based on the Brownian Bridge Diffusion Model (BBDM) is proposed, which models image-to-image translation as a stochastic Brownian bridge process, and learns the translation between two domains directly through the bidirectional diffusion process rather than a conditional generation process. To the best of our knowledge, it is the first work that proposes Brownian Bridge diffusion process for image-to-image translation. Experimental results on various benchmarks demonstrate that the proposed BBDM model achieves competitive performance through both visual inspection and measurable metrics.

1. Introduction

Image-to-image translation spans diverse vision tasks but remains difficult because existing GAN and conditional diffusion approaches face training, diversity, generalization, and theoretical limitations. BBDM addresses these issues by directly learning domain mappings through a latent Brownian Bridge diffusion process and demonstrates competitive performance across tasks.

  • Motivation: Image-to-image translation learns mappings between distinct image domains and includes style transfer, semantic image synthesis, and sketch-to-photo synthesis.A natural formulation learns the conditional distribution of target images given input-domain samples.
  • Limitations of prior methods: GAN-based methods are difficult to train, can drop modes, and often produce insufficiently diverse results because they model one-to-one mappings.These limitations occur despite their high-fidelity translation performance.
  • Limitations of prior methods: Conditional diffusion methods guide reverse diffusion by injecting encoded reference-image features into a U-Net, but lack a clear guarantee of the desired conditional distribution.Their model generalization is poor, limiting adaptation largely to applications such as inpainting and super-resolution.
  • Proposed method: BBDM directly maps input and output domains through a stochastic Brownian Bridge and bidirectional diffusion process rather than conditional generation.It performs diffusion in the same latent space used by LDM to accelerate training and inference, while avoiding conditional-information leverage.
  • Contributions: BBDM is presented as the first Brownian Bridge diffusion method for image-to-image translation and achieves competitive performance on varied translation tasks.The paper reports both quantitative and qualitative experimental support.

2. Related Work

Prior image-to-image translation methods include paired and unpaired GAN frameworks, while diffusion models have expanded to multiple image-generation and restoration tasks. Conditional diffusion injects conditions into training but may not guarantee the desired conditional distribution, whereas Brownian bridges constrain diffusion by both endpoints.

  • Image-to-image translation: Pix2Pix introduced conditional-GAN image-to-image translation, while CycleGAN and DualGAN enabled unpaired translation through dual learning across two domains.Subsequent GAN-based methods addressed unsupervised cross-domain, multi-domain, and few-shot translation settings.
  • Diffusion models: GAN-based techniques suffer from training instability and mode collapse, whereas diffusion models have achieved strong results in generation, inpainting, super-resolution, and text-to-image generation.The reviewed diffusion applications include image generation, inpainting, super-resolution, and text-to-image generation.
  • Diffusion models: Conditional diffusion models retain the forward process and inject conditions into the training objective, but p(x_t|y) does not explicitly appear, making the desired conditional distribution difficult to guarantee.This limitation motivates alternatives to treating image-to-image translation solely as conditional generation.
  • Brownian Bridge: A Brownian bridge is a continuous-time stochastic process whose distributions are conditioned on starting and ending states, with intermediate states forming a bridge between x_0 and x_T.The process is tied down at both endpoints, x_0 and x_T.

3. Method

BBDM formulates image-to-image translation as a stochastic Brownian Bridge diffusion process between paired domains, rather than conditional generation from Gaussian noise. It performs diffusion in VQGAN latent space and reverses from the reference image toward the source domain without repeatedly conditioning the prediction network on that reference.

  • Latent-space pipeline: Diffusion is performed in the latent space of VQGAN to improve learning efficiency and model generalization.
  • Brownian Bridge diffusion: BBDM models translation between paired domains as a Brownian Bridge whose destination is the clean conditional input y, rather than pure Gaussian noise.The forward process reaches x_T = y, defining a direct mapping from domain A to domain B.
  • Variance schedule: The Brownian Bridge variance starts and ends at zero, peaks at the middle step, and can be scaled to control sampling diversity.The default scaling factor is s = 1.
  • Reverse diffusion: The reverse process starts from the reference image x_T = y and predicts x_{t-1} from x_t without using y as a prediction-network input at every step.The reverse model learns the predicted mean with a neural network, while the reverse variance is analytically specified rather than learned.

4. Experiments

Experiments evaluate BBDM across diverse image-to-image translation tasks using visual, fidelity, perceptual, and diversity criteria. Results show competitive translation quality, strong FID performance, generalization across tasks, and predictable trade-offs in ablations.

  • Datasets and baselines: BBDM is evaluated on semantic synthesis, sketch-to-photo, and style transfer tasks against Pix2Pix, CycleGAN, DRIT++, CDE, and LDM.The datasets are CelebAMask-HQ, edges2shoes, edges2handbags, and faces2comics.
  • Qualitative comparison: Diffusion-based methods achieve competitive performance, while BBDM directly learns a diffusion process between domains and avoids leveraging conditional information.GAN baselines show limitations including oversmoothing, distance from the target distribution, or reduced performance on small-scale datasets.
  • Generalization and diversity: BBDM generates samples with high fidelity and diversity and achieves comparable performance on inpainting, colorization, and face-to-label translation.Diverse samples are attributed to the stochastic property of the Brownian Bridge.
  • Quantitative comparison: BBDM achieves the best FID performance on all four tasks and competitive LPIPS scores.Quantitative comparisons use FID, LPIPS, and diversity measurements.
  • Ablation studies: Sampling beyond 200 steps slightly improves FID and diversity while LPIPS remains nearly unchanged, whereas increasing variance factor s raises diversity but lowers quality and fidelity.For fewer than 200 steps, sample quality and diversity improve rapidly as sampling steps increase.

5. Conclusion and Future Work

The paper proposes BBDM, an image-to-image translation framework based on a Brownian Bridge diffusion process that directly learns translation between two domains. It produces promising results across several tasks, while leaving room for further improvement.

  • Conclusion: BBDM proposes image-to-image translation based on a Brownian Bridge diffusion process.The method is presented as a new image-to-image translation approach.
  • Conclusion: Unlike other diffusion-based methods, BBDM directly learns translation between two domains rather than using conditional generation.The framework models translation through the Brownian Bridge diffusion process.
  • Conclusion: BBDM generates promising results on several different tasks.The conclusion reports results across multiple tasks without specifying quantitative values in the supplied passage.
  • Future Work: The authors conclude that BBDM still has much room for improvement.They identify future work as an ongoing direction but the supplied passage truncates the specific example.

A. Deduction Details of Training Objective

The training-objective deduction rewrites the Brownian-bridge transition terms as Gaussian probability density functions and equates the left and right sides of Eq. (11). This yields the posterior mean and variance expressions equivalent to Eqs. (12) and (13).

  • Gaussian posterior formulation: The reverse transition q_BB(x_{t-1}|x_t, x_0, y) is represented as a Gaussian with mean μ̃_t(x_t, x_0, y) and covariance δ̃_t I.The corresponding expression is also expanded in probability-density-function form.
  • Gaussian posterior formulation: The right side of Eq. (11) is decomposed into PDFs for q_BB(x_t|x_{t-1}, y), q_BB(x_{t-1}|x_0, y), and q_BB(x_t|x_0, y).The transition q_BB(x_t|x_{t-1}, y) follows Eq. (8), while the marginal PDFs follow Eq. (4).
  • Coefficient matching: Equating the PDFs of the two sides of Eq. (11) produces coefficient equations for the posterior mean and variance, equivalent to Eqs. (12) and (13).The derived variance is δ̃_t = δ_{t|t-1}·δ_{t-1}/δ_t.

B. Implementation Details

BBDM uses an LDM-matched VQGAN and network architecture, reduced in size to enable training on a single GeForce GTX 3090 GPU. Its training procedure incorporates EMA and a ReduceLROnPlateau learning-rate scheduler.

  • Network hyperparameters: BBDM adopts the same VQGAN model and network architecture as LDM for fair comparison.The model size is reduced by modifying the total number of middle layers and middle-feature channels.
  • Network hyperparameters: The reduced network size enables BBDM to be trained on a single GeForce GTX 3090 GPU.The network details are provided in Table 6.
  • Training and sampling details: BBDM training uses Exponential Moving Average (EMA) together with a ReduceLROnPlateau learning-rate scheduler to improve performance.The corresponding hyperparameters are reported in Tables 7 and 8.

C. User Study

A subjective user study evaluates the proposed method against CDE, LDM, and OASIS using randomly selected CelebAMask-HQ editing samples and participant comparisons.

  • C. User Study: The study compares the proposed method with CDE, LDM, and OASIS, which have comparable FID measurements, using 12 randomly selected CelebAMask-HQ sample groups.For each sample, participants are randomly shown pairs of editing results.
  • C. User Study: Each participant completes 72 clicks because every image has four different editing results presented through pairwise comparisons.The study invited 112 users aged between 20 and 50.

D. Additional Qualitative Results

Additional qualitative experiments compare BBDM with other challenging methods and present more diverse samples across several datasets. Further visual results cover inpainting, colorization, and face-to-label tasks.

  • Qualitative comparisons: Additional qualitative comparisons are provided on CelebAMask-HQ, edges2shoes, edges2handbags, and faces2comics.These results are shown in Figures 9 and 11.
  • Diverse samples: More diverse samples are presented for CelebAMask-HQ, edges2shoes, and edges2handbags.These samples are shown in Figures 10 and 12.
  • Additional tasks: Additional samples demonstrate results for inpainting, colorization, and face-to-label tasks.These results are provided in Figure 13.
Loading 2205.07680v2…