PaperlayerView exact revision on arXiv

Source-linked AI summary

EGSDE: Unpaired Image-to-Image Translation via Energy-Guided Stochastic Differential Equations

Min Zhao, Fan Bao, Chongxuan Li, Jun Zhu

arXiv:2207.06635v5cs.CV

TL;DR

Existing SBDM-based unpaired I2I methods ignore source-domain training data, limiting their use of domain-specific and domain-independent information. EGSDE guides a pretrained SDE with a cross-domain energy function built from two feature extractors, and it outperforms SBDM baselines in almost all settings while preserving faithfulness. The method also supports realism–faithfulness trade-offs, with tuned AFHQ FID values of 51.04 for Cat →Dog and 50.43 for Wild →Dog.

  • Problem

    Existing unpaired I2I SBDM methods do not leverage source-domain training data, despite the need to balance target realism with source faithfulness.

  • Method

    EGSDE uses an energy function pretrained across both domains to guide a pretrained SDE, with feature extractors targeting domain-specific and domain-independent features.

  • Results

    EGSDE consistently outperforms SBDM-based methods in almost all settings and achieves state-of-the-art realism without harming faithful performance.

  • Takeaways & Limitations

    EGSDE enables flexible realism–faithfulness trade-offs through weighting hyper-parameters and further improves tuned realism results on AFHQ.

  • Takeaways & Limitations

    The domain-independent extractor is a simple low-pass filter rather than a more sophisticated learned extractor.

Abstract

from arXiv · show

Score-based diffusion models (SBDMs) have achieved the SOTA FID results in unpaired image-to-image translation (I2I). However, we notice that existing methods totally ignore the training data in the source domain, leading to sub-optimal solutions for unpaired I2I. To this end, we propose energy-guided stochastic differential equations (EGSDE) that employs an energy function pretrained on both the source and target domains to guide the inference process of a pretrained SDE for realistic and faithful unpaired I2I. Building upon two feature extractors, we carefully design the energy function such that it encourages the transferred image to preserve the domain-independent features and discard domain-specific ones. Further, we provide an alternative explanation of the EGSDE as a product of experts, where each of the three experts (corresponding to the SDE and two feature extractors) solely contributes to faithfulness or realism. Empirically, we compare EGSDE to a large family of baselines on three widely-adopted unpaired I2I tasks under four metrics. EGSDE not only consistently outperforms existing SBDMs-based methods in almost all settings but also achieves the SOTA realism results without harming the faithful performance. Furthermore, EGSDE allows for flexible trade-offs between realism and faithfulness and we improve the realism results further (e.g., FID of 51.04 in Cat to Dog and FID of 50.43 in Wild to Dog on AFHQ) by tuning hyper-parameters. The code is available at https://github.com/ML-GSAI/EGSDE.

1 Introduction

Unpaired I2I must balance target-domain realism with preservation of source-domain features, but existing SBDM methods ignore source-domain training data. EGSDE addresses this by guiding a pretrained SDE with an energy function learned across both domains and achieves strong benchmark results.

  • Unpaired I2I transfers source images to a related target domain while changing domain-specific features and preserving domain-independent features.
  • Existing SBDM-based I2I methods train diffusion models only on the target domain and do not use source-domain training data.
  • EGSDE guides inference from a pretrained SDE with an energy function pretrained across both source and target domains.Its energy function is designed to preserve domain-independent features and discard domain-specific ones.
  • The method compares against GAN- and SBDM-based baselines on Cat →Dog, Wild →Dog, and Male →Female tasks using four metrics.Experiments use AFHQ and CelebA-HQ datasets.
  • EGSDE consistently outperforms SBDM-based methods in almost all settings and achieves state-of-the-art realism without harming faithful performance.Tuning hyper-parameters further yields FID values of 51.04 for Cat →Dog and 50.43 for Wild →Dog.

2 Background

Score-based diffusion models learn to reverse a noise-adding stochastic process, and their reverse-time dynamics can generate target-domain images. In unpaired I2I, methods add source-image conditioning to balance realism and faithfulness, but existing approaches do not use source-domain training data.

  • 2.1 Score-based Diffusion Models: SBDMs gradually perturb data toward Gaussian noise and learn a reverse process that recovers the data distribution.
  • 2.1 Score-based Diffusion Models: The forward diffusion process is represented by an SDE governed by drift and diffusion coefficients that determine the perturbation kernel.The drift is typically affine, allowing the linear Gaussian perturbation kernel to be sampled in one step.
  • 2.1 Score-based Diffusion Models: A score model approximates the gradient of the log marginal density, inducing a reverse-time SDE for generation.
  • 2.2 SBDMs in Unpaired Image to Image Translation: Unpaired I2I designs a conditional target-domain distribution that changes domain-specific features while preserving domain-independent source features.
  • 2.2 SBDMs in Unpaired Image to Image Translation: ILVR uses target-domain diffusion for realism and refines samples with low-pass-filtered residuals from perturbed source images for faithfulness.
  • 2.2 SBDMs in Unpaired Image to Image Translation: SDEdit uses target-domain diffusion for realism and begins from a noisy source image to preserve overall structure while discarding local details.
  • 2.2 SBDMs in Unpaired Image to Image Translation: Existing methods ignore source-domain training data, which can produce sub-optimal realism and faithfulness in unpaired I2I.

3 Method

EGSDE guides a pretrained target-domain SDE with an energy function pretrained across source and target domains, defining conditional sampling for realistic and faithful unpaired translation. Its energy combines feature-based terms that discard domain-specific information while preserving domain-independent structure, and its sampling admits a product-of-experts interpretation.

  • 3 Method: EGSDE defines a valid conditional distribution by composing a pretrained SDE with an energy function under mild regularity conditions.Sampling starts from a perturbation of the source image and follows the reverse-time SDE to endpoint t = 0.
  • 3.1 Choice of Energy: The energy decomposes into realistic and faithful terms that respectively discard domain-specific features and preserve domain-independent features.The terms are weighted by λs and λi and use similarities between generated samples and perturbed source images.
  • 3 Method: The target-only pretrained SDE primarily contributes realism, while the cross-domain energy function is designed to improve both realism and faithfulness.Unlike prior methods, the energy function uses training data from both source and target domains.
  • 3.1 Choice of Energy: A classifier-derived domain-specific extractor supports realism by matching features whose reduction encourages transferred samples to discard domain-specific information.The method uses cosine similarity because it preserves spatial information and empirically improves FID.
  • 3.1 Choice of Energy: A low-pass-filter domain-independent extractor supports faithfulness by preserving overall structure while discarding local texture information.Its negative squared L2 similarity empirically preserves more domain-independent features; more sophisticated extractors could also be trained.
  • 3.2 Sampling and 3.3 EGSDE as Product of Experts: EGSDE sampling uses Euler-Maruyama with Monte Carlo estimation and can repeat the procedure K times, while its discretized distribution is interpreted as three experts.The experts comprise the pretrained SDE and two energy terms, with the score and domain-specific term serving realism and the domain-independent term serving faithfulness.

4 Related work

Related work spans GAN-based and score-based approaches to unpaired image-to-image translation, with methods differing in mapping structure, conditioning, and use of source data.

  • GAN-based unpaired I2I methods mainly use two-side or one-side mappings, with cycle consistency widely adopted in two-side frameworks.
  • Qualitative comparisons cover Cat →Dog, Wild →Dog, and Male →Female, emphasizing preservation of domain-independent features and removal of domain-specific features.
  • SBDMs-based translation methods leverage strong generative ability across text-driven, paired, and unpaired image translation settings.
  • For unpaired I2I, ILVR and SDEdit use target-domain SBDMs and refine inference with the test source image, ignoring source-domain training data.
  • EGSDE differs by using an energy function pretrained across source and target domains to improve realism and faithfulness.

5 Experiment

Experiments evaluate EGSDE across three unpaired translation tasks using realism, faithfulness, and human-preference measures, including baseline comparisons and ablations of its controls.

  • Datasets: Experiments use 256 × 256 CelebA-HQ and AFHQ images for Male→Female, Cat→Dog, and Wild→Dog translation tasks.CelebA-HQ and AFHQ provide 1000 and 500 testing images per category or domain, respectively.
  • Evaluation Metrics: Evaluation measures realism with FID, faithfulness with L2 distance, PSNR, and SSIM, and both properties through AMT pairwise comparisons.
  • Two-Domain Unpaired Image Translation: EGSDE consistently outperforms SBDM-based methods in almost all settings and achieves state-of-the-art realism without harming faithfulness.
  • Two-Domain Unpaired Image Translation: 8.35, 8.76, and 7.5 FID improvements over SDEdit occur on Cat →Dog, Wild →Dog, and Male →Female, respectively, with lower L2 distance.
  • Two-Domain Unpaired Image Translation: 25.17 and 42.51 FID improvements over CUT occur on Cat →Dog and Wild →Dog, while AMT preferences for EGSDE exceed 50% against all baselines.
  • Ablation Studies: Larger λs produces more realistic images, whereas larger λi produces more faithful images, demonstrating separate expert functions.
  • Ablation Studies: Larger initial time M yields more realistic and less faithful images, exposing a realism–faithfulness trade-off.
  • Ablation Studies: EGSDE outperforms SDEdit at every repetition count K across all metrics, while preserving source color and background when K=3.

6 Conclusions and Discussions

EGSDE uses an energy function pretrained across both domains to guide a pretrained SDE toward realistic and faithful unpaired translation. The paper also interprets discretized EGSDE sampling as a product of experts, while identifying limitations and extensions discussed in the supplied material.

  • 6 Conclusions and Discussions: EGSDE guides a pretrained SDE with an energy function learned across source and target domains.The energy design uses feature extractors to preserve domain-independent features and discard source domain-specific ones.
  • 6 Conclusions and Discussions: EGSDE outperforms state-of-the-art I2I methods on three widely adopted unpaired translation tasks.
  • 6 Conclusions and Discussions: A limitation is the use of a low-pass filter as the domain-independent feature extractor instead of a more sophisticated learned extractor.The paper leaves training a more sophisticated extractor based on disentangled representation learning for future work.
  • 6 Conclusions and Discussions: Increasing the initial time M produces more realistic but less faithful images.
  • 6 Conclusions and Discussions: The paper cautions that the method should be used carefully to avoid potential negative social impacts from generating misleading fake images.
  • A.4 EGSDE as Product of Experts: The discretized energy-guided SDE is approximately equivalent to sampling from a product of experts.

A.5 The Connection with Classifier Guidance

Classifier guidance can be expressed as a special energy-function design within EGSDE. Under the stated formulation, solving VP-EGSDE with Euler–Maruyama is equivalent to classifier guidance, which also admits a product-of-experts interpretation.

  • A.5 The Connection with Classifier Guidance: Classifier guidance is a special EGSDE design obtained by setting the energy proportional to the negative log of a time-dependent classifier.
  • A.5 The Connection with Classifier Guidance: VP-EGSDE with an Euler–Maruyama solver is equivalent to classifier guidance.
  • A.5 The Connection with Classifier Guidance: Classifier-guided samples are approximately distributed according to a product of experts involving the pretrained SDE marginal and conditioning distribution.

B.1 Datasets

The experiments use CelebA-HQ for Male→Female and AFHQ for Cat→Dog, Wild→Dog, and multi-domain translation. Images are standardized to 256×256 resolution, while training and pretrained-model configurations vary by task.

  • B.1 Datasets: CelebA-HQ contains male and female face domains and is used for the Male→Female task.It has 10,057 male training images, 17,943 female training images, and 1,000 test images per category.
  • B.1 Datasets: AFHQ contains cat, dog, and wild animal-face domains and supports Cat→Dog, Wild→Dog, and multi-domain translation.Its training counts are 5,153 cat, 4,739 dog, and 4,738 wild images, with 500 test images per domain.
  • B.1 Datasets: All images are resized to 256×256 and scaled to [−1, 1] during training and sampling.Training additionally applies random horizontal flipping with p = 0.5.
  • B.1 Datasets: Cat→Dog and Wild→Dog use the public pretrained SBDM from ILVR, while Male→Female uses an SBDM trained on female-category data.The Male→Female model is trained for 1M iterations using SDEdit-recommended code and settings.
  • B.1 Datasets: The domain-specific feature extractor is the penultimate representation of a classifier trained on both source and target domains.The classifier is initialized from an ImageNet-pretrained classifier and trained for 5K iterations for two-domain I2I or 10K for multi-domain I2I.

B.5 Training and Inference Time

On Cat→Dog, training the domain-specific feature extractor takes 7 hours on five 2080Ti GPUs, and EGSDE sampling takes 1.42 times as long as ILVR.

  • B.5 Training and Inference Time: 7 hours on five 2080Ti GPUs are required to train the domain-specific feature extractor on Cat→Dog.
  • B.5 Training and Inference Time: EGSDE sampling takes 1.42 times as long as ILVR for batch size 1.The paper notes that faster sampling methods could further improve inference speed.

B.6 Evaluation

The evaluation specifies FID procedures, human preference assessment, and reproduction settings for the compared baselines.

  • FID: FID is computed with the pytorch-fid implementation, using dataset-specific reference and preprocessing protocols.AFHQ uses test data without preprocessing, while CelebA-HQ uses training data resized to 256, 299 and normalized with specified ImageNet statistics.
  • Human evaluation: Human preference is measured through pairwise AMT comparisons of baseline and EGSDE translations for faithfulness and realism.Workers select the more satisfactory translated image for each source image; each comparison is rewarded at $0.02.
  • Baseline reproduction: Baselines are reproduced from public code, with task-specific training and inference settings for methods including CUT and ILVR.CUT uses a public pretrained model on Cat→Dog and is trained for 2M iterations on other tasks, while ILVR uses 1000 diffusion steps.

C.1 Choice of the Similarity Metrics

The ablations examine similarity metrics, energy design, initialization, repeated denoising, weighting, qualitative behavior, and comparison with StarGAN v2.

  • Choice of the Similarity Metrics: Cosine similarity for Ss improves FID, while negative squared L2 distance for Si better preserves domain-independent source features.These choices form the default similarity-metric setting used in the experiments.
  • An Alternative of Energy Function: A simpler energy function using only the original source image performs poorly because it compares a noise-free image with samples during gradual denoising.The alternative avoids expectation with respect to x_t, but Table 7 reports inferior performance.
  • Effect of the Initial Time: Larger initial time M produces more realistic but less faithful images by preserving less source information at the start.This trade-off is also summarized in the initial-time ablation table.
  • Repeating K Times: EGSDE outperforms SDEdit at every repetition step K across all metrics, whereas SDEdit’s faithful metrics decrease sharply as K increases.SDEdit uses the source image only at the initial time M, which limits faithfulness during repeated generation.
  • Weighting Hyper-parameters: Larger λs yields more realistic images, while larger λi yields more faithful images, enabling a realism–faithfulness trade-off.The λs = λi = 0 setting corresponds to SDEdit; the reported effects are consistent across the examined tasks.
  • Comparison with StarGAN v2: EGSDE outperforms StarGAN v2 under both FID measurements and retains domain-independent features such as background and color.StarGAN v2’s faithful metrics are reported as much worse, consistent with qualitative comparisons.

D Multi-Domain Image Translation

EGSDE is extended to multi-domain translation from Cat and Wild to Dog on AFHQ, retaining its two-domain energy-guided inference design.

  • Multi-Domain Image Translation: The multi-domain task uses Cat and Wild as source domains and Dog as the target domain on AFHQ.The experiments are repeated five times to reduce randomness.
  • Method: EGSDE retains an energy function pretrained across source and target domains while replacing Es with a three-class classifier feature extractor.The extractor uses all but the last layer of the classifier.
  • Results: EGSDE outperforms the baselines in almost all realism and faithfulness metrics, demonstrating generalization to multi-domain translation.The comparison includes reported quantitative results and variants involving classifier guidance and EGSDE-DDIM.
Loading 2207.06635v5…