PaperlayerView exact revision on arXiv

Source-linked AI summary

SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations

Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun-Yan Zhu, Stefano Ermon

arXiv:2108.01073v2cs.CVcs.AI

TL;DR

Guided image synthesis must balance realism with faithfulness, but existing GAN-based methods can require task-specific data, retraining, inversions, or losses. SDEdit adds noise to user guides and progressively denoises them with a pretrained SDE prior. It outperforms GAN-based methods across stroke synthesis, stroke editing, and image compositing, while its generated images can be difficult for existing forensic detectors to identify.

  • Problem

    Guided image synthesis seeks realistic images faithful to user guides, while existing conditional GANs and GAN inversions can require task-specific data, retraining, or losses.

  • Method

    SDEdit adds Gaussian noise to a user guide and solves a reverse SDE from an intermediate time to produce a realistic, faithful image without task-specific training or inversion.

  • Results

    SDEdit outperforms existing GAN-based methods on stroke-based image synthesis, stroke-based image editing, and image compositing without task-specific training.

  • Takeaways & Limitations

    SDEdit balances realism and faithfulness across guides with different fidelity levels using a single pretrained SDE-based generative model.

  • Takeaways & Limitations

    Existing GAN-focused forensic detectors successfully detect less than 3% of SDEdit-generated images on LSUN bedroom, compared with up to 93% for GAN-based generation.

Abstract

from arXiv · show

Guided image synthesis enables everyday users to create and edit photo-realistic images with minimum effort. The key challenge is balancing faithfulness to the user input (e.g., hand-drawn colored strokes) and realism of the synthesized image. Existing GAN-based methods attempt to achieve such balance using either conditional GANs or GAN inversions, which are challenging and often require additional training data or loss functions for individual applications. To address these issues, we introduce a new image synthesis and editing method, Stochastic Differential Editing (SDEdit), based on a diffusion model generative prior, which synthesizes realistic images by iteratively denoising through a stochastic differential equation (SDE). Given an input image with user guide of any type, SDEdit first adds noise to the input, then subsequently denoises the resulting image through the SDE prior to increase its realism. SDEdit does not require task-specific training or inversions and can naturally achieve the balance between realism and faithfulness. SDEdit significantly outperforms state-of-the-art GAN-based methods by up to 98.09% on realism and 91.72% on overall satisfaction scores, according to a human perception study, on multiple tasks, including stroke-based image synthesis and editing as well as image compositing.

1 INTRODUCTION

Guided image synthesis must balance realism with faithfulness to user-provided guides, while existing GAN-based approaches can require task-specific data, retraining, or losses. SDEdit uses a pretrained SDE generative model to denoise noisy guides and achieves strong human-evaluated results across several synthesis and editing tasks.

  • Guided synthesis aims to produce images that are both realistic and faithful to user inputs such as coarse colored strokes.
  • Conditional GANs require data collection and model retraining for each new editing task, while GAN inversions require additional task-specific design choices.
  • SDEdit adds noise to a guided input, initializes an SDE from the noisy image, and progressively denoises it to preserve structure while improving realism.
  • SDEdit uses one pretrained SDE model trained on unlabeled data and tunes the realism-faithfulness trade-off through the amount of Gaussian noise and denoising duration.
  • 98.09% higher realism and 91.72% higher overall satisfaction were achieved against state-of-the-art GAN-based approaches in stroke-based synthesis human evaluations.

2 BACKGROUND: IMAGE SYNTHESIS WITH STOCHASTIC DIFFERENTIAL EQUATIONS (SDES)

SDE-based image synthesis transforms data into Gaussian noise through forward diffusion and recovers images by solving a learned reverse SDE. The framework uses noise schedules, score estimation, and stochastic numerical updates to generate samples.

  • SDEs generalize ODEs by injecting random noise, producing a time-varying stochastic process x(t) during diffusion.
  • The forward SDE starts from a data sample and progressively transforms the data distribution toward Gaussian noise.
  • VE-SDE and VP-SDE use different noise and data-magnitude schedules, but both convert the data distribution to Gaussian noise as time increases.
  • The reverse SDE travels from t = 1 to t = 0 using the noise-perturbed score function to recover samples from the data distribution.
  • Denoising score matching learns the score model with a time-specific objective whose weighted sum forms the overall training objective.
  • Euler-Maruyama approximates the reverse SDE by iteratively updating the state with a learned score and Gaussian random variable z.

3 GUIDED IMAGE SYNTHESIS AND EDITING WITH SDEDIT

SDEdit performs guided image synthesis and editing by initializing reverse-SDE denoising from a noisy user guide. Its key design choice, t0, controls a trade-off between realism and faithfulness.

  • Method: SDEdit uses a pretrained SDE model to synthesize and edit images from RGB-pixel guides without task-specific paired data.It adds Gaussian noise to the guide at an intermediate time and solves the reverse SDE to produce the output.
  • Setup: Guides can range from coarse colored strokes to strokes over real images or image patches, while outputs should be realistic and faithful.Realism concerns perceptual plausibility, whereas faithfulness concerns similarity to the guide.
  • Realism-faithfulness trade-off: Increasing t0 improves realism but decreases faithfulness, so SDEdit must select t0 to balance the two objectives.For reasonable guides, t0 ∈[0.3, 0.6] is reported to work well.
  • Realism-faithfulness trade-off: If t0 is too large, outputs become random realistic images; if it is too small, realism may be insufficient for guides far from the model distribution.The trade-off reflects increasing score and Gaussian-noise contributions as t0 increases.
  • Extensions: The method can support selective editing by adding a mask channel that preserves guide regions intended to remain identical.The general VE-SDE algorithm uses iterative denoising steps after noisy initialization.

4 RELATED WORK

Related approaches include conditional GANs, GAN inversion, and other score-based or energy-based generative models. SDEdit is positioned alongside these methods while emphasizing realistic, faithful, and diverse outputs from human stroke inputs.

  • Conditional GANs: Conditional GANs learn direct mappings from user inputs to edited images across synthesis, editing, and inpainting tasks.Their application to new editing tasks can require additional training data and retraining.
  • GAN inversion and editing: GAN inversion projects an input into an unconditional GAN’s latent space before synthesizing an edited image from a modified latent code.Prior methods vary in projection layers and whether network weights are fine-tuned per image.
  • Other generative models: Score-based and energy-based generative models have achieved image sample quality comparable to GANs, although prior work largely focused on unconditional synthesis.SDEdit applies an SDE-based model to guided synthesis and editing.
  • SDEdit: SDEdit produces realistic, faithful, and diverse images from human-drawn stroke inputs, extending guided generation beyond the related model families.The paper evaluates human-created stroke guides alongside algorithm-simulated guides.

5 EXPERIMENTS

Experiments evaluate SDEdit on stroke-based synthesis and editing, image compositing, and multiple datasets using realism and faithfulness metrics against GAN and traditional blending baselines. Across these settings, SDEdit produces realistic, faithful results and achieves strong human-evaluation outcomes.

  • Experimental setup: Experiments use publicly available pretrained checkpoints to compare SDEdit with GAN-based methods across LSUN and CelebA-HQ tasks.The evaluated applications are stroke-based synthesis, stroke-based editing, and image compositing.
  • Evaluation: Evaluation measures realism with KID and human comparisons, and faithfulness with normalized pixelwise L2 distance, with LPIPS used in selected experiments.Human evaluations also measure overall satisfaction combining realism and faithfulness.
  • Stroke-based image synthesis: SDEdit beats GAN baselines by more than 80% on realism and 75% on overall satisfaction in stroke-based synthesis experiments.The comparison uses both human-created and algorithm-simulated stroke guides, with 3000 additional pairwise human comparisons for overall satisfaction.
  • Stroke-based image synthesis: SDEdit preserves stroke semantics while generating realistic and diverse images, whereas the baselines struggle with realism from stroke-painting inputs.The qualitative comparisons include user-created and simulated stroke guides.
  • Image compositing: SDEdit achieves realistic, faithful image compositing with better LPIPS scores and up to 83.73% higher overall satisfaction than baselines.The study compares SDEdit with traditional blending and GAN-based methods on CelebA-HQ using human comparisons and masked LPIPS.
  • Stroke-based image editing: SDEdit produces realistic, faithful edits while avoiding undesired modifications that can blur regions outside the user’s strokes.Figure 6 compares results on LSUN bedroom, CelebA-HQ, and LSUN church against in-domain GAN and StyleGAN2-ADA baselines.

6 CONCLUSION

The paper proposes SDEdit, an SDE-based method for guided image synthesis and editing that balances realism and faithfulness without task-specific training. It reports improvements over GAN-based methods across synthesis, editing, and compositing tasks.

  • Contribution: SDEdit is a guided image synthesis and editing method based on generative stochastic differential equations.Its design targets balanced realism and faithfulness.
  • Contribution: Unlike GAN inversion and conditional GAN methods, SDEdit avoids task-specific optimization, new guide-image datasets, and model retraining.The method is demonstrated without task-specific training.
  • Conclusion: SDEdit outperforms existing GAN-based methods on stroke-based synthesis, stroke-based editing, and image compositing.These applications constitute the paper’s demonstrated scope.

ETHICS STATEMENT

The ethics statement describes both beneficial and harmful uses of SDEdit, including easier photo-realistic content creation and potential deception or misinformation. It also reports limitations in detecting SDEdit-generated images and possible evaluation bias.

  • Positive impacts: SDEdit can lower barriers to photo-realistic visual content creation for users with or without artistic expertise.The authors frame this as a potential positive societal impact.
  • Negative impacts: Because SDEdit can generate images difficult for humans to distinguish from real ones, malicious users could exploit it for deception and misinformation.The authors note that impacts depend on applications and usage.
  • Detection limitations: GAN-focused forensic methods detect fewer than 3% of SDEdit-generated LSUN bedroom images, compared with up to 93% of GAN-generated images.The authors argue that forensic methods for SDE-based models are critical as these methods become more prevalent.
  • Evaluation caveat: Human-evaluation metrics may be affected by biases from MTurk evaluators and from users’ input guidance.The human studies use Amazon Mechanical Turk pairwise comparisons.

REPRODUCIBILITY STATEMENT

The paper provides supplementary proofs, analyses, experimental details, and human-evaluation information across the appendices.

  • Appendix A provides proofs for the paper’s theoretical results.
  • Appendix C provides extra SDEdit details and pseudocode.
  • Appendix D documents the experimental settings.
  • Appendix E provides additional experimental results.
  • Appendix F provides details on the human evaluation.
  • The supplementary material includes an extra ablation-study and analysis section for SDEdit.

B.1 ANALYSIS ON THE QUALITY OF USER GUIDE

SDEdit supports varied user guides and flexible editing while exposing a controllable realism–faithfulness trade-off. It also avoids task-specific training and outperforms the cited comparison methods in the reported settings.

  • Analysis on the quality of user guide: SDEdit tolerates different input-guide qualities and consistently outperforms baselines across the tested setups.The setups vary stroke detail and include mismatched or unusual guide examples.
  • Flexible image editing: SDEdit achieves reasonable results for editing closing eyes, opening mouths, and changing lip color.
  • Realism-faithfulness trade-off: Smaller t0 produces more faithful but less realistic images, while larger t0 increases realism at the expense of faithfulness.
  • Comparison with SC-FEGAN: SDEdit is more realistic than SC-FEGAN with the same stroke guide and remains more realistic when SC-FEGAN also receives an extra sketch.
  • Comparison with Song et al. (2021): SDEdit generates realistic and faithful results without extra task-specific models, whereas the compared Song et al. approach fails on these user-guided tasks through random inpainting.
  • Applications: The unified masked SDEdit procedure supports stroke synthesis, stroke editing, and image compositing using different inputs and editable regions.

D.1 IMPLEMENTATION DETAILS

The implementation uses publicly available pretrained SDE checkpoints and specified application-dependent settings, with quantitative evaluation based on simulated stroke guides and KID or squared L2 distance.

  • Models and data: SDEdit uses publicly available pretrained SDE checkpoints rather than training new SDE models.
  • Stroke-based image synthesis: Stroke-based synthesis uses K = 1, N = 500, and t0 = 0.5, with K = 1 to 3 reported as reasonable choices.
  • Human-stroke simulation: The stroke-simulation procedure filters 256×256 images and reduces them to six colors for large-scale quantitative analysis.
  • Evaluation: KID compares generated images with real validation images, while squared L2 distance compares generated images with simulated stroke paintings.
  • Runtime: SDEdit takes 29.1s per 256×256 image on one 2080Ti GPU, faster than StyleGAN2-ADA but slower than encoder-based GAN inversion.

E.1 EXTRA RESULTS ON LSUN DATASETS

Additional experiments show that SDEdit produces realistic, faithful, and localized edits across LSUN and face-related settings, while supporting class-conditional generation and preserving unmasked regions.

  • Stroke-based image generation: SDEdit generates realistic images sharing the input paintings’ structure without paired data on LSUN bedroom.
  • Stroke-based image editing: SDEdit produces realistic and faithful stroke edits while avoiding undesired modifications outside user-specified pixels.
  • Image compositing: SDEdit generates faithful and realistic composited images for hairstyle editing and adding glasses on CelebA-HQ.
  • Attribute classification: SDEdit (VP) outperforms all baseline methods on every selected attribute in the simulated-stroke CelebA experiment.
  • Class-conditional generation: Class-conditional SDEdit generates diverse results consistent with the input class labels from the same stroke inputs.
  • Comparison with baselines: SDEdit preserves unmasked regions automatically, whereas GAN post-processing leaves strong boundary artifacts even after blending.

F.1 STROKE-BASED IMAGE GENERATION

The section presents SDEdit results for stroke-based image generation and editing across bedroom, church, face, cat, and horse images, plus compositing and class-conditional generation. Evaluations compare realism and combined faithfulness-plus-realism satisfaction against baselines.

  • Human evaluation: The evaluation measures realism through 1,500 pairwise comparisons against SDEdit for generated bedroom images.The study synthesizes 400 bedroom images per method and uses 100 evaluation tasks.
  • Human evaluation: User satisfaction combines faithfulness and realism in pairwise comparisons showing the guide, an SDEdit output, and a baseline output.A separate worker group performs 3,000 comparisons for the bedroom evaluation.
  • Stroke-based image generation and editing: SDEdit is evaluated on stroke-based generation and editing across bedroom, church, face, cat, and horse images.The figures include simulated and user-provided stroke inputs, with models pretrained on LSUN, CelebA, and related datasets.
  • Image compositing: SDEdit results also cover image compositing, including brown hair, glasses, and blond hair edits on CelebA-HQ images.The compositing models are pretrained on FFHQ, including 1024×1024 results for the VE formulation.
  • Trade-offs and conditioning: The figures illustrate a realism–faithfulness trade-off and class-conditional generation from stroke paintings.For coarser cat and horse guides, selecting t0 = 0.6 sacrifices some faithfulness for more realistic images; other examples use t0 = 0.5.
Loading 2108.01073v2…