PaperlayerView exact revision on arXiv

Source-linked AI summary

Diffusion Models already have a Semantic Latent Space

Mingi Kwon, Jaeseok Jeong, Youngjung Uh

arXiv:2210.10960v2cs.CV

TL;DR

Diffusion models generate high-quality images but lack a semantic latent space for controllable editing. Asyrp discovers such a space, h-space, within frozen pretrained models and designs editing around measurable strength and quality deficiency. The resulting space supports homogeneous, linear, robust, and timestep-consistent attribute manipulation.

  • Problem

    Diffusion models lack a semantic latent space needed for controlling the generative process, despite strong generative performance.

  • Method

    Asyrp uses an asymmetric reverse process to discover h-space, a semantic latent space in a frozen diffusion model, with implicit neural directions across timesteps.

  • Results

    h-space exhibits homogeneity, linearity, robustness, and timestep consistency, while the generative process uses editing strength and quality deficiency to support versatile, high-quality editing.

  • Takeaways & Limitations

    The method provides a semantic latent-space framework for image editing in pretrained diffusion models without requiring model finetuning.

  • Takeaways & Limitations

    Exact latent vectors for real images can be difficult to obtain and may produce unexpected appearance changes, while ϵ-space lacks necessary semantic-latent properties.

Abstract

from arXiv · show

Diffusion models achieve outstanding generative performance in various domains. Despite their great success, they lack semantic latent space which is essential for controlling the generative process. To address the problem, we propose asymmetric reverse process (Asyrp) which discovers the semantic latent space in frozen pretrained diffusion models. Our semantic latent space, named h-space, has nice properties for accommodating semantic image manipulation: homogeneity, linearity, robustness, and consistency across timesteps. In addition, we introduce a principled design of the generative process for versatile editing and quality boost ing by quantifiable measures: editing strength of an interval and quality deficiency at a timestep. Our method is applicable to various architectures (DDPM++, iD- DPM, and ADM) and datasets (CelebA-HQ, AFHQ-dog, LSUN-church, LSUN- bedroom, and METFACES). Project page: https://kwonminki.github.io/Asyrp/

1 INTRODUCTION

Diffusion models offer strong generation and inversion properties, but existing editing methods have ambiguity, quality, or training-cost limitations. Asyrp addresses this by discovering an h-space semantic latent space in a frozen diffusion model for controllable image editing.

  • Motivation: Diffusion models achieve state-of-the-art image quality and mode coverage, and deterministic inversion makes them suitable for editing real images.Simply editing intermediate noisy latent variables can degrade results, motivating more structured control procedures.
  • Existing approaches: Image guidance ambiguously mixes attributes from a guiding image and an unconditional result, while offering limited control over change magnitude.The approach does not clearly specify which guide attribute should be reflected.
  • Existing approaches: GAN latent spaces support consistent semantic directions across images, but finding an exact latent vector for a real image is often difficult and can cause unexpected appearance changes.This limits direct transfer of GAN-style editing to real images.
  • Existing approaches: Encoder-based diffusion editing adds semantic conditioning but requires training from scratch and does not match pretrained diffusion models.This motivates discovering semantic structure directly within frozen pretrained models.
  • Proposed approach: Asyrp discovers h-space in a frozen diffusion model, where shifts edit image attributes with homogeneity, linearity, robustness, and timestep consistency.The method also designs the generative process using editing strength and quality deficiency as quantifiable measures.

2 BACKGROUND

Diffusion models generate images by progressively denoising noisy samples through a reverse process. DDIM provides a non-Markovian formulation whose deterministic setting enables nearly perfect inversion, while CLIP supplies image-text semantic similarity for directional editing.

  • DDPM: DDPM learns a data distribution by denoising noisy images through a Gaussian forward process parameterized by a Markov chain.The forward process progressively adds noise to data samples.
  • DDPM: The reverse process starts from xT sampled from N(0, I) and repeatedly applies learned transitions pθ(xt−1 | xt) using a noise predictor.This progressive denoising process generates samples from noise.
  • DDPM: The reverse-process variance is set through σ2_t I, specifying the stochasticity of each denoising step.
  • DDIM: DDIM uses a non-Markovian reverse process, and setting η = 0 makes it deterministic with nearly perfect inversion.When η = 1 for all timesteps, the process becomes DDPM.
  • CLIP: CLIP compares image and text embeddings from separate encoders, using their similarity to represent semantic similarity.Directional loss supports homogeneous editing without mode collapse.
  • CLIP: The paper uses ‘smiling face’ and ‘face’ as target and source descriptions for editing the facial attribute of smiling.

3 DISCOVERING SEMANTIC LATENT SPACE IN DIFFUSION MODELS

The paper identifies why direct latent manipulation fails in diffusion models and introduces Asyrp, which discovers a controllable semantic h-space in a frozen model by altering predicted terms while preserving the reverse-process direction.

  • Motivation and failure of naive manipulation: Directly optimizing the initial noise with directional CLIP loss can distort images or produce incorrect attribute manipulations.
  • Motivation and failure of naive manipulation: Shifting predicted terms in both Pt and Dt causes their intermediate changes to cancel, leaving the generated distribution and reconstructed image nearly unchanged.
  • Asymmetric reverse process: Asyrp breaks this interference by altering Pt while preserving Dt, so the reverse process follows the original direction while reflecting the desired h-space attribute change.
  • Asymmetric reverse process: The method controls the bottleneck feature maps ht of the U-Net because they have smaller spatial resolutions and high-level semantics than predicted noise.
  • Properties of h-space: The resulting h-space supports consistent effects across samples, linear attribute-strength control, simultaneous multi-attribute editing, quality preservation, and approximate timestep consistency.
  • Implicit neural directions: An implicit function ft generates Δht from timestep t and bottleneck features, improving learning robustness and convergence while generalizing to unseen timesteps and sampling subsequences.

4 GENERATIVE PROCESS DESIGN

The generative process divides editing, traditional denoising, and quality boosting into designed intervals. Their lengths are selected using perceptual measures that balance sufficient attribute change, image quality, and content preservation.

  • Process overview: The full process has three phases: Asyrp editing, traditional denoising, and quality boosting, with formulas determining each phase length.
  • Editing interval: Editing occurs early because diffusion models establish high-level context early and imperceptible fine details later.
  • Editing interval: LPIPS measures editing strength over [T, t], and the shortest sufficient interval empirically uses LPIPS(x, Ptedit) = 0.33.
  • Editing interval: For attributes requiring larger visual changes, the method increases editing strength using δ = 0.33d(ET(ysource), ET(ytarget)) and selects LPIPS(x, Ptedit) = 0.33 − δ.
  • Quality boosting: Quality boosting injects stochastic noise during [tboost, 0], using quality deficiency γt = LPIPS(x, xt) to choose a short interval that improves quality while limiting content changes.
  • Process integration: The selected tedit and tboost values are used to modify DDIM’s generative process, with the complete algorithm described in the appendix.

5 EXPERIMENTS

Experiments show that Asyrp enables versatile semantic editing across datasets and diffusion architectures, including attributes and domains unseen during training. Analyses support h-space properties of homogeneity, linearity, robustness, and timestep consistency, while user studies report stronger overall preference than DiffusionCLIP.

  • Versatility of h-space with Asyrp: Asyrp edits images across CelebA-HQ, LSUN-church, METFACES, AFHQ-dog, and LSUN-bedroom using frozen pretrained diffusion models.The experiments cover DDPM++, iDDPM, and ADM settings without altering the models during inference.
  • Versatility of h-space with Asyrp: Asyrp synthesizes attributes and domains unseen during training, including church-to-department, factory, and temple edits and smiling Poodles and Yorkshire dogs.Human-face edits also include different identities, painting styles, and ancient primates.
  • Quantitative comparison: 80 participants preferred Asyrp over DiffusionCLIP across image quality, natural attribute change, and overall preference.The comparison used 40 sets of original images, Asyrp results, and DiffusionCLIP results, including attributes unseen in training.
  • Analysis on h-space: A single ∆ht produces the same attribute change across images, whereas optimized ∆ϵt can distort other inputs.Applying the mean h-space shift to 20 random samples produces almost identical results.
  • Analysis on h-space: Linear scaling of ∆h controls attribute-change magnitude, negative scales generalize to unseen directions, and combining ∆h vectors produces combined semantic changes.These observations support h-space linearity beyond the trained positive direction.
  • Analysis on h-space: Random perturbations in h-space preserve realistic images with small or semantic changes, while perturbations in ϵ-space severely distort images.The experiments compare random directions with magnitudes matched to example ∆ht and ∆ϵt vectors.
  • Analysis on h-space: ∆ht, its mean, and a time-invariant global shift produce similar results across timesteps, although ∆ht is selected for best quality and manipulation.The global shift is defined over the editing interval, whose length is Te.

6 CONCLUSION

Asyrp discovers an h-space semantic latent space in pretrained diffusion models for image editing, with properties supporting consistent manipulation and quality. The paper also identifies scope limits and directions for extending the approach.

  • Asyrp enables semantic image manipulation in frozen pretrained diffusion models through h-space.The method avoids fine-tuning whole models and does not require high computation or specially designed architectures.
  • h-space provides homogeneity, linearity, robustness, and consistency across timesteps for editing.
  • Diffusion-based editing benefits from nearly perfect DDIM inversion, avoiding the difficult inversion required by GAN-based latent manipulation.
  • Diffusion models remain slower than GANs, and h-space semantics in latent diffusion models such as Stable Diffusion require further research.
  • Asyrp seldom changes overall style or peripheral objects, focusing edits on attributes of the main object.Style transfer using frozen diffusion models is identified as future work.

C PROOF OF THEOREM 1

Theorem 1 explains why shifting both predicted components in the reverse process fails to manipulate the reconstructed image: their effects cancel. The proof uses the diffusion schedule and shifted-noise formulation.

  • Shifting both Pt and Dt produces the same output as the original reverse process because their intermediate changes cancel.Figure 12 illustrates the cancellation caused by applying the same noise shift to both terms.
  • The reverse-process derivation uses a decreasing βt schedule from βT = βmax to β1 = βmin ≈ 0.The text gives example values βmax = 0.001 and βmin = 0.00001.

D ADDITIONAL SUPPORTS FOR h-space WITH ASYRP

Additional analyses support h-space as a semantic and robust editing space, identify the eighth U-Net layer as its location, and guide editing-interval selection across datasets.

  • Shifting both Pt and Dt produces no noticeable difference from the original DDIM reverse process.
  • h-space is more robust than ϵ-space: noise in ϵ-space degrades images without semantic change, whereas larger h-space noise changes semantics without severe artifacts.
  • The eighth U-Net layer is selected as h-space because it lacks skip connections, compresses information spatially, and precedes upsampling blocks.Exhaustive layer comparisons report that the eighth layer suits the semantic latent space best.
  • Preserving the sum of ∆ht across inference steps maintains similar attribute changes, while 1000-step sampling improves quality over 40-step sampling.Sampling takes about 1.5 seconds for 40 steps and 40 seconds for 1000 steps.
  • Editing intervals that are too short or too long respectively cause insufficient changes or excessive unnecessary manipulation.The tedit formula works across five datasets, although the LPIPS = 0.33 choice is empirical.

H QUALITY BOOSTING

The quality-boosting process improves fine details and reduces noise in both original DDIM and Asyrp settings while preserving near-perfect inversion. Its behavior depends on interval and attribute-specific parameters.

  • The quality-boosting process uses editing and quality-deficiency measures to design the generative process.
  • Quality boosting improves fine details and reduces noise in the original DDIM process while preserving nearly perfect inversion.Larger boosting intervals reduce preservation, while tboost is otherwise not sensitive.
  • Quality boosting also improves image quality in the Asyrp process.The reported degradation source is the DDIM process rather than Asyrp.
  • Attribute-specific coefficients govern training, with λCLIP values near 0.8 suitable for most in-domain attributes.For unseen domains, higher coefficients produce more noticeable changes; λrecon is set to CLIP similarity ∗3.
  • Random sampling can replace real-image inversion during training, trading content preservation against larger possible edits.This is used for attributes where identity preservation is not important.

K.1 USER STUDY

The user study compares Asyrp and DiffusionCLIP through quality, attribute naturalness, diversity, and overall preference, alongside directional and segmentation metrics. Asyrp shows stronger semantic editing, while DiffusionCLIP retains advantages in holistic style and structure-preserving scores.

  • User-study design: The evaluation covers in-domain and unseen-domain CelebA-HQ attributes plus department-store, ancient, and wooden LSUN-church styles.Smiling and sad are in-domain; Pixar and Neanderthal are unseen-domain attributes.
  • Qualitative comparison: DiffusionCLIP performs better on holistic style changes but shows weaker semantic changes, noisier results, and less diversity than Asyrp.The paper attributes these problems to fine-tuning the whole diffusion model.
  • User-study design: The survey evaluates image quality, natural attribute changes, diversity, and overall preference for CelebA-HQ and LSUN-church.The study uses groups of images for LSUN-church and compares methods under multiple criteria.
  • Quantitative evaluation: Asyrp outperforms DiffusionCLIP on directional CLIP similarity for all evaluated attributes.The comparison uses 100 samples per attribute and evaluates both CelebA-HQ and LSUN-church settings.
  • Quantitative evaluation: DiffusionCLIP achieves better or competitive segmentation-consistency scores, partly because it changes texture or color while preserving structure and shape.The paper notes that higher segmentation consistency does not guarantee better qualitative editing.
  • Quantitative evaluation: Asyrp permits desirable structural changes for attributes such as smiling, which can lower segmentation consistency despite improving semantic edits.Figure 28 illustrates example segmentations used to compute segmentation consistency.

L.2 COMPARE THREE METHODS

The paper compares implicit neural directions, timestep-specific optimized directions, and an aggregated global direction. The implicit direction offers a practical balance, while timestep-specific directions can preserve quality and global directions may introduce small appearance changes.

  • Method comparison: The comparison includes implicit neural direction ft, optimized timestep directions ∆ht, and optimized global direction ∆hglobal.The directions represent alternative ways to parameterize semantic edits.
  • Method comparison: Optimizing ∆ht requires separate optimization and hyperparameters at every timestep, including higher learning rates for larger t.Training ft takes a similar amount of time to optimizing ∆hglobal.
  • Method comparison: ft and ∆ht produce the best quality, whereas ∆hglobal can cause slight differences in hair and other details.The comparison identifies a quality trade-off between timestep-specific and global directions.
  • Method comparison: ∆ht can be obtained from ft, while ∆hglobal can be obtained by aggregating timestep-specific directions.The paper chooses ft because of the stated practical advantages.
  • Additional experiments: Asyrp can generate conditional samples from random Gaussian noise, although this experiment is outside the paper’s main scope.The results are presented as uncurated random sampling.
  • Additional experiments: ImageNet experiments verify that Asyrp can edit models trained on large datasets, but the latent space is not partitioned simply by object classes.Different forms of an orange require different latent representations.
  • Additional experiments: Mixed interpolation across multiple attributes is possible, including combined attribute shifts such as smiling and young.The paper reports that interpolation with any attribute is possible.

N.3 MORE RESULTS ON ALL DATASETS

Additional experiments show Asyrp operating across multiple datasets and supporting combinations of semantic attributes. The reported examples extend beyond the primary CelebA-HQ and LSUN-church results.

  • Additional dataset results: Additional results are provided for CelebA-HQ, LSUN-church, AFHQ, LSUN-bedroom, and METFACES.These examples cover the datasets listed in the paper’s extended qualitative results.
  • Additional dataset results: The paper supplies further qualitative examples on CelebA-HQ and LSUN-church alongside AFHQ, LSUN-bedroom, and METFACES.The examples are presented in Figures 34–36.
  • Attribute combinations: Asyrp supports combined interpolation of smiling and young attributes.Figure 33 reports mixed interpolation between multiple attributes.
Loading 2210.10960v2…