PaperlayerView exact revision on arXiv

Source-linked AI summary

Uncovering the Disentanglement Capability in Text-to-Image Diffusion Models

Qiucheng Wu, Yujian Liu, Handong Zhao, Ajinkya Kale, Trung Bui, Tong Yu, Zhe Lin, Yang Zhang, Shiyu Chang

arXiv:2212.08698v1cs.CV

TL;DR

Image editing seeks changes to target attributes without unwanted changes to image content, while existing approaches may require fine-tuning or auxiliary inputs. This paper uses partial text-embedding changes and optimized mixing weights in stable diffusion, reporting broad attribute disentanglement and performance that matches or exceeds fine-tuned diffusion-editing baselines.

  • Problem

    Image editing needs attribute changes that preserve semantic content, while many existing methods require fine-tuning or auxiliary inputs.

    Source: §1 · p. 2, §2 · p. 3

  • Method

    The method partially replaces a neutral text embedding with a styled one and optimizes their combination weights for disentangled editing.

    Source: §1 · p. 2, §3.3 · p. 4

  • Results

    Stable diffusion disentangles a wide range of attributes, and the proposed method matches or outperforms diffusion-editing baselines requiring fine-tuning.

    Source: §5 · p. 8

  • Takeaways & Limitations

    Stable diffusion’s inherent disentanglement can support generalizable image editing with around 50 optimized parameters and no model fine-tuning.

    Source: §5 · p. 8

    The model has difficulty disentangling attributes involving small objects, particularly when the target attribute correlates with other image parts.

    Source: §4.1 · p. 7

Abstract

from arXiv · show

Generative models have been widely studied in computer vision. Recently, diffusion models have drawn substantial attention due to the high quality of their generated images. A key desired property of image generative models is the ability to disentangle different attributes, which should enable modification towards a style without changing the semantic content, and the modification parameters should generalize to different images. Previous studies have found that generative adversarial networks (GANs) are inherently endowed with such disentanglement capability, so they can perform disentangled image editing without re-training or fine-tuning the network. In this work, we explore whether diffusion models are also inherently equipped with such a capability. Our finding is that for stable diffusion models, by partially changing the input text embedding from a neutral description (e.g., "a photo of person") to one with style (e.g., "a photo of person with smile") while fixing all the Gaussian random noises introduced during the denoising process, the generated images can be modified towards the target style without changing the semantic content. Based on this finding, we further propose a simple, light-weight image editing algorithm where the mixing weights of the two text embeddings are optimized for style matching and content preservation. This entire process only involves optimizing over around 50 parameters and does not fine-tune the diffusion model itself. Experiments show that the proposed method can modify a wide range of attributes, with the performance outperforming diffusion-model-based image-editing algorithms that require fine-tuning. The optimized weights generalize well to different images. Our code is publicly available at https://github.com/UCSB-NLP-Chang/DiffusionDisentanglement.

1. Introduction

The introduction asks whether stable diffusion models can disentangle attributes as GANs do, then finds that partial text-embedding changes can edit style while preserving semantic content. It presents a lightweight optimization method and reports broad attribute coverage, transfer across images, and competitive editing results.

1. Introduction

  • Small edits are harder to disentangle when the target attribute correlates with other image parts.Source: Table 1 · p. 2
  • Disentanglement should change one aspect while preserving others and transfer learned edits across images; whether diffusion models have this capability was an open question.Source: §1 · p. 2
  • With denoising noises fixed, partially changing a neutral text embedding toward a styled description shifts the image’s style without changing semantic content.The example changes “a photo of person” toward “a photo of person with smile.”Source: §1 · p. 2
  • The method optimizes two text-embedding mixing weights for content preservation and style matching, using around 50 parameters without fine-tuning the diffusion model.Source: §1 · p. 2
  • The proposed editing algorithm matches or outperforms diffusion-editing baselines that require fine-tuning.It optimizes the mixing weights of two text embeddings for style matching and content preservation.Source: §1 · p. 2
  • Stable Diffusion disentangles attributes from global painting styles to local facial expressions, and learned weights generalize across images.The introduction points to Table 1 for attribute coverage and Fig. 1 for image-pair examples.Source: §1 · p. 2

2. Related Works

Prior work explores latent-space disentanglement and diffusion-based editing, but some approaches require extensive fine-tuning or auxiliary inputs. The paper places its approach in text-embedding space and uses a fixed pretrained diffusion model.

Disentanglement in Generative Models

  • GAN disentanglement methods identify latent directions for attribute-only changes, using approaches including classifiers, PCA, contrastive learning, and information maximization.Source: §2 · p. 2
  • A prior diffusion method learns U-Net hidden-layer shifts but struggles with holistic styles; the paper uses text embeddings and reports comparable or better results with 1.2% of its parameters.Source: §2 · p. 3
  • Another approach learns an image-specific representation and requires retraining a diffusion model from scratch, whereas this work fixes the pretrained model.Source: §2 · p. 3

Diffusion Models

  • Text-to-image diffusion models take text descriptions as input and generate images that match them.Source: §2 · p. 3

Image Editing

  • Many image-editing methods require fine-tuning diffusion models or auxiliary inputs such as masks or multiple examples, while this work edits without either.Source: §2 · p. 3

3. Attribute Disentanglement in Stable Diffusion Models

The paper examines whether Stable Diffusion can separate style changes from semantic content, then develops text-embedding optimization for disentangled generation and image editing. Its approach uses fixed noise, learned mixing weights, and an inversion-based process for real-image editing.

3.1. Preliminaries on Diffusion Models

  • DDIM denoises initial Gaussian noise under stepwise text embeddings; setting variance to zero makes generation deterministic, and Stable Diffusion runs this process in hidden embedding space.The denoising network parameters are fixed in this setup.Source: §3.1 · p. 3

3.2. The Disentanglement Properties

  • With the same initial noise, later-step replacement of the neutral prompt with the smile prompt preserves identity while changing expression; full replacement changes identity.The paper reports consistent observations across other objects and styles.Source: §3.2 · p. 4

3.3. Optimizing for Disentanglement

  • The editor learns per-step soft weights between neutral and styled embeddings, balancing perceptual content preservation against directional CLIP style matching, with about 50 parameters and no model fine-tuning.Soft combinations include full and partial replacement as special cases.Source: §3.3 · p. 4

3.4. Extension to Image Editing

  • The learned text embedding can be applied directly to a new image to produce the same editing effect.The overview illustrates this transfer with a castle converted to a children’s drawing style.Source: Figure 3 · p. 5
  • For real-image editing, inversion estimates initial noise; correction terms close reconstruction error and remain fixed during editing, with re-diffusion adopted to further improve edited-image quality.The process first reconstructs the input using the neutral prompt.Source: §3.4 · p. 5

4. Experiments

Experiments examine stable diffusion’s inherent disentanglement, its performance in image editing, and its robustness to text-description changes. The results show broad capability and transfer, with limits on small-object edits and some prompt variations.

4.2. Evaluation on Disentangled Image Editing

  • Our method outperforms DIFFUSIONCLIP in almost all three evaluation aspects for 6 of 8 attributes, but is less competitive on “sketch” and “pixar”.The aspects are attribute similarity, content preservation, and overall quality.Source: Figure 5 · p. 7, §4.2 · p. 7

4.1. Exploring the Disentanglement Capability

  • Stable diffusion disentangles many global styles and local attributes, and learned weights transfer to unseen images with the same object and target attribute.The study also reports difficulty with attributes involving small objects, which can alter correlated attributes such as style or identity.Source: §4.1 · pp. 6–7, Figure 4 · p. 6

4.2. Evaluation on Disentangled Image Editing

  • The subjective comparison used 20 source images per dataset from CelebA and LSUN-church, with four edit types per dataset and three questions about editing quality.The main baseline was DIFFUSIONCLIP, which requires fine-tuning; subjects compared paired edits in randomized order.Source: §4.2 · p. 7
  • Against other diffusion editing baselines, our method maintains semantic details better than BLENDED-DIFFUSION, is relatively on par with DIFFUSIONCLIP and PROMPT-TO-PROMPT, and has competitive overall quality versus IMAGIC.Compared with IMAGIC, it may change slightly more semantic content.Source: §4.2 · pp. 7–8, Figure 6 · p. 8

4.3. Ablation Study

  • Changing how the target attribute is appended to c(1) had little effect, while adding modifiers could amplify edits and one modifier was sufficient.The tested c(1) variants used direct concatenation, a comma, or “with”.Source: §4.3 · p. 8
  • Editing remained largely consistent across most c(0) variants but failed with an image-captioning-model description; results were also reported as robust to variations in the images used to learn weights.The authors suggest that lengthy captions with more details may overwhelm the target attribute description.Source: §4.3 · p. 8

5. Conclusion

The conclusion reports that stable diffusion has inherent disentanglement capability, which can be used for image editing through optimized combinations of two text embeddings. The method uses 50 parameters and outperforms fine-tuning-based baselines on image editing.

5. Conclusion

  • Stable diffusion’s inherent disentanglement can be triggered by partially replacing a style-neutral text embedding with one describing the desired style.The proposed algorithm optimizes the combination weights for style matching and content preservation, using 50 parameters, and outperforms fine-tuning-based baselines on image editing.Source: §5 · p. 8

A. Implementation Details

Implementation uses pretrained models without changing their parameters. Loss balancing and combination-weight initialization differ between person and scene edits.

A. Implementation Details

  • For person edits, a larger perceptual-loss weight and an initialization favoring c(0) encourage stricter content preservation than for scene edits.The directional CLIP loss uses pretrained ViT-B/32, and perceptual loss uses pretrained VGG-16.Source: A. Implementation Details · p. 12
  • The experiments use pretrained stable-diffusion-v1-4, with all pretrained models left unchanged.The stable diffusion model was used without changing any parameters.Source: A. Implementation Details · p. 12

B. Text Descriptions Used for Attribute Disentanglement and Image Editing

Each target attribute or edit pairs a style-neutral description with one containing explicit styles, denoted c(0) and c(1). The method is described as robust to text-description choices beyond those listed.

C. Additional Examples on Partially Replacing Text Embeddings

Partially replacing the neutral text embedding with an explicit-style embedding during later denoising steps can preserve image content while modifying the target attribute. The examples also show that the best replacement timing varies by attribute, while the method applies to a broad range of attributes.

C. Additional Examples on Partially Replacing Text Embeddings

  • Later-step replacement of c(0) with c(1) can preserve neutral-image content while strengthening the target-attribute modification as more steps are replaced.Source: C. Additional Examples on Partially… · p. 12
  • The best replacement time t′ varies by attribute: t′ = 0.8T works for “red brick” and “in sunset” but not two other tested attributes.This motivates searching for an optimal combination of text embeddings.Source: C. Additional Examples on Partially… · p. 12

D. More Examples of Disentangled Attributes

E. Details of Subjective Evaluation

The subjective evaluation compared the method with fine-tuned DIFFUSIONCLIP checkpoints on face and church edits, using ratings of attribute match, naturalness, content preservation, and overall quality. The authors report that their method produced more natural images while modifying only the target object.

  • The evaluation used the first 20 source images from Celeb-A and LSUN-church for four edits per dataset, comparing against DIFFUSIONCLIP’s edit-specific fine-tuned checkpoints.Eleven Amazon Mechanical Turk master workers judged randomly ordered results.Source: E. Details of Subjective Evaluation · p. 12
  • Workers assessed target-attribute consistency and naturalness, preservation of original-image information, and overall editing quality.Source: E. Details of Subjective Evaluation · pp. 12, 14
  • The authors observed that DIFFUSIONCLIP sometimes over-changed attributes, introducing artifacts and altering other content, while their method produced more natural images and modified only the target object.Source: E. Details of Subjective Evaluation · p. 14

F. More Comparisons with Baselines

Qualitative comparisons found the method broadly comparable to baselines, with stronger, more natural edits for global attributes but difficulty disentangling small edits. Comparisons with some methods were limited by unavailable code or auxiliary-label requirements.

  • Some baselines could not be included in subjective evaluation because their code was unavailable or they required unavailable auxiliary labels.Source: F. More Comparisons with Baselines · p. 14
  • The method achieved comparable qualitative results overall, with stronger and more natural edits for global attributes but difficulty disentangling small edits such as cake decorations.Source: F. More Comparisons with Baselines · p. 15
  • The comparison with DIFFUSIONCLIP was blurry because its source-paper inputs were low resolution.The paper directs readers to Figures 10 and 11 for a more comprehensive DIFFUSIONCLIP comparison.Source: F. More Comparisons with Baselines · p. 15

G. Effects of Varying Text Descriptions

Edits were largely robust to variations in neutral descriptions and in how target attributes were appended, although caption-generated descriptions sometimes weakened the effect. Adding correlated modifiers could strengthen the target effect.

The way of appending target attributes in c(1)

  • Direct, comma-separated, and preposition-linked target descriptions produced desired attributes across people, buildings, and natural scenes.The tested forms were not intended as an exhaustive list of possible concatenation styles.Source: G. Effects of Varying Text Descriptions · p. 15

More complex target attribute description in c(1)

The variations of c(0)

  • Editing was largely robust to c(0) variants, but caption-generated descriptions sometimes weakened the target effect when their details outweighed it.The authors suggest that long captions containing many details may outweigh the target attribute description.Source: G. Effects of Varying Text Descriptions · p. 18

H. Effects of Varying Optimization Images

Weights learned from different optimization images produced similar transfers to unseen images. Optimizing on more images improved disentanglement, especially preservation of identity cues.

Choice of optimization image

Number of optimization images

Loading 2212.08698v1…