PaperlayerView exact revision on arXiv

Source-linked AI summary

DEADiff: An Efficient Stylization Diffusion Model with Disentangled Representations

Tianhao Qi, Shancheng Fang, Yanze Wu, Hongtao Xie, Jiawei Liu, Lang Chen, Qian He, Yongdong Zhang

arXiv:2403.06951v2cs.CV

TL;DR

Encoder-based stylization can impair text controllability because reference features mix style and semantics, and reconstruction training can favor the reference image. DEADiff separates style and content representations and their cross-attention injection, and trains with paired images; the paper reports an optimal balance between style transfer and text control.

  • Problem

    Encoder-based reference-style transfer can diminish text controllability because reference features mix style with semantics and reconstruction training can lead the model to neglect text conditions.

    Source: §1 · pp. 1–2

  • Method

    DEADiff uses style- and content-instructed Q-Formers, injects their decoupled representations into separate cross-attention subsets, and trains them with paired images rather than identical reconstruction targets.

    Source: §1 · p. 2

  • Results

    DEADiff is reported to imitate reference style while following varied prompts, with an optimal equilibrium between stylization capability and text control.

    Source: §1 · p. 2, §5 · p. 8

  • Takeaways & Limitations

    DEADiff eliminates the need to manually adjust feature-fusion weights typically required by methods such as T2I-Adapter.

    Source: §1 · p. 2

Abstract

from arXiv · show

The diffusion-based text-to-image model harbors immense potential in transferring reference style. However, current encoder-based approaches significantly impair the text controllability of text-to-image models while transferring styles. In this paper, we introduce DEADiff to address this issue using the following two strategies: 1) a mechanism to decouple the style and semantics of reference images. The decoupled feature representations are first extracted by Q-Formers which are instructed by different text descriptions. Then they are injected into mutually exclusive subsets of cross-attention layers for better disentanglement. 2) A non-reconstructive learning method. The Q-Formers are trained using paired images rather than the identical target, in which the reference image and the ground-truth image are with the same style or semantics. We show that DEADiff attains the best visual stylization results and optimal balance between the text controllability inherent in the text-to-image model and style similarity to the reference image, as demonstrated both quantitatively and qualitatively. Our project page is https://tianhao-qi.github.io/DEADiff/.

1. Introduction

Encoder-based style transfer can weaken text control because reference features mix style and semantics and reconstruction training favors the reference image. DEADiff separates style and semantic representations and their injection, using paired-image training to retain prompt fidelity while transferring style.

1. Introduction

  • Encoder-based methods efficiently inject reference-image features to guide style transfer, but their mixed style and semantic information can conflict with text semantics and weaken prompt control.Reconstruction training can further encourage the model to focus on the reference image rather than the original text condition.Source: §1 · pp. 1–2
  • DEADiff uses Q-Formers instructed by style and content descriptions, then injects their decoupled representations into mutually exclusive cross-attention layer subsets.Its non-reconstruction training uses paired images sharing the reference style for style representations and paired images sharing semantics for content representations.Source: §1 · p. 2
  • DEADiff imitates reference styles while following varied text prompts, is more efficient than optimization-based methods, and preserves text control better than traditional encoder-based methods.It also removes the need to manually adjust feature-fusion weights typically required by methods such as T2I-Adapter.Source: §1 · p. 2
  • The paper identifies three contributions: dual-decoupled representation extraction, disentangled conditioning across cross-attention layers, and two paired datasets for non-reconstruction training.These address semantic conflict from both the learning-task and model-structure perspectives.Source: §1 · p. 2

2. Related Work

The related work traces stylized text-to-image generation from diffusion models to optimization-based and encoder-based methods. It highlights trade-offs in style fidelity, text fidelity, and per-reference computation.

2.1. Diffusion-based Text-to-Image Generation

  • Diffusion models restore target data distributions disrupted by forward diffusion; text-to-image systems use U-Net cross-attention for text features, while latent diffusion enables efficient high-resolution generation.The section identifies DPMs [26], initial diffusion image-generation works [4] [8] [28], later text-to-image models [21] [22] [25], and LDMs, also known as Stable Diffusion [22].Source: §2.1 · p. 2

2.2. Stylized Image Generation with T2I Models

  • Textual inversion projects a style image into a learnable text-token embedding, while DreamBooth and Custom Diffusion can better capture style but reduce prompt fidelity through overfitting.Parameter-efficient methods including InST, LoRA, and StyleDrop preserve prompt fidelity, but require minutes to hours of fine-tuning per reference image.Source: §2.2 · p. 3
  • Optimization-free encoders extract reference features for U-Net cross-attention or text embedding, but reconstruction objectives capture both content and style; shuffling methods target style extraction.T2I-Adapter-Style and IP-Adapter use Transformer image encoders, while BLIP-Diffusion uses a Q-Former; StyleAdapter and ControlNet-shuffle shuffle image patches or pixels.Source: §2.2 · p. 3

3. Method

DEADiff learns style and content representations from paired images, injects them into separate cross-attention layers, and trains the Q-Formers and added projection layers.

3.1. Preliminary

  • Stable Diffusion encodes images into latents, denoises them through a U-Net conditioned on text, then maps denoised latents back to pixels.The training objective supervises denoising, with ε sampled from a standard Gaussian distribution.Source: §3.1 · p. 3

3.2. Dual Decoupling Representation Extraction

  • Style queries learn from distinct same-style reference and target images, while content queries learn from same-subject images with different styles; a complementary reconstruction task retains image information.The style task conditions generation on the extracted style representation and the target content caption; the content task combines content queries with target style words.Source: §3.2 · pp. 3–4

3.3. Disentangled Conditioning Mechanism

  • The Disentangled Conditioning Mechanism assigns semantics to coarse, low-resolution layers and style to fine, high-resolution layers, using a joint text-image attention operation.Text and image keys and values are concatenated before a single attention operation with U-Net query features.Source: §3.3 · p. 4

3.4. Paired Datasets Construction

  • The paired datasets are generated from nearly 12,000 subject words and nearly 700 style words, yielding 1.06 million image-text pairs and task-specific image pairings.The style task uses over 160,000 items; the content task uses 1.06 million items.Source: §3.4 · pp. 4–5

3.5. Training and Inference.

  • Training supervises three learning tasks with the stated loss, optimizing only the Q-Formers and newly added linear projection layers.The inference process is illustrated in Figure 2(b).Source: §3.5 · p. 5

4. Experiment

Experiments compare DEADiff with state-of-the-art methods, examine its components, and demonstrate applications. The reported results show a balance of text fidelity and style similarity, with image quality and user preference advantages over comparison methods.

4.1. Experiment Settings

  • Experiments use Stable Diffusion v1.5 with 16 cross-attention layers, evaluating 32 style images and 35 text prompts across style similarity, text alignment, image quality, and subjective preference.Layers 4–8 inject content representations; the other layers inject style representations.Source: §4.1 · p. 5

4.2. Comparison with State-of-the-Arts

  • DEADiff achieves the highest style similarity except against untuned T2I-Adapter and IP-Adapter, comparable text alignment to CAST and StyTr2, higher image quality than all other methods, and greater user preference.Against StyleDrop, it is slightly inferior in color accuracy but comparable or better in artistic style and text fidelity.Source: §4.2 · pp. 6–7, Table 1 · p. 6, Figure 4 · p. 6

4.3. Ablation Study

  • Ablations show that disentangled conditioning improves text alignment but leaves semantic conflict; non-reconstructive STRE reduces reference-content leakage, and STRE with SERE yields strong text control and style replication.SERE alone has limited impact; its combination with STRE helps decouple the two representations without omissions.Source: §4.3 · p. 7, Table 2 · p. 7

4.4. Applications

  • DEADiff supports stylization with depth ControlNet while maintaining layout, stylizes reference semantics through text prompts, mixes multiple reference styles, and switches between base text-to-image models.The semantic stylization is reported as superior to IP-Adapter.Source: Figure 7 · p. 7, §4.4 · p. 8, Figure 8 · p. 8

5. Conclusion

DEADiff targets text-control losses in encoder-based stylized diffusion through disentangled representations and reports a balance between stylization and text control.

5. Conclusion

  • DEADiff combines dual decoupling representation extraction with a disentangled conditioning mechanism to address text-control decline in encoder-based stylized diffusion.The paper identifies these as targeted designs for the problem it analyzes.Source: §5 · p. 8
  • Empirical evidence indicates DEADiff reaches an optimal equilibrium between stylization capability and text control.The conclusion frames this as the method's overall empirical finding.Source: §5 · p. 8
  • Future work could further improve style similarity and decouple instance-level semantic information.Source: §5 · p. 8

6. Supplementary

The supplementary evaluations report DEADiff's balance of style similarity and text alignment, favorable user-study ratings, and comparisons across efficiency, ControlNet, customization, and additional examples.

6.1. Quantitative Comparisons

  • T2I-Adapter at condition weight 1.0 scores 0.241 in style similarity and 0.224 in text alignment, while reducing the weight drops style similarity to 0.184; DEADiff lies outside its trade-off curve.The paper also reports that DEADiff outperforms CAST, StyTr2, and InST in both style similarity and text alignment, with an image-quality advantage over all compared methods.Source: Figure 11 · p. 11, §6.1 · p. 11, Table 3 · p. 12

6.4. Comparison with ControlNet 1.1 Shuffle

  • Against ControlNet 1.1 Shuffle, DEADiff performs better in reference-style carving, text fidelity, and generated image quality.The comparison is presented visually in Figure 12.Source: Figure 12 · p. 11, §6.4 · p. 12

6.2. User Study

  • In a 24-user study with 18 reference styles and 21 prompts, DEADiff outperforms all state-of-the-art methods on text-image alignment, image quality, style similarity, and overall preference.The study collected 2016 votes.Source: §6.2 · p. 11, Table 4 · p. 12

6.3. Inference Efficiency

  • Adding 1900 MB of memory causes only a marginal increase in average inference time on one A100-80G GPU.The cited comparison reports memory usage and sampling time.Source: §6.3 · p. 12, Table 5 · p. 12

6.5. Combination with DreamBooth/LoRA

  • Because the original U-Net parameters remain frozen, DEADiff is compatible with DreamBooth/LoRA for combining subject control with style control.Figure 13 illustrates the combination using a dog and a cat as customized subjects.Source: §6.5 · p. 12, Figure 13 · p. 12

6.6. More Examples

  • Additional visual examples show DEADiff imitating reference styles while following text-prompt instructions.The paper presents these examples to demonstrate effectiveness and universality.Source: §6.6 · p. 12, Figure 14 · p. 13

Related papers

Loading 2403.06951v2…