PaperlayerView exact revision on arXiv

Source-linked AI summary

Style Injection in Diffusion: A Training-free Approach for Adapting Large-scale Diffusion Models for Style Transfer

Jiwoo Chung, Sangeek Hyun, Jae-Pil Heo

arXiv:2312.09008v2cs.CV

TL;DR

Diffusion style-transfer methods can require costly optimization or fail to exploit large-scale models. This paper introduces optimization-free self-attention style injection with key-value substitution, plus controls for content and color, and reports state-of-the-art results against prior methods.

  • Problem

    Existing diffusion-based style-transfer methods may require time-consuming optimization or fail to leverage widely used large-scale diffusion models.

  • Method

    The method transfers style without optimization by substituting content self-attention keys and values with those from a style image, then adds query preservation, temperature scaling, and initial latent AdaIN.

  • Results

    The proposed method significantly outperforms previous methods and achieves state-of-the-art performance on the style-transfer dataset.

  • Takeaways & Limitations

    Self-attention feature manipulation provides an optimization-free way to combine style textures with content semantics and spatial layout in large-scale diffusion models.

  • Takeaways & Limitations

    The method relies on the hypothesis that initial latent-noise statistics substantially affect generated colors and brightness.

Abstract

from arXiv · show

Despite the impressive generative capabilities of diffusion models, existing diffusion model-based style transfer methods require inference-stage optimization (e.g. fine-tuning or textual inversion of style) which is time-consuming, or fails to leverage the generative ability of large-scale diffusion models. To address these issues, we introduce a novel artistic style transfer method based on a pre-trained large-scale diffusion model without any optimization. Specifically, we manipulate the features of self-attention layers as the way the cross-attention mechanism works; in the generation process, substituting the key and value of content with those of style image. This approach provides several desirable characteristics for style transfer including 1) preservation of content by transferring similar styles into similar image patches and 2) transfer of style based on similarity of local texture (e.g. edge) between content and style images. Furthermore, we introduce query preservation and attention temperature scaling to mitigate the issue of disruption of original content, and initial latent Adaptive Instance Normalization (AdaIN) to deal with the disharmonious color (failure to transfer the colors of style). Our experimental results demonstrate that our proposed method surpasses state-of-the-art methods in both conventional and diffusion-based style transfer baselines.

1. Introduction

The paper presents a training-free style-transfer method for large-scale diffusion models by replacing self-attention keys and values from content with those from style images. Query preservation, attention temperature scaling, and initial latent AdaIN address content disruption and color mismatch, while experiments report state-of-the-art performance.

  • Motivation: Existing diffusion-based style-transfer methods often require time-consuming optimization, while DiffStyle is difficult to apply to widely used Latent Diffusion Models.These limitations motivate extending training-free style transfer to large-scale pretrained diffusion models.
  • Motivation: Self-attention queries preserve related content patches by assigning them similar attention patterns to corresponding style features.Query features reflect patch similarities, helping maintain content relationships during style transfer.
  • Motivation: Large-scale diffusion features match content queries with style keys that share local texture and semantics, enabling texture transfer based on similarities such as edges.The paper specifically highlights high similarity between edge queries in content and edge-like keys in style images.
  • Method: The proposed method injects style by substituting content self-attention keys and values with those of the style image, especially in later decoder layers associated with local textures.The manipulation requires neither optimization nor supervision such as text.
  • Method: Query preservation, attention temperature scaling, and initial latent AdaIN improve style injection by mitigating content disruption and correcting disharmonious colors.These components are introduced as extensions to the basic self-attention manipulation approach.
  • Results: Extensive experiments on a style-transfer dataset report that the method significantly outperforms previous methods and achieves state-of-the-art performance.The introduction states this result without providing quantitative values.

2. Related Work

Diffusion-based neural style transfer has used pretrained models through textual inversion, fine-tuning, text conditioning, or training-free feature manipulation. The paper positions its method as an optimization-free approach that merges style and content within self-attention while preserving content structure.

  • Diffusion Model-based Neural Style Transfer: Neural style transfer modifies a content image to adopt a style image while retaining the original content.Diffusion models extend this example-guided image-generation task by leveraging pretrained generative models.
  • Diffusion Model-based Neural Style Transfer: Prior diffusion methods use textual inversion, fine-tuning with style-disentanglement losses, or text inputs as style or content conditions.These approaches differ in whether style is represented through textual embeddings, optimized model parameters, or text conditioning.
  • Diffusion Model-based Neural Style Transfer: DiffStyle provides training-free style transfer through h-space and skip connections, but changes both textures and semantics when applied to Stable Diffusion.The reported semantic changes include alterations to spatial layout.
  • Diffusion Model-based Neural Style Transfer: The paper proposes harmoniously merging style and content features within Stable Diffusion self-attention without optimization.This directly addresses the limitations reported for earlier training-free transfer on Stable Diffusion.
  • Diffusion Model-based Neural Style Transfer: Unlike text-focused editing methods, the proposed approach conditions on separate style and content images and adjusts intermediate feature statistics during transfer.The paper describes this combination as transferring texture while using the other image for content and spatial information.

3. Background

Latent Diffusion Models train and generate in a compressed latent space, reducing computation while preserving semantic information. Stable Diffusion uses a U-Net whose resolution blocks combine residual, self-attention, and cross-attention modules.

  • Latent Diffusion Models: Latent Diffusion Models encode images into lower-dimensional latent representations before diffusion training and decode generated latents back into images.This focuses modeling on semantic information while reducing computation costs.
  • Latent-space training: The diffusion model is trained to predict noise added to a noised latent at a uniformly sampled timestep, conditioned on y.The objective uses noise prediction over latent-space examples.
  • Stable Diffusion: Stable Diffusion uses a U-Net in which each resolution block sequentially contains a residual block, self-attention block, and cross-attention block.The paper focuses on the self-attention block for style transfer.
  • Stable Diffusion: The self-attention computation projects features into query, key, and value representations, with d denoting the projected query dimension.The model uses an empty text prompt rather than text conditioning in this work.

4. Method

The method performs training-free style transfer by injecting style-image self-attention features into content generation while preserving content queries. Initial latent AdaIN addresses color transfer, and temperature scaling sharpens attention weakened by style substitution.

  • 4. Method: Artistic style transfer modifies a content image to adopt a style image while maintaining the content image’s semantic information.The method operates on encoded image representations during diffusion.
  • 4. Method: The framework inverts content and style images, initializes stylized noise through latent-statistics processing, and applies feature injection during reverse diffusion.The reverse process combines content and style information through self-attention operations.
  • 4.1. Attention-based Style Injection: Style injection replaces the stylized latent’s self-attention keys and values with style-image keys and values, especially in decoder layers associated with local textures.This transfers style features while retaining the content generation pathway.
  • 4.1. Attention-based Style Injection: Query preservation blends stylized and content queries throughout reverse diffusion, with higher γ retaining more content and lower γ strengthening style transfer.The method applies this operation to the latter decoder layers relevant to local textures.
  • 4.2. Attention Temperature Scaling: Replacing keys with style features lowers query-key similarity, reducing attention-map variation and making post-softmax attention overly smooth.The paper identifies this smoothing as detrimental to sharp outputs.
  • 4.2. Attention Temperature Scaling: Temperature scaling multiplies pre-softmax attention by τ > 1 to sharpen the attention map; τ = 1.5 restores its standard deviation near the original value.The default τ is the average ratio measured over all timesteps.
  • 4.3. Initial Latent AdaIN: Self-attention injection transfers textures but often preserves the content image’s color tone, motivating initial latent AdaIN for color harmonization.Adding style queries, keys, and values still leaves the content color tone largely unchanged.

5. Experiments

Experiments evaluate the method on standard content-style datasets against conventional and diffusion-based baselines using content-, style-, and combined-fidelity measures. Results consistently favor the proposed method, while ablations and additional analyses examine its components, style-content control, temperature scaling, and text-guided comparisons.

  • Evaluation Protocol: ArtFID combines LPIPS-based content fidelity and FID-based style fidelity, while CFSD measures spatial correlation to reduce texture bias in content evaluation.CFSD is computed from patch-feature correlation maps and their KL-divergence.
  • Experimental Setup: Experiments use Stable Diffusion 1.4 with DDIM sampling and evaluate 800 stylized images formed from MS-COCO content and WikiArt style images.Images are center-cropped to 512 × 512; default hyperparameters are γ = 0.75 and τ = 1.5.
  • Quantitative Comparison: The proposed method surpasses conventional methods on ArtFID, FID, CFSD, and LPIPS, including the lowest reported FID and much lower CFSD.The comparison covers nine conventional and three diffusion-based style transfer baselines.
  • Quantitative Comparison: The method achieves the best LPIPS, FID, and ArtFID performance among diffusion-based style transfer methods, with an inference time of 12.4 seconds per content-style pair.The reported time includes 8.2 seconds for DDIM inversion and 4.2 seconds for sampling on a TITAN RTX GPU.
  • Qualitative Comparison: Qualitative comparisons show strong content-structure preservation alongside style transfer, whereas baselines often lose structure or fail to transfer style.The paper highlights bridge-structure preservation and provides additional zoomed comparisons.
  • Ablation and Analysis: Style injection guides style and content, initial latent AdaIN transfers color tone, and temperature scaling sharpens details and resolves blurriness.Increasing γ favors style at the expense of content, while increasing τ improves style-transfer performance with diminishing gains.
  • Additional Analysis: Adjusting γ lets users control the style-content trade-off, and the method outperforms text-guided methods because textual conditions omit style-image texture and color information.The γ sweep reports superiority across ranges of content and style fidelity.

6. Conclusion

The paper presents a training-free style-transfer method that adapts a pre-trained large-scale diffusion model by replacing self-attention keys and values with those from a style image. Query preservation, attention temperature scaling, and initial latent AdaIN address content disruption and disharmonious color, and experiments report superiority over state-of-the-art baselines.

  • The method enables training-free style transfer in a pre-trained large-scale diffusion model by substituting self-attention keys and values from the style image.
  • Query preservation and attention temperature scaling mitigate content disruption, while initial latent AdaIN addresses disharmonious color.
  • Experimental results show superiority over state-of-the-art techniques in conventional and diffusion-based style-transfer baselines.

7. Appendix

The appendix evaluates color transfer, attention temperature scaling, query preservation, text-prompt inversion, and comparisons against conventional and diffusion-based baselines. Results support improved color-tone transfer, sharper local textures, content-aligned stylization, and favorable user preferences.

  • Color transfer: Initial Latent AdaIN prominently improves transfer of the style image’s color tone, while each proposed component contributes to color transfer.The ablation evaluates color transfer using RGB-uv histograms and Histogram Loss based on Hellinger distance.
  • Attention temperature scaling: Attention temperature scaling prevents blurry results, preserves local style textures, and helps synthesize sharp images.The reported experiment uses γ = 0.3.
  • Quantitative evaluation: On a newly sampled set of 20 contents and 40 styles, the proposed method retains its performance enhancement, supporting hyperparameter generalization.The samples have no overlap with the original images.
  • Query preservation: Interpolated stylized queries remain near in-distribution content features, indicating query preservation maintains nearby content representations during style injection.For t=[10, 20, 30, 40], reported distances and nearest-neighbor counts support proximity to in-distribution nearby content.
  • User study: Users preferred the proposed method over AesPA-Net and InST by 57.2% and 76.7%, respectively, and the method has faster inference than InST.The study involved 18 users and 10 examples per user.
  • Qualitative comparison: Qualitative comparisons show better local-texture transfer than baselines, while StyleDiffusion more often changes image structure significantly.The comparisons include diffusion baselines, AesPA-Net, AdaAttN, and StyleDiffusion.
Loading 2312.09008v2…