PaperlayerView exact revision on arXiv

Source-linked AI summary

Visual Style Prompting with Swapping Self-Attention

Jaeseok Jeong, Junho Kim, Yunjey Choi, Gayoung Lee, Youngjung Uh

arXiv:2402.12974v2cs.CV

TL;DR

Text-to-image diffusion models make exact style control difficult, and existing reference-image approaches can require costly fine-tuning or suffer content leakage. This paper swaps reference keys and values into late self-attention layers while retaining original queries, without fine-tuning. The authors report that the method outperforms existing approaches in style prompting and text-prompt matching.

  • Problem

    Text-to-image diffusion models can vary in style for the same prompt, while existing reference-image methods may require costly fine-tuning or inadequately transfer visual elements due to content leakage.

    Source: §1 · p. 1

  • Method

    The method retains original queries and swaps in reference-image keys and values in later self-attention layers, without fine-tuning.

    Source: §1 · pp. 1–2

  • Results

    The authors report that visual style prompting qualitatively and quantitatively outperforms existing methods.

    Source: §5 · p. 8

  • Takeaways & Limitations

    Visual style prompting is compatible with ControlNet and Dreambooth-LoRA, allowing more specific control of results.

    Source: §3.2 · p. 6

    The method cannot synthesize content that pretrained diffusion models cannot generate, and output quality depends on those models’ performance.

    Source: §5 · p. 8

Abstract

from arXiv · show

In the evolving domain of text-to-image generation, diffusion models have emerged as powerful tools in content creation. Despite their remarkable capability, existing models still face challenges in achieving controlled generation with a consistent style, requiring costly fine-tuning or often inadequately transferring the visual elements due to content leakage. To address these challenges, we propose a novel approach, \ours, to produce a diverse range of images while maintaining specific style elements and nuances. During the denoising process, we keep the query from original features while swapping the key and value with those from reference features in the late self-attention layers. This approach allows for the visual style prompting without any fine-tuning, ensuring that generated images maintain a faithful style. Through extensive evaluation across various styles and text prompts, our method demonstrates superiority over existing approaches, best reflecting the style of the references and ensuring that resulting images match the text prompts most accurately. Our project page is available https://curryjung.github.io/VisualStylePrompt/.

1. Introduction

Text-to-image diffusion models can produce varied results from the same prompt, making exact style control difficult. The paper proposes training-free visual style prompting that uses a reference image to guide style while text specifies content.

1. Introduction

  • Text-to-image diffusion models produce varied images for a single prompt, and even detailed descriptions cannot fully constrain the resulting style.The passage attributes variation in part to random-noise sampling and discrepancies between textual and visual modalities.Source: §1 · p. 1
  • Reference images can specify visual elements that text descriptions cannot fully convey, providing a way to guide style more directly.The method uses the reference as a visual prompt while the text prompt describes desired image content.Source: §1 · p. 1
  • Prior reference-image approaches use fine-tuning, new text embeddings, or adapted cross-attention, and often face costly tuning or content leakage.They may also inaccurately reflect the intended style.Source: §1 · p. 1
  • Swapping reference keys and values into late self-attention layers while retaining original queries guides style and helps alleviate content leakage.The method requires no fine-tuning and is described as versatile across text-to-image diffusion models.Source: §1 · pp. 1–2
  • The paper evaluates its approach across styles and text prompts, including style fidelity, text-prompt reflection, content leakage, and output diversity.It also reports applying the method to existing techniques and using real reference images.Source: §1 · p. 2

2. Visual style prompting

Text prompts can specify style ambiguously, while visual style prompting uses a reference image to guide style without fine-tuning. It keeps queries from the original process and swaps in reference keys and values in selected self-attention layers.

2.1. Discussion on attention modules in T2I diffusion

  • Self-attention uses spatial features for queries, keys, and values, so borrowing reference keys and values can transfer visual elements that are difficult to describe in text.The method keeps queries from the original features and swaps in keys and values from the reference process.Source: §2.1 · pp. 2–3
  • Text prompts can be ambiguous about style, and prior reference-image approaches use fine-tuning, new embeddings, or cross-attention adapters with limits on the styles they can reflect.Cross-attention approaches are limited to styles paired with text prompts during training.Source: §2.1 · p. 2

2.2. Swapping self-attention

  • The method swaps reference-process keys and values into the original denoising process for T steps, aiming to render original content with reference style elements.The figure depicts the repeated swap across the two denoising processes.Source: Figure 3 · p. 3, §2.2 · p. 3
  • Applying swaps across all self-attention layers can let reference-image content influence the result, such as cat-like output despite a dog prompt.The approach therefore examines where in the network to apply the swap.Source: §2.2 · p. 3

2.3. Choosing blocks for swapping self-attention

  • Swapping bottleneck blocks can avoid reference-content leakage but produce disrupted images; the method instead swaps only upblock keys and values.The authors conjecture that unclear content layout in downblocks can make feature substitution disruptive.Source: §2.3 · pp. 3–4
  • With the swap endpoint fixed, changing its start point yields abrupt changes in style similarity, text alignment, diversity, and content leakage; the selected point balances these metrics.The analysis uses four metrics to choose the starting point.Source: §2.3 · p. 4, Figure 5 · p. 4
  • Late-upblock attention focuses more on corresponding regions and can reassemble texture and color, while early-upblock attention covers wider areas with different styles.The comparison links late-upblock swapping to more dog-like results and early-upblock swapping to some cat-like attributes.Source: §2.3 · p. 4, Figure 6 · p. 4

3. Experiments

The experiments compare visual style prompting with competing methods on style fidelity, text alignment, content leakage, and diversity. A user study and qualitative and quantitative comparisons favor the proposed method, while competitors show trade-offs between style similarity and prompt alignment.

3. Experiments

  • The evaluation compares StyleAlign, IP-Adapter, Dreambooth-LoRA, and StyleDrop using 720 images and DINO style similarity, CLIP text alignment, and LPIPS diversity.The images span 40 reference images, 120 content prompts, and six initial noises; experiments use SDXL and also validate on Stable Diffusion v1.5.Source: §3 · pp. 4–5, §3.1 · p. 5

3.1. Comparison against competitors

  • Figure 8 compares CLIP text alignment and DINO style similarity; IP-Adapter has higher style similarity but seriously neglects text prompts.The figure encodes other methods as blue points and our method as an orange point.Source: Figure 8 · p. 5, §3.1 · p. 6
  • Qualitative comparisons show our method uses reference style while preserving prompt-specified content; competitors often introduce unreferenced textures or leak reference content.Examples of leakage include retaining a layout, a screaming person, or a castle from the reference.Source: §3.1 · p. 5, Figure 7 · p. 5
  • In a 62-participant user study, the majority rated our method best at reflecting reference style and text-prompt content.Participants compared six synthesized images per method across 20 sets.Source: §3.1 · p. 6, Table 1 · p. 6
  • Across different initial noises, our results show varied poses and viewpoints, whereas other methods barely change and limit the pretrained model’s diversity.Figure 9 provides the qualitative comparison across the same style.Source: §3.1 · p. 6, Figure 9 · p. 6

3.2. Combining existing techniques

Visual style prompting can be combined with ControlNet and Dreambooth-LoRA, and can use real images as style references through diffusion inversion. The real-image experiments use stochastic inversion, while the authors identify better inversion methods as future work.

3.2. Combining existing techniques

  • Visual style prompting is compatible with ControlNet and Dreambooth-LoRA, allowing more specific control of results.Figure 13 illustrates its use with these techniques.Source: §3.2 · p. 6, Figure 11 · p. 7

3.3. Real image as a reference image

  • Real images can serve as style references after inversion through a diffusion process, while the generated image retains the text-prompt content.The reference denoising process uses painting names or simple text descriptions as conditioning.Source: §3.3 · p. 7
  • The authors empirically choose stochastic inversion over DDIM inversion and identify better-compatible real-image inversion methods as future work.Source: §3.3 · p. 7

4. Related work

Prior customization methods use training, embedding optimization, or adapted cross-attention, and may face content leakage. Related work also uses visual prompts and self-attention features for other forms of diffusion-model control.

4.1. Customization of T2I diffusion models

  • DreamBooth and LoRA fine-tune pretrained models, textual inversion learns embeddings, and adapter methods convert visual features into cross-attention embeddings; these approaches require additional training.The paper notes that per-style or per-model training impedes practical use.Source: §4.1 · p. 7

4.2. Content leakage in style customization

  • Customization methods can leak reference-image content; StyleAlign shares keys and values across processes and uses AdaIN preprocessing, which the authors suspect may reduce text alignment.Figure 12 illustrates style reflection with minimal content changes, while Figure 13 shows compatibility with ControlNet and Dreambooth-LoRA.Source: §4.2 · pp. 7–8, Figure 12 · p. 7, Figure 13 · p. 8

4.3. Controlling diffusion models with visual prompt

  • Visual-prompt control includes ControlNet for structural content guidance and image-analogy methods, which require paired original and edited images.Source: §4.3 · p. 8

4.4. Using self-attention features in diffusion models

  • Prior work uses self-attention features to preserve image structure or identity and improve temporal consistency; Figure 14 shows style prompting with real-image references after inversion.Source: §4.4 · p. 8, Figure 14 · p. 8

5. Conclusion and Limitation

The paper concludes that swapping self-attention supports training-free style prompting and outperforms existing methods qualitatively and quantitatively. Its scope depends on pretrained-model capability, and strong reference-style adherence can suppress text-specified style.

5. Conclusion and Limitation

  • The authors report that visual style prompting qualitatively and quantitatively outperforms existing methods, with self-attention preferred to cross-attention and a measured way to choose the swapping range.Source: §5 · p. 8
  • Generation remains limited to what the pretrained diffusion model can produce, and output quality depends on that model’s performance.Source: §5 · p. 8
  • When both sources specify style, strong adherence to the reference tends to make the method ignore the style specified in text.Source: §5 · p. 8
  • The authors identify compatible real-image inversion and extension to domains such as video as future research directions.Source: §5 · p. 8

6. Impact statements

The impact statement raises concerns about copyright infringement and misrepresentation of original artworks, and calls for responsible-use guidelines.

6. Impact statements

  • Generative models raise ethical concerns about copyright infringement and misrepresentation of original artworks.The statement also says their societal impact on cultural perceptions and artistic expression requires consideration and calls for ethical guidelines and regulations.Source: §6 · p. 9
  • The statement identifies cultural perceptions and artistic expression as areas potentially affected by generative models.It calls for establishing ethical guidelines and regulations for responsible use.Source: §6 · p. 9

A.1. Style-Content prompt list

The prompt list pairs varied requested content with visual styles, including Hokusai-inspired art and realistic fire photography.

A.1. Style-Content prompt list

  • A Hokusai-style prompt requests a book, cup, or tree, while a realistic fire-photography prompt requests a dragon, ghost mask, or bird.The latter also specifies a black background.Source: §A.1 · p. 12

4. The persistence of memory in style of Salvador Dali: (1) table (2) ball (3) flower

The prompt list combines a range of styles with varied requested subjects, including surrealist, mosaic, pop-art, and other visual treatments.

4. The persistence of memory in style of Salvador Dali: (1) table (2) ball (3) flower

  • Prompts under this heading include pop-art, woodcut, post-modern, Chinese ink, and abstract-impressionist styles paired with varied subjects.Examples include a compass in woodcut prints, a horse in colorful Chinese ink, and a piano in abstract impressionism.Source: §4 · p. 12

10. A teapot in mosaic art style.: (1) A kangaroo (2) A skyscraper (3) A lighthouse

  • This prompt group pairs mosaic, glitch-art, graffiti, cartoon-line, and other styles with requested subjects such as a kangaroo, robot, or flower.It also includes prompts for realistic cloud photography and pointillism.Source: §10 · p. 12

20. The scream in Edvard Munch style: (1) A rabbit (2) a horse (3) a giraffe

  • This group includes prompts pairing requested subjects with Vermeer, bokeh-photography, low-poly, fluid-art, anime, mixed-media, and Klimt styles.Examples include a wild flower in bokeh photography style and a cactus in mixed-media arts style.Source: §20 · p. 12

28. Horseshoe in vector illustrations style.: (1) Vintage typewriter (2) Snail (3) Tornado

  • This group pairs subjects such as a dog, puppet, and flowers with play-doh, neo-futurist, and watercolor treatments.Other listed styles include vector illustration, medieval fantasy, origami, and vintage still-life photography.Source: §28 · pp. 12–13

Related papers

Loading 2402.12974v2…