Source-linked AI summary
Style Aligned Image Generation via Shared Attention
Amir Hertz, Andrey Voynov, Shlomi Fruchter, Daniel Cohen-Or
TL;DR
T2I models often interpret shared style descriptions inconsistently, while existing remedies require costly optimization and human-supported content–style disentanglement. StyleAligned uses minimal attention sharing during diffusion, and its evaluations report style-consistent, high-quality images faithful to prompts and reference styles without optimization or fine-tuning. The paper also identifies limits in full attention sharing and current diffusion inversion methods.
Problem
T2I models often diverge in their interpretations of the same stylistic descriptor, while existing fine-tuning approaches are computationally expensive and usually require human input to disentangle content and style.
Method
StyleAligned shares minimal attention operations during diffusion to align style across generated images and uses diffusion inversion to condition style-consistent generation on a reference image.
Results
Across diverse styles and text prompts, StyleAligned demonstrates high-quality synthesis and fidelity to prompts and reference styles while operating without optimization or fine-tuning.
Takeaways & Limitations
StyleAligned establishes style consistency and visual coherence across generated images while preserving diverse structures and content.
Takeaways & Limitations
Full attention sharing can cause content leakage and reduce set diversity, while current diffusion inversion methods limit some applications.
Abstract
from arXiv · showhide
Large-scale Text-to-Image (T2I) models have rapidly gained prominence across creative fields, generating visually compelling outputs from textual prompts. However, controlling these models to ensure consistent style remains challenging, with existing methods necessitating fine-tuning and manual intervention to disentangle content and style. In this paper, we introduce StyleAligned, a novel technique designed to establish style alignment among a series of generated images. By employing minimal `attention sharing' during the diffusion process, our method maintains style consistency across images within T2I models. This approach allows for the creation of style-consistent images using a reference style through a straightforward inversion operation. Our method's evaluation across diverse styles and text prompts demonstrates high-quality synthesis and fidelity, underscoring its efficacy in achieving consistent style across various inputs.
1. Introduction
Large-scale T2I models produce appealing text-aligned images, but often interpret shared style descriptions inconsistently across image sets. StyleAligned addresses this with minimal attention sharing, enabling style-consistent generation without optimization or fine-tuning.
- State-of-the-art T2I models often produce images that diverge in their interpretations of the same stylistic descriptor.
- Existing mitigation fine-tunes T2I models on style-sharing images, an expensive process that usually requires human input to disentangle content and style.
- StyleAligned enables consistent style interpretation across generated image sets using minimal attention sharing during diffusion, without optimization.
- Diffusion inversion extends StyleAligned to style-consistent generation from a reference style image without optimization or fine-tuning.
- Across diverse styles and text prompts, the method demonstrates high-quality synthesis and fidelity to prompts and reference styles.
2. Related Work
Related work spans text-to-image generation, attention control, style transfer, and T2I personalization. StyleAligned differs by targeting diverse structures and content while maintaining consistent style, without an optimization phase or several training images.
- Text-to-image generation: T2I diffusion models generate high-quality images from text and support tasks including inpainting, image-to-image translation, local editing, and subject-driven generation.
- Attention Control in diffusion models: Attention maps control generated layout and content, while prior attention-based methods have enhanced fidelity, diversity, or image editing.
- Style Transfer: Classic and deep-learning style-transfer methods optimize handcrafted or deep features, resample textures, or inject attention features from a style image.
- T2I Personalization: T2I personalization commonly optimizes over small collections of concept images, with style personalization relying on fine-tuned lightweight adapter layers.
- T2I Personalization: StyleAligned generates consistent image sets without an optimization phase or several training images, unlike personalization approaches focused on reproducing the input subject.
3. Method overview
StyleAligned generates image sets with consistent style by sharing self-attention with a reference during diffusion, while limiting sharing to preserve content diversity. Query-key AdaIN balances attention between target and reference features.
- StyleAligned diffusion: StyleAligned shares self-attention from a reference image to target images during each diffusion denoising step.The method operates within self-attention layers of attention-based T2I diffusion models.
- Style-aligned image sets: The method aims to generate images aligned with separate text prompts while maintaining a consistent style interpretation across the set.A shared style description alone can produce unaligned images because each image lacks awareness of the others during generation.
- Ablation study: Full attention sharing can cause content leakage and reduced diversity, whereas the complete method preserves diversity between sets and consistency within each set.The ablation compares full sharing, removal of query-key AdaIN, and the full configuration qualitatively and quantitatively.
- Restricted attention sharing: Target features attend to themselves and to one reference image, rather than sharing attention across every image in the set.This design restricts content leakage and supports diverse sets, although reference-to-target attention flow can remain low without further adjustment.
- Query-key AdaIN: Query-key AdaIN normalizes target queries and keys using the corresponding reference queries and keys before shared attention.The shared-attention layer updates target features using both target values Vt and reference values Vr.
4. Evaluations and Experiments
The evaluation measures object-text alignment and within-set style consistency, compares StyleAligned with variants and personalization methods, and examines controllability, reference-image transfer, and attention behavior.
- A four-image set requires 29 seconds on one A100 GPU when StyleAligned is applied across SDXL’s 70 self-attention layers.
- The evaluation uses CLIP cosine similarity for object-text alignment and pairwise DINO embedding cosine similarity for style consistency.
- StyleAligned achieves much higher style consistency than unmodified SDXL, at the expense of text similarity.
- Ablation Study: Full attention sharing further increases style consistency but lowers text alignment, while omitting AdaIN substantially lowers style consistency.
- Comparisons: Compared with personalization methods, StyleAligned produces more consistent style attributes while avoiding content leakage from the reference image.
- Additional Results: Reducing shared-attention layers produces more diverse sets while retaining common reference attributes, and DDIM inversion enables style alignment from an input image.
- Additional Results: Shared attention maps connect semantically related regions, such as bodies, heads, and backgrounds, rather than performing global style transfer.
5. Conclusions
StyleAligned uses minimal attention sharing with AdaIN modulation during diffusion to establish style consistency and visual coherence across generated images. The method produces high-quality images faithful to descriptions and reference styles across diverse prompts and styles, while scalability and control remain future directions.
- StyleAligned introduces minimal attention sharing with AdaIN modulation during diffusion to establish style consistency and visual coherence.
- The method produces high-quality, style-consistent images across diverse styles and textual prompts while adhering to descriptions and reference styles.
- Future work includes improving scalability, adaptability, and control over shape and appearance similarity among generated images.
- Current diffusion inversion limitations motivate assembling style-aligned datasets for training style-conditioned text-to-image models.
A. StyleAligned from an Input Image
StyleAligned transfers the style of a reference image to new content using inversion and text prompts, including cases where text alone fails to reproduce a famous painting’s style.
- For Peter Bruegel’s “The Tower of Babel,” text-only guidance fails to reproduce the original painting’s style across generated places.
- Style transfer from real reference examples is demonstrated in additional figures.
- Extremely famous reference images can cause the default style-transfer approach to generate content almost identical to the reference instead of following the target prompt.
B. Integration with Other Methods
StyleAligned extends style-consistent generation to personalized diffusion, MultiDiffusion panoramas, and ControlNet-conditioned outputs. These integrations use shared attention with a reference image or parallel reference generation.
- Integration Overview: The method is demonstrated on top of different diffusion-based image generation methods, including personalized models and MultiDiffusion.The supplementary material presents StyleAligned as an add-on capability rather than a replacement generator.
- Subject-Driven Generation: StyleAligned can be applied to personalized diffusion models by sharing attention between personalized images and a generated reference-style image.Personalized content uses LoRA weights during inference, while style alignment is applied during generation.
- MultiDiffusion: StyleAligned enables style-aligned panoramas with MultiDiffusion by sharing crop attention with a reference image generated in parallel.Compared with vanilla MultiDiffusion, the method also helps preserve style within each image.
- Additional Conditions: StyleAligned is combined with ControlNet to support additional conditioning inputs such as depth maps and pose.ControlNet adds conditioning through residual features in the diffusion U-Net, while StyleAligned supplies shared-attention operations.
C. Additional Comparisons
Additional comparisons examine StyleAligned against encoder-based personalization and diffusion-based editing approaches. The comparisons focus on text alignment and set consistency, while noting that editing baselines assume input-structure preservation.
- Comparison Scope: Additional comparisons cover encoder-based personalization methods and editing approaches over the evaluation set used in the main paper.The supplementary material summarizes the full quantitative results in Table 2.
- Encoder Based Approaches: StyleAligned image sets are reported as more consistent and better aligned to the reference than encoder-based approaches in qualitative comparisons.The comparison includes BLIP-Diffusion, ELITE, and IP-Adapter; BLIP-Diffusion and ELITE also struggle to match text descriptions consistently.
- Quantitative Comparison: The quantitative comparison evaluates text alignment with CLIP score and set consistency with DINO embedding similarity.The full results report standard deviations across 100 image-set results.
- Zero Shot Editing Approaches: Diffusion editing baselines such as SDEdit and Prompt-to-Prompt assume preservation of the input image’s structure.These methods trade off alignment to the text against alignment to the input image.
- Zero Shot Editing Approaches: The compared method can achieve text alignment while preserving high set consistency.This statement summarizes the reported comparison outcome without supplying an unprovided metric value.
D. User Study and Evaluation Settings
Evaluation uses diverse prompts and image examples to assess style alignment and text alignment across methods. The study includes automatic metrics, user judgments, and qualitative demonstrations across personalization, editing, and conditioning settings.
- Evaluation Settings: The evaluation set uses 100 text prompts, each describing four objects in the same style.Generated image sets are evaluated with automatic CLIP and DINO scores and through user evaluation.
- Qualitative Results: Qualitative examples cover style transfer across varied images and target prompts.Figures 17 and 18 show samples of the proposed style-transfer techniques.
- Qualitative Results: Personalized diffusion examples apply reference styles to personalized content using papercut art and flat-design prompts.Figure 19 shows results across personalized diffusion models fine-tuned over personalized content.
- Qualitative Results: MultiDiffusion panoramas, ControlNet depth, and ControlNet pose are included among the demonstrated applications.The corresponding figures present style-aligned outputs for each setting.
- Comparative Evaluation: A qualitative comparison evaluates StyleAligned against encoder-based personalization methods.Figure 23 provides the comparison visualization.
- User Study: The user study asks participants to choose which of two method outputs is better for style alignment and text alignment.Figure 24 depicts the comparison format using rows of images produced by different methods.
- Evaluation Settings: The evaluation materials include prompts and reference-image examples used to construct the comparison sets.The supplementary prompt list and reference-image passages document examples such as objects, scenes, and people.