Source-linked AI summary
StyleDrop: Text-to-Image Generation in Any Style
Kihyuk Sohn, Nataniel Ruiz, Kimin Lee, Daniel Castro Chin, Irina Blok, Huiwen Chang, Jarred Barber, Lu Jiang, Glenn Entis, Yuanzhen Li, Yuan Hao, Irfan Essa, Michael Rubinstein, Dilip Krishnan
TL;DR
Text-to-image models struggle to reproduce nuanced styles that are ambiguous or difficult to express in language. StyleDrop uses Muse, adapter tuning, and iterative human or automated feedback to learn a style from very few images. It achieves strong style consistency while retaining text-to-image capability, including with a single style reference image, though single-image tuning can leak content and the comparison with diffusion models is not conclusive.
Problem
Natural-language prompts do not reliably specify nuanced visual styles involving design patterns, textures, materials, color schemes, and shading.
Method
StyleDrop tunes adapters on Muse using style-reference text construction and iterative training with CLIP or human feedback.
Results
StyleDrop achieves competitive Text scores and higher Style scores than the reported Muse baseline, while supporting style tuning from one image and outperforming compared methods qualitatively and quantitatively.
Takeaways & Limitations
StyleDrop provides efficient, flexible style customization that can preserve text-to-image generation capability and extend to independently combined content and style personalization.
Takeaways & Limitations
Single-image style tuning can produce sub-optimal quality because of overfitting and content leakage, and the superiority of vision transformers over diffusion models remains inconclusive.
Abstract
from arXiv · showhide
Pre-trained large text-to-image models synthesize impressive images with an appropriate use of text prompts. However, ambiguities inherent in natural language and out-of-distribution effects make it hard to synthesize image styles, that leverage a specific design pattern, texture or material. In this paper, we introduce StyleDrop, a method that enables the synthesis of images that faithfully follow a specific style using a text-to-image model. The proposed method is extremely versatile and captures nuances and details of a user-provided style, such as color schemes, shading, design patterns, and local and global effects. It efficiently learns a new style by fine-tuning very few trainable parameters (less than $1\%$ of total model parameters) and improving the quality via iterative training with either human or automated feedback. Better yet, StyleDrop is able to deliver impressive results even when the user supplies only a single image that specifies the desired style. An extensive study shows that, for the task of style tuning text-to-image models, StyleDrop implemented on Muse convincingly outperforms other methods, including DreamBooth and textual inversion on Imagen or Stable Diffusion. More results are available at our project website: https://styledrop.github.io
1 Introduction
StyleDrop addresses the difficulty of expressing nuanced visual styles through text by tuning a text-to-image model from very few style references. It combines Muse, adapter tuning, and iterative feedback, and is evaluated against competing personalization methods.
- StyleDrop targets styles that are difficult to describe with text because of nuanced color schemes, illumination, and other visual characteristics.
- StyleDrop combines a transformer-based text-to-image model, adapter tuning, and iterative training with feedback.
- The method can tune stylized text-to-image synthesis using as few as one image representing the desired style.
- StyleDrop extends style customization to content customization by combining independently trained content and style adapters.
- Evaluation compares StyleDrop on Muse with DreamBooth and Textual Inversion using Imagen and Stable Diffusion backbones, with prompt and style fidelity metrics plus a user study.
2 Related Work
The related work covers personalization methods, neural style transfer, and parameter-efficient fine-tuning. StyleDrop differs by using a generative vision transformer for text-conditioned style synthesis across diverse visual styles.
- Textual inversion and PEZ learn image-associated text representations without changing the parameters of the text-to-image model.
- DreamBooth fine-tunes an entire text-to-image model on a few subject images, while LoRA and adapter tuning improve fine-tuning efficiency.
- Unlike diffusion-based personalization methods, StyleDrop is built on Muse, a generative vision transformer.
- Prior diffusion-based style-learning methods use handfuls or dozens of reference images and are limited to painting styles, whereas StyleDrop covers diverse visual styles with one image.
- Neural Style Transfer optimizes style and content consistency to produce stylized images, whereas StyleDrop uses text-to-image models to generate content.
- Parameter-efficient fine-tuning updates far fewer parameters than full-model fine-tuning and has been applied to text-to-image diffusion models and vision transformers.
3 StyleDrop: Style Tuning for Text-to-Image Synthesis
StyleDrop tunes a Muse-based text-to-image transformer using parameter-efficient adapters and prompts that separate content from style. Iterative feedback training then reduces content leakage and improves the balance between style and text fidelity, while independently trained adapters can combine personalized content and style.
- Overview: StyleDrop uses Muse with parameter-efficient fine-tuning and iterative training with feedback as its two key components.The framework also supports sampling from two fine-tuned models.
- Muse backbone: Muse generates images through transformer layers that process visual tokens conditioned on text embeddings.Its modules include a text encoder, transformer, sampler, image encoder, and decoder; the super-resolution module is omitted here.
- Parameter-efficient fine-tuning: The framework can use prompt tuning, LoRA, or adapter tuning while keeping the encoder, decoder, and text encoder fixed and adapting the transformer.Existing transformer parameters remain fixed while newly introduced parameters θ are learned.
- Constructing text prompts: Training prompts combine content descriptions with style descriptors so learned parameters model style while disentangling it from content.For example, a content description such as “cat” is followed by a style descriptor such as “watercolor painting.”
- Iterative training with feedback: Single-image style tuning can produce sub-optimal quality because the model may overfit to reference content and leak it into generated images.The paper illustrates leakage through repeated rendering of a reference house in the background.
- Iterative training with feedback: Iterative training selects successful synthesized image-text pairs for further training, reducing content leakage and improving style-text balance.Selection can use CLIP feedback or human feedback; CLIP improves textual fidelity without losing too much style fidelity, while human feedback can capture subtle style properties.
- Sampling from two adapters: Two independently trained adapters can be sampled together to personalize both content and style without jointly optimizing content and style images.The method combines a style adapter and a content adapter during generation.
4 Experiments
Experiments evaluate StyleDrop across diverse visual styles, compare it with existing methods, and study iterative feedback, compositional personalization, data efficiency, and style-property editing. Results show strong style consistency, efficient learning from limited data, and flexible control over content and style.
- Experimental Setting: Experiments use diverse references spanning watercolor, oil painting, flat illustrations, 3D rendering, sculptures, and varied materials.The study collects a few dozen images and evaluates StyleDrop across a broad range of visual styles.
- StyleDrop Results: StyleDrop captures texture, shading, and structure across 18 styles and provides more control over style than previous approaches.Figure 1 uses 18 style images with the same text prompt, while comparisons include DreamBooth, LoRA DreamBooth, and textual inversion.
- Quantitative Results: StyleDrop preserves text alignment while improving visual style alignment, reaching Text 0.323 versus Muse 0.322 and Style 0.694 versus 0.556.Iterative training improves Text scores but can reduce Style scores because synthetic-image selection may drift from the reference style.
- Quantitative Results: StyleDrop on Muse achieves stronger style consistency across prompts and references than DreamBooth on Imagen, which reaches Style 0.644 versus StyleDrop’s 0.694.The comparison reports a larger Style-score increase for StyleDrop, from 0.556 to 0.694, than for DreamBooth on Imagen, from 0.569 to 0.644.
- My Object in My Style: Two independently trained adapters can combine personalized content and style, producing images that match both the reference subject and the desired style.StyleDrop alone captures style but can generate objects inconsistent with reference subjects; the combined sampling method addresses both.
- Comparative Study of DreamBooth on Imagen and StyleDrop on Muse: Imagen benefits from 10 synthetic images and a descriptive style descriptor, whereas rare-token training shows no notable improvement with more images.The comparison suggests StyleDrop’s advantage on Muse is partly due to its fine-tuning data efficiency.
- Style Property Edit with Concept Disentanglement: Descriptive style descriptors enable selective editing, such as removing “melting” while retaining “golden 3d rendering” characteristics.This provides an additional control over style properties that is less available with rare-token training.
- Iterative Training: Iterative training reduces concept leakage and improves Text scores, although Style scores can be misleading when generated concepts remain semantically mixed.The reported Text scores are 0.303, 0.322, 0.339, and 0.328 across strategies, while Style scores range from 0.542 to 0.567.
5 Conclusion
StyleDrop enables stylized text-to-image synthesis from as little as one style image while using fewer than 1% trainable parameters. The paper notes broader style coverage and model comparisons as open directions, alongside potential misuse in reproducing artists’ styles without consent.
- StyleDrop enables synthesis in a user-provided style from as little as one style image and a text description.
- Training requires fewer than 1% of the model parameters to be learnable.
- A broader study of visual styles spanning formal attributes, media, history, eras, and art movements remains future work.
- The superiority of generative vision transformers over diffusion models for few-shot style transfer is not conclusive.
- StyleDrop may improve creative productivity while enabling unauthorized copying of individual artists’ styles.
B.1.1 Adapter Architecture
StyleDrop applies parameter-efficient adapters throughout the transformer and can share adapter weights across layers. The architecture and synthesis settings are tuned to balance convergence, parameter count, and style preservation.
- Adapters are inserted after both the cross-attention and MLP blocks at every transformer layer.Up weights are initialized to zero, while down weights use a truncated normal distribution with standard deviation 0.02.
- Weight sharing across transformer layers reduces the number of adapter parameters by roughly the number of layers.Sharing is enabled by setting is_shared to True.
- A learning rate above 0.00003 for the base model often overfits the style reference image’s content.
- Descriptive style descriptors are not required but add capabilities such as style editing.
B.2 Details on Human Evaluation
The human evaluation compares StyleDrop and baselines through binary preference judgments on style matching and text alignment. The study uses repeated ratings across styles, and individual-vote analysis confirms the majority-vote trend despite additional ties.
- Three binary comparison tasks evaluate DreamBooth versus StyleDrop and two StyleDrop training variants.The comparisons cover Round 1, human-feedback iterative training, and CLIP-feedback iterative training.
- 300 queries across six styles are each rated independently by five raters, yielding 4500 answers.
- Raters compare generated images on both style matching to a reference image and fit to the reference text.
- Individual vote counts contain more ties than majority voting reports, but preserve the overall trend.
B.3.1 Classifier-Free Guidance
Classifier-free guidance separates style adaptation from text-prompt conditioning, allowing StyleDrop to trade off style fidelity and text fidelity. Excessive guidance in either direction can make one factor dominate generation.
- λA controls style adaptation, while λB controls text-prompt fidelity.
- Increasing λA improves style consistency until excessive values make reference-image content dominate and text conditioning collapse.
- Increasing λB improves text fidelity but eventually replaces the learned style with a more generic Muse style.
- Fixing λB = 5.0 and varying λA is sufficient for most style–text fidelity trade-offs.
B.4 Extended Baseline Comparison
The extended comparison evaluates StyleDrop against multiple style-tuning baselines, user-feedback variants, and guidance settings across qualitative and preference-study evidence. It also examines performance across diverse style references and reports that PEZ does not outperform the compared methods.
- User preference study: Human preference evaluation uses binary image comparisons, with five raters independently assessing alignment to the style reference or text prompt.The positions of the two images are randomized for each query.
- User preference study: The preference-study comparisons include StyleDrop Round 1 against DreamBooth on Imagen and against a human-feedback StyleDrop variant.The displayed comparisons record ties in the supplied results text.
- Guidance-scale ablation: Guidance-scale ablations vary λA for style adaptation and λB for text-prompt conditioning, revealing content leakage for StyleDrop (R1) under unfavorable settings.Large λA or small λB can produce content leakage, whereas StyleDrop (HF) is described as robust to guidance-scale changes.
- Qualitative comparison: Additional no-cherry-picking comparisons place DreamBooth, StyleDrop rounds, and feedback variants side by side for six reference styles.The compared feedback variants include human feedback and CLIP-score feedback.
- Baseline comparison: StyleDrop is compared with DreamBooth, Textual Inversion, and PEZ across Imagen and Stable Diffusion baselines.The comparison includes DreamBooth on Imagen, LoRA DreamBooth and Textual Inversion on Stable Diffusion, and Hard Prompt Made Easy (PEZ).