Source-linked AI summary
Inversion-Based Style Transfer with Diffusion Models
Yuxin Zhang, Nisha Huang, Fan Tang, Haibin Huang, Chongyang Ma, Weiming Dong, Changsheng Xu
TL;DR
Existing artistic image-generation methods struggle to transfer high-level elements of a painting or require detailed text that cannot fully describe its uniqueness. InST learns a painting’s style as a textual embedding from one image and guides diffusion generation, with experiments reporting superior image-to-image and text-to-image results across varied artistic settings. Its color transfer remains limited when content and reference colors differ substantially.
Problem
Prior methods struggle to transfer object shapes and semantic elements, while diffusion generation may require detailed text that cannot capture a specific painting’s key idea.
Method
InST uses attention-based textual inversion to learn high-level textual embeddings from a single painting and conditions a diffusion generator with the learned representation.
Results
InST achieves superior image-to-image and text-to-image generation results compared with state-of-the-art approaches across numerous paintings, artists, and styles.
Takeaways & Limitations
The method supports artistic image synthesis that transfers a painting’s appearance while retaining content control through text descriptions.
Takeaways & Limitations
Color transfer may fail to match content and reference colors semantically when their colors differ substantially.
Abstract
from arXiv · showhide
The artistic style within a painting is the means of expression, which includes not only the painting material, colors, and brushstrokes, but also the high-level attributes including semantic elements, object shapes, etc. Previous arbitrary example-guided artistic image generation methods often fail to control shape changes or convey elements. The pre-trained text-to-image synthesis diffusion probabilistic models have achieved remarkable quality, but it often requires extensive textual descriptions to accurately portray attributes of a particular painting. We believe that the uniqueness of an artwork lies precisely in the fact that it cannot be adequately explained with normal language. Our key idea is to learn artistic style directly from a single painting and then guide the synthesis without providing complex textual descriptions. Specifically, we assume style as a learnable textual description of a painting. We propose an inversion-based style transfer method (InST), which can efficiently and accurately learn the key information of an image, thus capturing and transferring the artistic style of a painting. We demonstrate the quality and efficiency of our method on numerous paintings of various artists and styles. Code and models are available at https://github.com/zyxElsa/InST.
1. Introduction
InST addresses the difficulty of transferring a painting’s high-level artistic attributes while avoiding the need for extensive textual descriptions. It learns style from a single painting and uses that representation to generate diverse images with controllable content.
- Motivation: Painting style includes materials, colors, brushstrokes, semantic elements, and object shapes that are difficult to describe with ordinary language.These factors motivate example-guided artistic image generation from a favorite painting.
- Limitations of prior methods: Existing style-transfer methods often struggle to transfer object shapes and semantic elements from an example painting.Text-guided methods instead rely on prompts, which may not capture a specific painting’s key idea.
- InST framework: InST learns a painting’s style from a single input image and transfers it to a natural image using a simple text prompt.The learned style includes material, brushstrokes, colors, object shapes, and semantic elements while preserving diversity.
- InST framework: An attention-based textual inversion module uses a painting’s CLIP image embedding to learn textual embeddings that encode key image information.Stochastic inversion is also used to maintain the semantic content of the input image.
- Results: Experiments across numerous paintings, artists, and styles report improved visual quality and artistic consistency over state-of-the-art approaches.The authors characterize the method as general, precise, and adaptable for image-to-image and text-guided generation.
2. Related Work
Prior work uses feature statistics, attention, text guidance, or diffusion inversion for artistic generation, but important limitations remain in transferring creative elements and learning concepts from few examples. InST instead learns an artistic representation from a single image and conditions diffusion generation through inversion.
- Image style transfer: Traditional and arbitrary style-transfer methods primarily model feature patches or statistics to transfer colors and brushstrokes.Recent approaches use deep features, attention, reversible flows, contrastive learning, or transformer-based composition.
- Image style transfer: Style-transfer methods have difficulty transferring artistic creativity factors such as object shape and decorative elements.This limitation distinguishes low-level style matching from broader artistic concept transfer.
- Text-to-image synthesis: Text-to-image systems generate artistic images from prompts, while CLIP-based methods align image and text representations for style-related generation.These systems depend on text guidance and may not capture artistic characteristics that ordinary text cannot describe.
- Inversion of diffusion models: Diffusion inversion seeks noise maps and conditioning vectors corresponding to images, but maintaining fidelity when generating new instances remains difficult.Some inversion methods require fine-tuning or optimization and may demand multiple example images.
- InST framework: InST learns a textual embedding from a single image and uses it to guide artistic generation without fine-tuning the generative model.Its framework combines inversion with a diffusion backbone and targets artistic style rather than only object concepts.
3. Method
InST learns a painting’s style as a learnable textual representation and uses diffusion inversion to transfer it while preserving content. Its method combines multi-layer cross-attention for style embedding with stochastic inversion for content-preserving noise initialization.
- Overview: InST uses a pre-trained diffusion model as both an inverted backbone and a generator for image-to-image and text-to-image synthesis.During inference, the reference image produces a textual embedding that guides generation from the content image.
- Textual Inversion: The method represents a painting with a learnable “new word” embedding obtained from its CLIP image representation rather than normal-language text.A placeholder token is associated with a learnable vector in textual space to encode artistic information that ordinary language may not express.
- Textual Inversion: Directly optimizing a single-image embedding is inefficient and risks inaccurate representations through overfitting.This motivates the proposed multi-layer cross-attention learning strategy, which extracts key image information more efficiently.
- Textual Inversion: Multi-layer cross-attention transforms CLIP image embeddings into the textual embedding used to condition the diffusion generator.The attention module computes query, key, and value projections and updates the representation through an attention operation.
- Stochastic Inversion: Stochastic inversion separates holistic style information in text conditions from detail information controlled by random noise.It predicts noise from the content image’s latent code and uses that noise as the initial generation input to preserve content semantics.
4. Experiments
Experiments compare InST with style-transfer and text-guided methods, then examine efficiency, editability, content preservation, and design choices. Results show stronger transfer of artistic semantics and techniques, while ablations clarify how stochastic inversion, Strength, attention, and dropout affect outcomes.
- Comparisons with Style-Transfer Methods: InST better transfers reference-image semantics and artistic techniques, including object shapes and distinctive elements that traditional style-transfer methods struggle to reproduce.Examples include facial forms, eyes, mountains, suns, stars, flower headwear, and roadsters.
- Comparison with Textual Inversion: About 1000 iterations are sufficient for InST to optimize toward the target text embedding, whereas Textual Inversion usually requires 10 times as many iterations.The comparison attributes the efficiency difference to InST’s attention-based inversion versus Textual Inversion’s simple optimization-based scheme.
- Comparison with Textual Inversion: InST outperforms Textual Inversion in accuracy and editability and remains more robust when additional semantic descriptions are introduced.Textual Inversion trained on a single image loses adaptability to added text and cannot preserve the specific artistic visual effect.
- Comparison with Text-Guided Methods: InST outperforms SDM in accuracy and better captures unique artistic attributes that are difficult to describe using only human-written captions.The qualitative comparison evaluates SDM conditioned on human captions and reports more accurate representation of the target style with InST.
- Ablation Study: Stochastic inversion preserves content information, while larger Strength values increase style-image influence and smaller values keep outputs closer to the content image.Without stochastic inversion, buildings can become trees or mountains, indicating stronger semantic interference from the style image.
- Ablation Study: Multi-layer attention improves editability by repeatedly interacting with CLIP embeddings, and dropout improves accuracy and editability by reducing overfitting.Dropout is applied in the attention module’s linear layer to prevent overfitting.
5. Conclusion
InST learns high-level textual descriptions from a single painting and uses them to guide text-to-image generation with a specific artistic appearance.
- InST learns high-level textual descriptions from a single painting to guide text-to-image generation.The learned description represents the painting’s artistic style.
- The framework uses attention-based textual inversion to obtain textual embeddings for the painting.This leverages aligned text and image feature spaces from CLIP.
- Extensive experiments report superior image-to-image and text-to-image generation results compared with state-of-the-art approaches.