Source-linked AI summary
Implicit Style-Content Separation using B-LoRA
Yarden Frenkel, Yael Vinker, Ariel Shamir, Daniel Cohen-Or
TL;DR
Image stylization must change appearance while preserving image content, but style and content are strongly coupled and existing fine-tuning approaches can overfit. B-LoRA jointly learns two SDXL block-specific LoRA adapters from one image, separating style and content for flexible stylization. The resulting components support style transfer, text-guided manipulation, and consistent style-conditioned generation without additional training or fine-tuning.
Problem
Style and content are strongly connected in image stylization, making it difficult to transform appearance while preserving content and structure.
Method
B-LoRA jointly optimizes LoRA weights in two specific SDXL transformer blocks from a single image to separate its style and content.
Results
The two B-LoRAs support style transfer, text-guided style manipulation, and consistent style-conditioned image generation without additional training or fine-tuning.
Takeaways & Limitations
B-LoRAs can be used as independent components for multiple stylization tasks and re-plugged into pretrained models with other learned blocks without further training.
Takeaways & Limitations
Color may be included in the style component, single-image styles may capture background elements, and complex scenes may compromise content preservation.
Abstract
from arXiv · showhide
Image stylization involves manipulating the visual appearance and texture (style) of an image while preserving its underlying objects, structures, and concepts (content). The separation of style and content is essential for manipulating the image's style independently from its content, ensuring a harmonious and visually pleasing result. Achieving this separation requires a deep understanding of both the visual and semantic characteristics of images, often necessitating the training of specialized models or employing heavy optimization. In this paper, we introduce B-LoRA, a method that leverages LoRA (Low-Rank Adaptation) to implicitly separate the style and content components of a single image, facilitating various image stylization tasks. By analyzing the architecture of SDXL combined with LoRA, we find that jointly learning the LoRA weights of two specific blocks (referred to as B-LoRAs) achieves style-content separation that cannot be achieved by training each B-LoRA independently. Consolidating the training into only two blocks and separating style and content allows for significantly improving style manipulation and overcoming overfitting issues often associated with model fine-tuning. Once trained, the two B-LoRAs can be used as independent components to allow various image stylization tasks, including image style transfer, text-based image stylization, consistent style generation, and style-content mixing.
1. Introduction
B-LoRA addresses the style-content entanglement that makes image stylization difficult by separating both components from a single image. Jointly trained adapters in two SDXL transformer blocks support flexible stylization while reducing overfitting and additional training requirements.
- Image stylization must alter visual features such as colors and textures while preserving semantic content and structure, creating a trade-off between transformation and preservation.
- B-LoRA separates style and content from any given image and distills them to support multiple style manipulation applications.
- The method fits a lightweight adapter that separates style and content per image, reducing overfitting while supporting text-based and reference-image style conditions.
- B-LoRA jointly optimizes LoRA weights in two SDXL transformer blocks to reconstruct a single input image from a text prompt; each block contains 10 consecutive attention layers.
- The two B-LoRAs can be used independently for style transfer, text-guided manipulation, and consistent style-conditioned image generation without additional training or fine-tuning.
- The method is presented as a practical alternative that provides advantages over approaches designed for individual stylization tasks and works with existing models.
2. Related Work
Prior stylization methods use neural features, generative models, attention manipulation, or personalization to control appearance, but often require domain data, fine-tuning, or separate style-content handling. B-LoRA instead learns separate style and content representations within SDXL.
- Style Transfer: Neural style transfer merges pretrained content and style features, while GAN-based domain-transfer methods require paired or unpaired domain-specific datasets and training.
- Style Transfer: Zero-shot diffusion methods manipulate generation through attention or spatial features, but some lack explicit content control and can leak structure from the style image.
- Style Transfer: StyleAligned aligns attention features across generated images, whereas B-LoRA decomposes style and content and learns a separate representation for each.
- Text-to-Image Personalization: Text-to-image personalization extends pretrained models to generate novel style or content concepts from small sets of images, using token optimization, fine-tuning, or both.
- LoRA for Image Stylization: LoRA-based stylization commonly trains models on image sets and combines them with control methods, while ZipLoRA adds optimization for each new style-content combination.
3. Preliminaries
SDXL is a latent diffusion model whose UNet uses cross- and self-attention across transformer blocks, with text conditions injected through cross-attention. LoRA adapts such models by learning low-rank residuals while freezing pretrained weights.
- SDXL Architecture: SDXL applies diffusion in the latent space of a pretrained autoencoder and uses a UNet with 70 attention layers divided into 11 transformer blocks.
- SDXL Architecture: SDXL encodes each text prompt with OpenCLIP and CLIP, concatenates the embeddings, and feeds the resulting conditioning into cross-attention layers.
- SDXL Architecture: To analyze a transformer block, the method injects an alternate prompt into that block while using the original prompt in all other blocks.
- SDXL Architecture: Cross-attention projects spatial features into queries and text embeddings into keys and values before computing attention maps.
- LoRA: LoRA represents the weight update as ∆W = BA with rank r much smaller than the matrix dimensions, while keeping pretrained weights W0 frozen.
- LoRA: In diffusion models, LoRA commonly adapts the query, key, value, and output projections in cross- and self-attention layers.
4. Method
B-LoRA analyzes SDXL to identify blocks associated with content and style, then jointly trains only those blocks to separate the two components of a single image. The resulting updates can be selectively recombined for image- and text-based stylization without further optimization.
- SDXL Architecture Analysis: SDXL analysis identifies blocks 2 and 4 as controlling generated content, while block 5 controls generated color.Injecting a tiger prompt into blocks 2 or 4 changes the content, whereas block 5 changes the image color.
- LoRA-Based Separation with B-LoRA: The method jointly optimizes LoRA weights for blocks 4 and 5 while reconstructing a single input image with a general prompt.Unlike full-model LoRA training, B-LoRA optimizes only two transformer blocks.
- LoRA-Based Separation with B-LoRA: Jointly training blocks 4 and 5 provides the best combination for reconstructing the input concept and capturing its content.The authors report that block 4 captures finer object details, while block 5 captures style-related information.
- LoRA-Based Separation with B-LoRA: B-LoRA reduces storage requirements by 70% and avoids overfitting concerns that motivate longer LoRA training in other methods.The approach trains only two blocks and uses the resulting updates without additional optimization or fine-tuning.
- LoRA-Based Separation with B-LoRA: B-LoRA implicitly separates content into ∆W 4 and style into ∆W 5, enabling either component to be applied independently during inference.The learned updates can be plugged into the pretrained SDXL model for style manipulation applications.
- B-LoRA for Image Stylization: For image-reference stylization, content and style B-LoRAs are learned separately and combined with a text prompt to generate the desired result.Using only the content update adapts the model to the content image, while text-based guidance can manipulate its style.
5. Results
B-LoRA produces stylized images while preserving content across image style transfer, text-based stylization, and consistent style generation. Comparisons indicate strong style transfer and content preservation, while quantitative evaluation reports the highest style alignment but lower object similarity.
- Qualitative Evaluation: B-LoRA transfers reference styles while preserving content in stylized images, including scenes and highly abstract styles.The method also preserves object content when the style is controlled by text.
- Applications: The method supports image style transfer, text-based image stylization, and consistent style generation from a single learned style representation.For consistent generation, the learned style remains preserved across generated images.
- Runtime: B-LoRA has runtime comparable to alternative methods and avoids ZipLoRA’s additional training phase for merging style and content LoRAs.The method trains its B-LoRAs once per image and reuses them without further optimization or fine-tuning.
- Qualitative Evaluation: Compared with alternatives, B-LoRA preserves the content subject while transferring the desired style, whereas ZipLoRA and StyleDrop can overfit content or lose style.StyleAligned preserves style but may introduce semantic content from the style image.
- Quantitative Evaluation: The method achieves the highest style alignment score, but its object similarity scores are lower than those of alternatives.The authors associate the lower object similarity with content overfitting issues observed in alternative methods.
- Quantitative Evaluation: Using a single content image decreases style consistency and increases content preservation scores across all methods.The authors interpret this pattern as suggesting overfitting.
6. Conclusions, Limitations and Future work
B-LoRA separates style and content with two independently usable adapters, but the method has limits involving color, background elements, and complex scenes. Future work targets finer-grained separation and combinations of multiple objects or styles.
- Two B-LoRAs encode style and content separately, supporting flexible use across image stylization tasks.
- Color is often assigned to the style component, which can weaken identity preservation when the content is stylized.
- A single reference image can cause the learned style component to include background elements instead of only the central object.
- Complex scenes with many elements may be difficult to capture accurately, potentially compromising content preservation.
- Future work proposes separating structure, shape, color, and texture, and combining LoRA weights from multiple objects or styles.
A. Comparisons
The comparisons evaluate B-LoRA against personalization and non-SDXL stylization methods using qualitative examples, a user study, and feature-similarity measurements. Against InstantStyle, both methods support consistent style generation, while B-LoRA is reported to preserve content better in some cases.
- User Study: The user study compares paired outputs from different methods on style adaptation and content preservation.The evaluation uses image pairs labeled A and B for participant preference judgments.
- Qualitative Comparisons: Qualitative comparisons assess B-LoRA against four state-of-the-art personalization baselines for image stylization.
- Beyond SDXL-Based Approaches: Additional comparisons include StyTr2, AdaAttn, and SWAG, using the same quantitative metrics as the main paper.
- Quantitative Evaluation: The quantitative evaluation measures average cosine similarity between output DINO features and reference style and content features.The experiment uses a single input image for evaluation.
- InstantStyle: Both B-LoRA and InstantStyle achieve consistent style generation, while B-LoRA can preserve content better when ControlNet fails to capture or overrides it.
B. Limitations
The method’s limitations concern identity preservation, style leakage, and content capture in complex scenes. The appendix discusses mitigation strategies involving adapter scaling, center cropping, and prompt placement across network blocks.
- Color Separation: Scaling the style adapter coefficient α to 0.4–0.5 helps preserve subject colors while minimizing interference with other style B-LoRA injections.
- Background Leakage: Center-cropping the style reference focuses B-LoRA training on the central object and reduces style leakage from background objects.
- Complex Scenes: The complex-scene limitation is investigated through five experiments that vary prompts assigned to the content block W 4, style block W 5, and remaining blocks.
- Complex Scenes: Injecting “A [c]” into W 4 while using “A [c] in [s] style” elsewhere often improves generation for complex scenes.
- Additional Comparisons: The appendix also compares the method with approaches that do not rely on SDXL.
C. Analysis and Ablation
Ablations examine block selection, prompt choice, style-content mixing, and the strength of adapter merging. Jointly optimizing blocks W 4 and W 5 gives the strongest separation, while prompt and alpha choices affect reconstruction and preservation.
- Layers Optimization: Jointly optimizing LoRA updates for blocks W 4 and W 5 produces the most satisfactory content-style reconstruction among tested block combinations.The combination also outperforms other combinations in the reported qualitative analysis.
- Layers Optimization: The W 4 and W 5 combination is preferred because using blocks 2 and 5 may reduce disentanglement when ∆W 5 learns content details.
- Prompt Selection: The prompt “A [c] in [s] style” fails to reconstruct structure, while adding class names creates style-transfer or structural problems.
- Prompt Selection: The class-free prompt “A [v]” preserves content structure while effectively transferring style from one object to another.
- Prompt Injection: Injecting “A [c]” into W 4 while using the compound prompt elsewhere often improves generation for complex scenes.
- Alpha Effect: Small alpha values can cause both style and content to be lost when merging the trained adapter update.
- Applications: B-LoRA supports style-content mixing, consistent style generation, and combinations of style, content, and text prompts.
E. Additional Results
B-LoRA supports image- and text-based stylization across diverse content and style references, including content-style mixing, object re-contextualization, and controlled style-content combinations.
- B-LoRA extracts content from style images and style from objects for object-mixing tasks.
- Randomly selected objects and styles demonstrate robustness across diverse content and style references.
- Content-only B-LoRA weights enable text-guided style manipulation while preserving the input object’s content and structure.
- The fourth prompt formulation better preserves the content object’s structure while transferring style than the first three formulations.
- Varying α provides quantitative control over the balance between style and content.
- Additional results show style transfer from either style or object reference images while re-contextualizing the content object.