Source-linked AI summary
ZipLoRA: Any Subject in Any Style by Effectively Merging LoRAs
Viraj Shah, Nataniel Ruiz, Forrester Cole, Erika Lu, Svetlana Lazebnik, Yuanzhen Li, Varun Jampani
TL;DR
ZipLoRA addresses the unresolved challenge of generating a user-provided subject in a user-provided style without compromising either concept. It cheaply merges independently trained subject and style LoRAs, achieving strong preservation of subject identity and reference-style characteristics while avoiding manual merger-weight tuning.
Problem
Existing personalization methods handle subjects and styles independently, but reliably generating a specific subject in a specific style remains an open problem.
Method
ZipLoRA cheaply merges independently trained subject and style LoRAs using observations about sparse weights and independently learned LoRA representations, without manual hyperparameter tuning.
Results
ZipLoRA succeeds at preserving the identity of reference subjects while capturing the unique characteristics of reference styles across varied subject-style combinations.
Takeaways & Limitations
ZipLoRA provides a streamlined, hyperparameter-free way to combine independently available subject and style LoRAs for controllable diffusion-model personalization.
Takeaways & Limitations
The reasons for SDXL’s strong style-learning performance relative to earlier models remain unresolved.
Abstract
from arXiv · showhide
Methods for finetuning generative models for concept-driven personalization generally achieve strong results for subject-driven or style-driven generation. Recently, low-rank adaptations (LoRA) have been proposed as a parameter-efficient way of achieving concept-driven personalization. While recent work explores the combination of separate LoRAs to achieve joint generation of learned styles and subjects, existing techniques do not reliably address the problem; they often compromise either subject fidelity or style fidelity. We propose ZipLoRA, a method to cheaply and effectively merge independently trained style and subject LoRAs in order to achieve generation of any user-provided subject in any user-provided style. Experiments on a wide range of subject and style combinations show that ZipLoRA can generate compelling results with meaningful improvements over baselines in subject and style fidelity while preserving the ability to recontextualize. Project page: https://ziplora.github.io
1 Google Research 2 UIUC
ZipLoRA merges independently trained style and content LoRAs to generate user-provided subjects in user-provided styles.
- ZipLoRA merges independently trained style and content LoRAs for personalized diffusion-model creation.
1. Introduction
The introduction identifies arbitrary subject–style generation as an open problem and motivates ZipLoRA as a cheap, hyperparameter-free merge of independently trained LoRAs. Its design targets interference between sparse, potentially aligned LoRA columns while preserving each concept’s generation properties.
- Problem motivation: Generating a specific user-provided subject in a specific user-provided style remains an open research problem.The motivation includes controlling diffusion models through images rather than potentially imprecise text.
- Parameter-efficient personalization: LoRA provides parameter-efficient personalization with lower memory and storage budgets than fine-tuning all base-model parameters.LoRA learns low-rank factorized weight matrices for attention layers.
- Existing merging: Linear combinations of subject and style LoRAs require time-consuming grid search and are not robust across combinations.Users tune coefficients and evaluate results subjectively to find suitable merges.
- Proposed approach: ZipLoRA cheaply merges independently trained subject and style LoRAs without restricting how they were trained or requiring manual hyperparameter tuning.The method focuses on merging one subject LoRA with one style LoRA.
- Design rationale: LoRA weight matrices are sparse, while highly aligned columns from separate LoRAs can degrade merged-model performance when directly summed.ZipLoRA uses these observations to reduce similar-direction sums while preserving subject and style properties.
- Model setting: SDXL can learn a style from a single exemplar image using a DreamBooth protocol without human feedback.The paper reports this as a property that makes ZipLoRA particularly effective on SDXL.
2. Related Work
Prior work spans image stylization and diffusion-model personalization, but existing approaches have limitations in domain coverage, controllability, or style–subject fusion. ZipLoRA addresses this setting by combining independently trained style and subject LoRAs.
- Image stylization: CNN-based and GAN-based methods advanced arbitrary style transfer and one-shot stylization, but GAN approaches often remain domain-limited.Many GAN methods fine-tune a pretrained GAN for a reference style.
- ZipLoRA: ZipLoRA learns per-column mixing coefficients while preserving individual subject and style outputs and reducing cosine similarity between their LoRA columns.
- Image stylization: Some stylization methods use text to modulate content but do not support a reference style image or re-contextualization.
- Diffusion personalization: Textual Inversion and DreamBooth personalize diffusion models for specific subjects, while Custom Diffusion jointly trains concepts but requires expensive training and can fail to disentangle style from subject.
- Combining LoRAs: Weighted LoRA summation offers strength control but requires careful coefficient search, whereas gradient fusion is expensive and uses a less flexible LoRA variant.
3. Methods
ZipLoRA merges independently trained subject and style LoRAs by learning column-wise merger coefficients that preserve each concept while reducing interference. The method exploits sparse updates and low-cost optimization rather than joint retraining or manual coefficient search.
- Problem Setup: The problem setup seeks a merged LoRA that combines independently trained subject and style LoRAs to stylize a custom object.The base diffusion model and individual LoRAs remain separately defined before merging.
- Direct Merge: Direct merging uses a weighted sum of content and style updates, with coefficients controlling each LoRA’s strength.The merge is ΔWm = wc · ΔWc + ws · ΔWs.
- ZipLoRA: LoRA update matrices are sparse, allowing low-magnitude elements to be zeroed while preserving generation quality, including after removing 90% of elements.The paper attributes this redundancy to the deliberately small rank of the updates.
- ZipLoRA: Highly aligned columns from independently trained LoRAs can interfere when directly summed, whereas orthogonal columns better preserve their information.The paper measures alignment with cosine similarity and links nonzero similarity to signal interference.
- ZipLoRA: ZipLoRA learns column-wise coefficients to reduce interference while preserving the subject and style generation capabilities of the original LoRAs.The base model and individual LoRAs are frozen, and only merger coefficient vectors are optimized.
Layerwise Mean cosine similarity between
Figure 3 highlights two design observations: LoRA updates are sparse, and highly aligned columns merge poorly. ZipLoRA therefore reduces column similarity while retaining the useful update content.
- Layerwise Mean cosine similarity between: A teddy bear rendered in watercolor illustrates the target combination of a subject reference and a style reference.The figure labels the composition as a toy in a specified style.
- Sparse LoRA Updates: Panel (a) shows that LoRA weight matrices are sparse, with most update elements having near-zero magnitude.The method uses this sparsity to discard low-magnitude elements.
- Layerwise Mean cosine similarity between: Panel (b) contrasts direct merging of highly aligned LoRA columns with ZipLoRA’s strategy of minimizing mean cosine similarity across layers.The comparison concerns columns from the object and style LoRAs.
- Layerwise Mean cosine similarity between: The merger coefficient vector assigns each LoRA column a separate contribution to the final merged update.Element-wise multiplication applies the coefficient vector across the corresponding update-matrix columns.
- Layerwise Mean cosine similarity between: ZipLoRA optimizes both concept preservation and reduced content-style interference through a cosine-similarity-based objective.The loss compares outputs from the merged LoRA with those from the original subject and style LoRAs.
4. Experiments
Experiments show that SDXL can learn styles from single images and that ZipLoRA merges independently trained subject and style LoRAs with strong fidelity, re-contextualization, and efficiency.
- 4.1. Style-tuning behavior of SDXL model: SDXL learns diverse reference styles from a single image using DreamBooth LoRA fine-tuning, capturing painting style, lighting, colors, and geometry.The authors leave the explanation for SDXL’s stronger style learning than earlier models open.
- 4.1. Style-tuning behavior of SDXL model: SDXL style-tuning performs significantly better than StyleDrop on Muse, DreamBooth on Imagen, and DreamBooth on Stable Diffusion.Unlike StyleDrop, SDXL style-tuning does not require human feedback.
- 4.2. Personalized Stylizations: ZipLoRA preserves reference-subject identity while capturing the unique characteristics of the reference style across varied subject-style combinations.The method combines independently trained style and object LoRAs rather than jointly training them.
- 4.2. Personalized Stylizations: ZipLoRA outperforms direct arithmetic merging and joint training qualitatively, avoiding distorted outputs and poor subject-style disentanglement.Joint training also requires 10× as many training steps as ZipLoRA.
- 4.2. Personalized Stylizations: ZipLoRA re-contextualizes reference subjects while preserving stylization and supports smooth control over stylization strength through a scalar style multiplier.The merged model retains the base model’s text-to-image generation capabilities.
- 4.2. Personalized Stylizations: ZipLoRA runs in 560 seconds, uses 1.6M trainable parameters, requires 21GB of GPU memory, and stores 6.5MB of merger coefficients.The reported comparisons are 3540s and 3890s for joint training and Custom Diffusion, 38GB of memory, and 360MB of storage for competing LoRAs.
5. Conclusion and Future Work
The paper introduces ZipLoRA for merging independently trained style and subject LoRAs, enabling subject-style personalization with a cheap, streamlined approach. Combining more than two LoRAs remains future work.
- 5. Conclusion and Future Work: ZipLoRA seamlessly merges independently trained style and subject LoRAs to generate subjects in user-provided styles.The method is presented as cheap, streamlined, and hyperparameter-free for simultaneous personalization.
- 5. Conclusion and Future Work: Combining more than two subject or style LoRAs is identified as future work.
A. Additional Implementation Details
The experiments train style and content LoRAs on SDXL using DreamBooth, then merge them with fixed ZipLoRA settings and 50 inference steps.
- Style and content LoRAs are trained with the standard DreamBooth protocol on SDXL.
- Training uses 1000 fine-tuning steps, batch size 1, learning rate 5e −5, rank 64, and frozen text encoders.
- ZipLoRA initializes merger coefficients to ones and optimizes them with λ = 0.01 and learning rate 0.01.
- All experiments omit the SDXL refiner and use 50 diffusion inference steps.
B. Performance of ZipLoRA on Stable Diffusion
ZipLoRA retains advantages on SDv1.5 despite that model's weaker style-learning capability, outperforming competing approaches in qualitative and fidelity evaluations.
- SDv1.5 limits ZipLoRA because its underlying style LoRA captures stylization less faithfully than SDXL.
- The SDv1.5 evaluation compares ZipLoRA with Direct Merge, Joint Training, Custom Diffusion, Mix of Show, and StyleDrop+DreamBooth.
- ZipLoRA produces superior stylization outputs and surpasses competing methods even on SDv1.5.
- Additional evaluations report subject, style, and text alignment scores for ZipLoRA and competing methods on SDv1.5.
- Additional qualitative comparisons on SDXL report stronger style and subject fidelity for ZipLoRA than the listed baselines.
- Stylization remains unaffected after removing 80% of LoRA-update elements, but degrades when more are removed.
C.3. Additional results of style-tuning using SDXL.
Additional experiments show that SDXL can learn styles from a single exemplar without human feedback, and user studies compare personalized stylizations by style and subject fidelity.
- SDXL learns to generate stylized images through DreamBooth without requiring human feedback.
- Each user-study participant sees a reference subject, a reference style, and outputs from two compared methods.
- Participants choose the output that best depicts the reference style while preserving reference-subject fidelity.
- Compared with SDv1.5, SDXL produces superior-quality outputs when learning a style from a single reference image.
- Across five studies totaling 1800 responses, ZipLoRA was preferred over the competing methods in every study.
- The project supports personalization of subjects such as animals or objects in styles such as watercolor or sketch.
F. Datasets and Image Attributions
The experiments use style and content images from existing StyleDrop and DreamBooth datasets, with attribution information provided through their project webpages.
- Style and content images come from datasets collected by StyleDrop and DreamBooth, respectively.
- The datasets contain no human-subject data or personally identifiable information.
- The paper provides image attributions for the experimental images and directs readers to the source manuscripts and project websites.
- Style-image attribution information is provided through the StyleDrop project webpage.
- Content-image attribution information is provided through the DreamBooth project webpage.