PaperlayerView exact revision on arXiv

Source-linked AI summary

K-LoRA: Unlocking Training-Free Fusion of Any Subject and Style LoRAs

Ziheng Ouyang, Zhen Li, Qibin Hou

arXiv:2502.18461v2cs.CV

TL;DR

Existing LoRA-fusion methods struggle to preserve subject and style together or require tuning and additional training. K-LoRA combines independently trained LoRAs through Top-K selection and diffusion-step-aware scaling, achieving strong subject similarity while retaining satisfactory style similarity without retraining. The paper also identifies a limitation in a simpler element-zeroing strategy, which produces suboptimal image quality.

  • Problem

    Existing LoRA-fusion approaches may lose style details or object characteristics and may require manual tuning or additional training.

  • Method

    K-LoRA fuses independently trained content and style LoRAs using Top-K attention-component selection and scaling factors across diffusion steps.

  • Results

    K-LoRA significantly improves subject similarity metrics over previous approaches while achieving satisfactory style similarity.

  • Takeaways & Limitations

    K-LoRA directly merges existing LoRA weights without retraining or manual hyperparameter tuning while preserving subject and style features.

  • Takeaways & Limitations

    Zeroing small LoRA matrix elements produced suboptimal image-generation quality because previously learned concepts were not correctly interpreted.

Abstract

from arXiv · show

Recent studies have explored combining different LoRAs to jointly generate learned style and content. However, existing methods either fail to effectively preserve both the original subject and style simultaneously or require additional training. In this paper, we argue that the intrinsic properties of LoRA can effectively guide diffusion models in merging learned subject and style. Building on this insight, we propose K-LoRA, a simple yet effective training-free LoRA fusion approach. In each attention layer, K-LoRA compares the Top-K elements in each LoRA to be fused, determining which LoRA to select for optimal fusion. This selection mechanism ensures that the most representative features of both subject and style are retained during the fusion process, effectively balancing their contributions. Experimental results demonstrate that the proposed method effectively integrates the subject and style information learned by the original LoRAs, outperforming state-of-the-art training-based approaches in both qualitative and quantitative results.

1. Introduction

Personalization and stylization require preserving content while controlling visual attributes, but their interdependence makes effective separation difficult. K-LoRA addresses this by selecting and scaling contributions from content and style LoRAs across diffusion steps without additional training.

  • Content denotes image objects and structures, whereas style covers attributes such as color, texture, and patterns.
  • Existing LoRA-fusion methods can lose style details, inconsistently preserve object characteristics, or require manual tuning and additional training.
  • Restricted diffusion steps can retain LoRA effects, with early steps reconstructing objects and coarse textures and later steps refining finer object and style details.
  • K-LoRA uses Top-K selection within attention layers to identify suitable components and applies scaling factors to reflect content and style roles across diffusion.
  • K-LoRA merges content and style LoRAs without retraining while preserving subject and stylistic features across challenging combinations.

2. Related Work

Prior work studies diffusion-model customization and the combination of LoRAs for integrating objects or fusing content with styles. LoRA-based methods are emphasized for efficient large-model fine-tuning and high-quality generation.

  • Diffusion customization teaches models to interpret user-provided definitions, using methods such as Textual Inversion, DreamBooth, and Custom Diffusion.
  • Textual Inversion fine-tunes embeddings, DreamBooth expands object categories with class-specific terms, and Custom Diffusion fine-tunes cross-attention layers.
  • LoRA and its variants are presented as efficient approaches for fine-tuning large models and producing high-quality results.
  • LoRA-combination research addresses both integrating multiple object concepts and fusing contents with styles.

3. Method

K-LoRA fuses content and style LoRAs by selecting dominant attention-layer elements and scheduling their relative influence across diffusion steps. It uses Top-K comparisons, rank-aligned selection, and scaling to preserve object structure while retaining style details.

  • Preliminaries: K-LoRA combines independently trained content and style LoRA weights to fuse their learned representations in a base diffusion model.The method targets effective fusion of two LoRA weight sets without additional training.
  • Diffusion-step observations: Earlier diffusion steps favor content LoRA for object reconstruction, whereas later steps favor style LoRA for style and detail refinement.Applying style too early can affect object reconstruction, while content applied early performs better than content applied late.
  • Top-K selection: K-LoRA selects important LoRA elements per attention layer by taking absolute values, choosing the Top-K entries, and comparing accumulated sums.The method uses dominant elements because smaller elements occupy many positions and can influence importance estimation.
  • Top-K selection: The selection count K is aligned with each LoRA layer’s rank, using rc and rs to determine appropriate content and style weights.The rank is treated as reflecting the amount of information contained in a LoRA matrix.
  • Time-dependent weighting: A diffusion-time scaling factor strengthens content early and gradually emphasizes style later during Top-K selection.The factor uses the current and total denoising steps with hyperparameters α and β.
  • Source balancing: A balancing factor γ compensates for numerical disparities between LoRAs from different sources before attention-layer selection.The resulting layer-wise proportions show substantially different dominant-component sums, supporting distinct weight selection.
  • Fusion behavior: Visualization shows object-focused selection in the initial portion of generation and style-focused selection later, with each retaining a smaller contribution from the other.This pattern supports the intended transition from object reconstruction to style refinement.

4. Experiments

Experiments evaluate K-LoRA across diffusion models, LoRA sources, qualitative and quantitative comparisons, user studies, and ablations. The results support its ability to preserve and combine subject and style information without additional training, while showing that Top-K selection and scaling-factor choices affect output quality.

  • Experiment setup: Experiments use SDXL v1.0 and FLUX with both locally trained and community-trained LoRAs.Locally trained LoRAs follow the ZipLoRA procedure, while community models are drawn from Hugging Face.
  • Quantitative comparisons: K-LoRA improves subject-similarity metrics over prior approaches while maintaining satisfactory style similarity across 18 object-style combinations.The comparison includes direct arithmetic merging, joint training, ZipLoRA, and B-LoRA, using CLIP for style and CLIP and DINO scores for subject similarity.
  • Qualitative comparisons: K-LoRA generally produces more seamless object-style integrations, whereas competing methods can distort shapes, overfit colors, or lose stylistic features.The qualitative comparison evaluates outputs generated with SD under a direct 1:2 fusion ratio without extensive parameter or seed adjustments.
  • User and GPT-4o evaluations: K-LoRA is most preferred in a user study and shows a significant advantage in GPT-4o evaluation.The study compares ZipLoRA, B-LoRA, and K-LoRA against reference subject and style images.
  • Ablation analysis: Random selection often preserves only one feature type or neither, supporting distinct object and style roles across earlier and later diffusion steps.The controlled random baseline selects content attention with probability 1/3 and style attention with probability 2/3.
  • Ablation analysis: Removing Top-K selection or the scaling factor can cause object distortion, style loss, or weaker object-style fusion.Small K values underrepresent both features, while excessively large K values can reduce style preservation and distort object shape.

5. Conclusions

K-LoRA merges independently trained style and subject LoRAs while preserving object detail and intricate style features without retraining or manual hyperparameter tuning.

  • K-LoRA seamlessly merges independently trained style and subject LoRA models.
  • The method preserves precise object features alongside intricate details from the original style.
  • Top-K selection and scaling factors combine object and style LoRA contributions at each diffusion step.
  • K-LoRA performs accurate style fusion without retraining or manual hyperparameter tuning.

Supplementary Material

The supplementary material documents evaluations across datasets, models, comparisons, prompts, scaling choices, and combinations of community and local LoRAs.

  • The supplementary material evaluates results on extensive datasets and community LoRAs across different models.
  • It compares K-LoRA with other methods and assesses the influence of complex prompts on performance.
  • Additional experiments test a new scale and combine community LoRA with local LoRA for integrated performance evaluations.

A. Visual Results

Visual evaluations on StyleDrop and DreamBooth datasets with Stable Diffusion, plus Hugging Face LoRAs on FLUX, show consistent integration of object and style information.

  • Experiments use StyleDrop and DreamBooth datasets with Stable Diffusion and Hugging Face LoRAs on FLUX.
  • Systematic combinations of object and style LoRAs produce consistent, high-quality visual outputs integrating both attributes.

B. Additional Comparisons

Additional comparisons find that StyleID transfers style while preserving texture quality, whereas K-LoRA retains object and style attributes when prompts alter actions, environments, or scene elements.

  • StyleID achieves style transfer with preserved texture quality, but generated objects may be blurred or the style insufficiently distinct.
  • StyleID relies on the original image’s fixed layout, which may limit generalization to different backgrounds and actions.
  • Prompt modifications allow K-LoRA to retain original object features and stylistic attributes while adding new elements or scene details.

D. New Scale

K-LoRA evaluates an alternative scale factor that strengthens style information early in generation, revealing a trade-off between capturing broad style structure and preserving fine stylistic details.

  • D. New Scale: The alternative scale factor uses α′ = 1.5 and β′ = 1.3 to enhance style information at the beginning of generation.This encourages the model to capture block information from the style LoRA.
  • D. New Scale: S∗ results capture background and color-block information from the style LoRA when style is enhanced during early diffusion steps.
  • D. New Scale: The same S∗ setting weakens the style LoRA’s texture and brushstroke learning effect, creating a trade-off for users choosing scale factors.
  • D. New Scale: Figure 10 compares K-LoRA generation results under different scaling factors using two randomly selected seeds for each object-style pair.

E. Robustness Analysis

Robustness tests across community and locally trained LoRAs show that Fixed Selection outperforms comparison methods while maintaining stable fusion across random seeds.

  • E. Robustness Analysis: K-LoRA’s Fixed Selection achieves superior learning of both object and style characteristics compared with Direct-Merge and Multi-LoRA composition.The evaluation uses community-sourced object LoRAs and locally trained style LoRAs.
  • E. Robustness Analysis: Random-seed tests show consistently stable fusion across a broad range of seed selections.The reported stability supports reliable integration across the tested conditions.

F. Additional Ablations

Additional ablations examine scale selection, generated examples, detailed texture comparisons, robustness, and prompt control to characterize K-LoRA across settings.

  • F. Additional Ablations: The experimentally optimal scale is α = 1.5 and β = 0.5, a configuration that works for almost all content-style pairs without further adjustment.
  • F. Additional Ablations: Additional FLUX and SD results show K-LoRA applied to different object-style combinations.The corresponding figures organize generated images by object and style references.
  • F. Additional Ablations: The StyleID comparison includes zoomed output patches to assess detailed texture information and stylistic features.Rows distinguish StyleID outputs from K-LoRA outputs and their associated zoomed patches.
  • F. Additional Ablations: Robustness validation compares Fixed Selection with direct arithmetic LoRA merging and Multi-LoRA Composition using community and locally trained LoRAs.
  • F. Additional Ablations: Prompt-control evaluations introduce new scenes, actions, and objects to test re-contextualization while maintaining stylistic consistency.
Loading 2502.18461v2…