PaperlayerView exact revision on arXiv

Source-linked AI summary

Diffu-LoRA: A Novel Low-Rank Adaptation for Personalized Diffusion Models

Tianjing Li, Wei Zhu

arXiv:2610.10550v1cs.CL

TL;DR

Personalizing text-to-image diffusion models from a few reference images requires preserving subject identity while following prompts for new contexts, and conventional LoRA leaves open how to distribute adaptation capacity across layers. Diffu-LoRA learns that allocation with gated low-rank updates, bilevel optimization, and progressive pruning while keeping the pretrained backbone frozen. Experiments report improved overall subject fidelity and prompt alignment against the evaluated fine-tuning baselines.

  • Problem

    Few-shot personalization must preserve a subject’s identity while supporting new prompts, but the distribution of limited LoRA capacity across diffusion layers remains an open question.

    Source: §1 · pp. 1–2

  • Method

    Diffu-LoRA learns a nonuniform allocation of gated low-rank updates through bilevel optimization and progressive pruning while keeping the pretrained backbone frozen.

    Source: §1 · p. 2, §4.1 · pp. 5–6

  • Results

    Experiments on DreamBooth subjects and additional collected datasets showed improved overall subject fidelity and prompt alignment against the evaluated fine-tuning baselines.

    Source: §1 · p. 2, §6 · p. 12

  • Takeaways & Limitations

    The results support learned rank allocation as a practical approach to parameter-efficient diffusion-model personalization.

    Source: §6 · pp. 11–12

Abstract

from arXiv · show

Personalizing text-to-image diffusion models from a few reference images requires preserving subject identity while following prompts that describe new contexts. Full-model fine-tuning is parameter-intensive, whereas low-rank adaptation (LoRA) reduces the number of trainable parameters but leaves open how adaptation capacity should be distributed across layers. We introduce Diffu-LoRA, a parameter-efficient method that learns this allocation through gated low-rank adaptation. Diffu-LoRA inserts trainable low-rank components into the linear layers of Transformer blocks and assigns a learnable gate to each component. Bilevel optimization updates the adaptation weights and gate parameters on separate data splits, while progressive pruning removes components with the lowest gate values to meet a prescribed rank budget. This procedure allocates adaptation capacity nonuniformly across layers while keeping the pretrained backbone frozen. Experiments with Stable Diffusion on subjects from DreamBooth and additional collected datasets show improved overall subject fidelity and prompt alignment relative to the evaluated fine-tuning baselines. Ablation studies examine the contributions of bilevel optimization, progressive pruning, and adapter placement. These results support learned rank allocation as a practical approach to parameter-efficient diffusion model personalization.

1 Introduction

Personalizing a diffusion model requires preserving subject identity while responding to new prompts, but few-shot fine-tuning is costly and uniform LoRA ranks may not suit every layer. Diffu-LoRA learns a nonuniform rank allocation with gated low-rank updates, bilevel optimization, and progressive pruning; the paper reports improved overall subject fidelity and prompt alignment.

1 Introduction

  • Few-shot personalization must preserve distinctive subject features while avoiding incidental reference details and allowing new poses, backgrounds, and styles.Large-backbone fine-tuning is costly, and limited references can lead to overfitting that harms responses to new prompts.Source: §1 · pp. 1–2
  • Diffu-LoRA places gated low-rank updates in Transformer-block linear layers and learns component gates separately from adaptation weights on distinct data splits.Low-gate components are progressively removed until the prescribed budget is met, with adaptation continuing between pruning stages.Source: §1 · p. 2
  • Diffu-LoRA combines component-wise gating, bilevel optimization, and progressive pruning to allocate low-rank capacity nonuniformly under a prescribed rank budget.The pretrained backbone remains frozen.Source: §1 · p. 2
  • The paper reports improved overall subject fidelity and prompt alignment against the evaluated fine-tuning baselines.Ablations examine bilevel optimization, progressive pruning, and adapter placement.Source: §1 · p. 2

2 Related Work

The related work frames Diffu-LoRA as a parameter-efficient approach that keeps a shared diffusion backbone frozen while learning how to distribute a fixed low-rank capacity across layers.

2.1 Parameter-Efficient Fine-Tuning

  • PEFT optimizes a small parameter set while freezing most or all of the backbone; Diffu-LoRA retains a shared frozen backbone and learns compact subject-specific updates.This is relevant when separate adaptations are maintained for individual subjects.Source: §2.1 · pp. 2–3

2.2 Low-Rank Adaptation and Its Variants

  • Unlike methods that improve the efficiency of a predefined LoRA configuration, Diffu-LoRA learns which components to retain across adapted layers under a fixed total capacity.It uses component-wise gates on a held-out split and progressively removes low-scoring components.Source: §2.2 · p. 3

3 Preliminaries

The preliminaries introduce the paper’s bilevel training setup, CLIP-based similarity measures, latent diffusion objective, and low-rank updates. These concepts establish the tools and terminology used for personalization and rank allocation.

3.1 Bilevel Optimization

  • Bilevel optimization alternates inner-variable updates on training data with outer-variable updates on a separate validation split.The separation discourages outer variables from merely fitting the examples used to optimize the model parameters.Source: §3.1 · pp. 3–4

3.2 CLIP

  • CLIP image and text embeddings provide a semantic agreement measure used to quantify visual similarity and prompt–image alignment.For normalized representations, their inner product is equivalent to cosine similarity.Source: §3.2 · p. 4

3.3 Text-to-Image Diffusion Models

  • A text-to-image latent diffusion model trains a denoiser to predict noise in a noisy image latent conditioned on the prompt.Personalization adapts a subset of denoiser parameters while retaining the pretrained model’s generative knowledge.Source: §3.3 · p. 4

3.4 Low-Rank Adaptation

  • LoRA represents a task-specific weight change with two trainable low-rank factors while keeping the original model parameters frozen.Diffu-LoRA differs from standard LoRA by learning how low-rank components are distributed across adapted layers.Source: §3.4 · p. 5

4 Methodology

Diffu-LoRA formulates few-shot subject personalization as adapting a frozen text-to-image model while preserving shared subject traits and avoiding incidental reference-image properties. It learns how to distribute a prescribed rank budget across layers through gated low-rank updates and progressive pruning.

4.1 Problem Formulation

  • Personalization adapts a pretrained text-to-image model from a few subject images, aiming to preserve shared visual traits without binding generation to incidental backgrounds, viewpoints, or poses.Instance prompts pair a rare subject identifier with a class noun, and a class-prior set is sampled from the frozen model.Source: §4.1 · p. 5
  • The objective selects layer-specific ranks under a global budget, separating reusable pretrained knowledge from compact subject-specific adapter parameters.The shared backbone remains frozen, and subject adapters can be loaded separately or merged into linear weights.Source: §4.1 · p. 6

4.2 Gated Low-Rank Adaptation

  • Diffu-LoRA inserts gated low-rank components into selected Transformer linear layers and learns nonuniform layer ranks under the target budget.Low-rank factors encode the subject, while gate logits determine which adaptation capacity is retained; factors update on Dtr and gates on Dval.Source: §4.2 · pp. 6–7

4.3 Progressive Rank Pruning

  • After a bilevel warm-up, progressive pruning removes the lowest-gated components and continues optimizing survivors until the target budget is reached.Removed masks and logits stay frozen, while surviving factors receive additional optimization between pruning stages.Source: §4.3 · p. 7

5 Experiments

The experiments compare Diffu-LoRA with full fine-tuning and conventional LoRA on Stable Diffusion subject-personalization benchmarks, assessing both subject fidelity and prompt alignment. Diffu-LoRA achieves the strongest overall balance on DreamBooth subjects, and results on additional collected subjects show improved prompt alignment over LoRA-All.

5 Experiments

  • The benchmark tests overall personalization and the contributions of bilevel optimization, progressive pruning, and adapter placement.It compares against full fine-tuning and conventional LoRA configurations.Source: §5 · p. 8

5.1 Datasets and Evaluation Protocol

  • The evaluation includes 12 DreamBooth subjects and six Custom Diffusion subjects, plus collected faces, anime characters, and distinctive objects.Face subjects have four to seven images from different viewpoints; evaluations are grouped into living subjects, inanimate objects, anime characters, and human faces.Source: §5.1 · p. 8

5.2 Implementation Details

  • All methods use Stable Diffusion v1.5, with selected blocks initialized at 192 components and pruned to a per-block budget of 96.Personalization runs for 200 steps; the bilevel warm-up lasts 20 steps, followed by 12 pruning stages.Source: §5.2 · p. 9

5.3 Evaluation Metrics

  • DINO and CLIP-I measure subject fidelity, while CLIP-T measures agreement between generated images and conditioning prompts.The paper assesses the joint behavior of all three metrics because high subject-fidelity scores alone may coexist with weak response to novel prompts.Source: §5.3 · pp. 9–10

5.4 Compared Methods

  • Comparisons use the same DreamBooth instance and prior-preservation objectives across full fine-tuning, attention-only LoRA, all-layer LoRA, and Diffu-LoRA.This design isolates parameterization and adapter placement from the personalization loss.Source: §5.4 · p. 10

5.5 Main Results

  • Diffu-LoRA achieves the strongest overall balance between subject fidelity and prompt compliance on the 12 DreamBooth subjects, with the highest CLIP-T score for every subject.Its advantage is especially clear on teapot, dog, cat, clock, and robot toy; isolated baselines retain slightly higher scores on a small number of individual subject–metric pairs.Source: §5.5 · pp. 10–11
  • Diffu-LoRA improves CLIP-T over LoRA-All for all six Internet-collected subjects and generally preserves or improves visual similarity.DINO decreases slightly for Hugging Face and Kiriko, while CLIP-I increases for both.Source: §5.5 · p. 11

5.6 Ablation Studies and Analysis

  • Ablations show that removing bilevel optimization reduces all three metrics, while omitting pruning or restricting adapters to attention weakens prompt alignment.No pruning occasionally raises DINO or CLIP-I marginally but fails to meet the prescribed compact rank budget.Source: §5.6 · p. 11

6 Conclusion

Diffu-LoRA learns compact, layer-dependent adapters under a global rank budget while keeping the pretrained backbone frozen. Across DreamBooth and additionally collected datasets, it provides a favorable balance between subject fidelity and prompt alignment compared with evaluated full-fine-tuning and conventional LoRA baselines.

6 Conclusion

  • Across DreamBooth and additionally collected datasets, learned rank allocation provides a favorable balance between subject fidelity and prompt alignment versus evaluated full-fine-tuning and conventional LoRA baselines.Source: §6 · pp. 11–12
  • Component-wise gates, bilevel optimization, and progressive pruning produce compact, layer-dependent adapters under a prescribed global rank budget while the pretrained backbone stays frozen.Source: §6 · p. 11
  • Ablations indicate the importance of the held-out gate objective, progressive pruning, and adapter placement.Source: §6 · p. 12
  • Future work will examine larger diffusion backbones, more diverse personalization scenarios, and allocation criteria that jointly consider rank, memory use, and inference cost.Source: §6 · p. 12

Related papers

Loading 2610.10550v1…