PaperlayerView exact revision on arXiv

Source-linked AI summary

SVDiff: Compact Parameter Space for Diffusion Fine-Tuning

Ligong Han, Yinxiao Li, Han Zhang, Peyman Milanfar, Dimitris Metaxas, Feng Yang

arXiv:2303.11305v4cs.CV

TL;DR

Personalizing text-to-image diffusion models is constrained by overfitting, language drift, storage demands, and difficulty handling multiple similar concepts. SVDiff fine-tunes only singular values, adds Cut-Mix-Unmix augmentation, and supports text-based single-image editing. The method uses approximately 2,200 times fewer parameters than vanilla DreamBooth, while experiments report similar or better results than full-weight fine-tuning.

  • Problem

    Existing diffusion-model personalization methods face overfitting or language drift, large parameter spaces, and difficulty learning multiple personalized concepts, especially in similar categories.

  • Method

    SVDiff fine-tunes singular values of pretrained weight matrices in a compact spectral-shift parameter space and adds Cut-Mix-Unmix augmentation for multiple concepts.

  • Results

    Approximately 2,200× fewer parameters than vanilla DreamBooth, while experiments achieve similar or better results than full-weight fine-tuning for single- and multi-subject generation.

  • Takeaways & Limitations

    Cut-Mix-Unmix improves multi-subject generation for similar categories, while spectral shifts also support single-image editing as a regularized use case.

  • Takeaways & Limitations

    Cut-Mix-Unmix performance decreases as more subjects are added, and backgrounds may be inadequately preserved; stitching artifacts can also occur.

Abstract

from arXiv · show

Diffusion models have achieved remarkable success in text-to-image generation, enabling the creation of high-quality images from text prompts or other modalities. However, existing methods for customizing these models are limited by handling multiple personalized subjects and the risk of overfitting. Moreover, their large number of parameters is inefficient for model storage. In this paper, we propose a novel approach to address these limitations in existing text-to-image diffusion models for personalization. Our method involves fine-tuning the singular values of the weight matrices, leading to a compact and efficient parameter space that reduces the risk of overfitting and language drifting. We also propose a Cut-Mix-Unmix data-augmentation technique to enhance the quality of multi-subject image generation and a simple text-based image editing framework. Our proposed SVDiff method has a significantly smaller model size compared to existing methods (approximately 2,200 times fewer parameters compared with vanilla DreamBooth), making it more practical for real-world applications.

1. Introduction

SVDiff targets overfitting, language drift, storage inefficiency, and difficulty learning multiple personalized concepts in diffusion-model fine-tuning. It introduces spectral shifts, Cut-Mix-Unmix augmentation, and text-based single-image editing.

  • Motivation: Large fine-tuning parameter spaces can cause overfitting or language drift, while similar personalized concepts remain difficult to learn together.These limitations affect personalization methods for large-scale text-to-image diffusion models.
  • Spectral Shifts: Spectral shifts fine-tune only singular values of pretrained weight matrices, creating a compact parameter space for diffusion-model adaptation.The approach is based on singular-value decomposition of weight kernels.
  • Multi-Subject Learning: Cut-Mix-Unmix augmentation enhances learning of multiple personalized concepts, including semantically similar categories such as cats and dogs.The method is proposed to improve multi-subject personalization by presenting correctly cut-and-mixed samples.
  • Efficiency: ≈2,200× fewer parameters than vanilla DreamBooth makes SVDiff substantially more compact for StableDiffusion fine-tuning.The comparison is measured on StableDiffusion.
  • Image Editing: SVDiff also provides a text-based single-image editing framework using its spectral-shift parameter space.The framework is presented as an application of the proposed parameter space.

2. Related Work

Related work has personalized text-to-image diffusion models by tuning embeddings, full weights, cross-attention layers, or adapters, while also exploring training-free adaptation. SVDiff transfers singular-value tuning from GANs to diffusion-model fine-tuning.

  • Text-to-Image Diffusion: Diffusion models have become effective tools for learning data distributions and synthesizing images, including large-scale text-to-image generation.StableDiffusion is identified as a popular latent-diffusion variant used in the paper’s main experiments.
  • Personalization: Existing personalization methods fine-tune text embeddings, full weights, cross-attention layers, or adapters using a few personalized images.Other approaches investigate training-free methods for fast adaptation.
  • Spectral-Parameter Tuning: The idea of fine-tuning only singular values originated in GAN research and is extended by SVDiff to diffusion models.Prior GAN work also explored semantic directions in this compact parameter space.

3. Method

The method fine-tunes diffusion models in a compact spectral-shift parameter space and adds Cut-Mix-Unmix for multi-subject personalization. It also supports single-image editing through prompt modification and optional DDIM inversion.

  • 3.2. Compact Parameter Space for Diffusion Finetuning: SVD reshapes convolutional kernels into a 2-D matrix W and decomposes it as W = UΣV^⊤, with singular values σ in descending order.The decomposition is computed once and can be cached.
  • 3.2. Compact Parameter Space for Diffusion Finetuning: Instead of fine-tuning full weights, spectral shifts update only the singular values and reassemble the weight matrix using the fixed singular vectors.The shift δ is the difference between updated and original singular values, with ReLU enforcing nonnegative reconstructed singular values.
  • 3.2. Compact Parameter Space for Diffusion Finetuning: The training objective combines the target denoising loss with weighted prior-preservation loss, while single-image editing sets λ = 0 because prior preservation is unavailable.Individually trained spectral shifts can also be summed or interpolated to create new models and transition between image styles.
  • 3.3. Cut-Mix-Unmix for Multi-Subject Generation: Cut-Mix-Unmix trains multiple concepts with explicitly mixed image-prompt pairs and cross-attention regularization that separates non-corresponding subject regions.Inference uses a prompt containing both personalized concepts, while negative prompts can sometimes alleviate stitching artifacts.
  • 3.4. Single-Image Editing: CoSINE fine-tunes on one image-prompt pair and performs edits by modifying the prompt; DDIM inversion improves non-structural edits by initializing inference from the encoded input image.Large structural changes may still require additional noise injection, and CoSINE targets flexible edits rather than exact reconstructions.

4. Experiment

Experiments evaluate SVDiff for single- and multi-subject generation, image editing, parameter subsets, weight combinations, and style transfer. Results show competitive subject preservation, benefits from Cut-Mix-Unmix, and useful editing and style-mixing behavior, with some trade-offs for similar concepts and parameter choices.

  • Multi-subject generation: 400 image pairs were evaluated by 10 raters each, with SVD favored over full weights 60.9% of the time and a standard deviation of 6.9%.
  • Single subject generation: SVDiff performs similarly to DreamBooth for single-subject generation and preserves subject identities better than Custom Diffusion in rows 2, 3, and 5.
  • Multi-subject generation: Cut-Mix-Unmix benefits both full-weight fine-tuning and SVDiff, helping disentangle subjects from similar categories in multi-subject generation.Without the augmentation, similar concepts are difficult to separate; semantically separated concepts can succeed without it.
  • Single image editing: SVDiff performs desired single-image edits when full-weight fine-tuning fails, including object removal, pose adjustment, and zooming in, while preserving the subject.Some backgrounds may be affected, and DDIM inversion is used for several edits.
  • Parameter subsets: Optimizing cross-attention generally preserves identity better than optimizing key and value projections, while isolated UNet up-, down-, or mid-blocks are insufficient.Among the isolated blocks, up-blocks appear best for identity preservation; 2D weights have the most influence and outperform UNet-CA on identity preservation.
  • Weight combination and style mixing: Combining spectral shifts or full-weight deltas retains individual subject features, but similar concepts may blend styles; dissimilar concepts remain separately represented.The analysis suggests spectral-shift combinations can interfere more because update directions are limited to shared eigenvectors.

5. Conclusion and Limitation

The conclusion presents spectral shift as a compact parameter space for diffusion fine-tuning, with comparable or better results than full-weight fine-tuning across generation tasks. Cut-Mix-Unmix improves multi-subject generation, while the method also supports single-image editing, subject to stated limitations.

  • SVDiff achieves similar or better results than full-weight fine-tuning in single- and multi-subject generation.
  • Cut-Mix-Unmix improves multi-subject generation and enables handling subjects from similar categories.
  • Spectral shift acts as a regularization method and enables single-image editing use cases.
  • Limitations: Cut-Mix-Unmix performance decreases as more subjects are added, and single-image editing may inadequately preserve the background.

A. Implementation Details

Implementation details cover the StableDiffusion-based experimental setup, evaluation materials, and learning-rate choices for spectral shifts. Spectral-shift training uses a much larger learning rate than full-weight fine-tuning, while undecomposed 1-D weights support a stability–adaptation trade-off.

  • Implementation: Experiments use StableDiffusion implementations of DreamBooth and Custom Diffusion, while LoRA uses a separate implementation with rank-1 kernels.The LoRA delta checkpoint is 5.62MB in the described setup.
  • Evaluation: The multi-subject user study asks which image contains both input objects with a consistent background.
  • Evaluation: 400 generated image pairs were used in the Cut-Mix-Unmix visual-quality study.
  • Learning rate: Spectral shifts require a learning rate about 1,000 times larger than full-weight fine-tuning, such as 10^-3 versus the full-weight setting.
  • Learning rate: For undecomposed 1-D weights, 10^-6 can prevent overfitting, while a larger learning rate enables more rapid adaptation.

B.1. DDIM Inversion

DDIM inversion is evaluated for single-image editing with SVDiff, LoRA, and DreamBooth. It improves SVDiff editing quality and input alignment for non-structural edits, but can worsen full-weight and some LoRA results.

  • DDIM inversion improves editing quality and alignment with input images for non-structural edits using SVDiff.The comparison includes SVDiff, LoRA, and DreamBooth with and without inversion.
  • DDIM inversion makes results worse for full-weight fine-tuning in edits (b, g, i) and for LoRA in edits (b, i).
  • LoRA generally underfits the input image in the listed no-inversion examples.

B.2. Comparison with Other Methods

The paper compares SVDiff with Instruct-Pix2Pix while noting that the comparison is not entirely fair because Instruct-Pix2Pix requires no per-image fine-tuning. Human evaluations also compare SVDiff (“SVD”) with full-weight fine-tuning (“Full”) across subject combinations.

  • Instruct-Pix2Pix does not require fine-tuning on individual images, making the comparison with SVDiff not entirely fair.
  • Table 2 compares SVDiff (“SVD”) and full-weight fine-tuning (“Full”) for different subject combinations using 1000 human ratings per combination.

C.1. User Study

The user-study section evaluates SVDiff and full-weight fine-tuning through human preferences across subject combinations, alongside analyses of layer subsets and spectral-shift similarity. Participants could also indicate when comparisons were difficult to decide.

  • User evaluation: 1000 human ratings were collected for each of four subject combinations comparing SVDiff (“SVD”) with full-weight fine-tuning (“Full”).
  • Alignment analysis: Single-subject text- and image-alignment scores were measured after fine-tuning 12 subsets of UNet layers across five subjects.
  • Spectral-shift analysis: Spectral-shift cosine similarities were averaged across layers, with high similarities observed between conceptually similar subjects.

C.2. Analysis of Cut-Mix-Unmix

The analysis examines Cut-Mix-Unmix, negative prompting, and cross-attention regularization as mechanisms for separating personalized concepts and reducing artifacts or background overfitting. Results include improved disentanglement, artifact removal, and reduced background overfitting, with some limitations.

  • Cut-Mix analysis: Default left-and-right Cut-Mix augmentations preserve meaningful relations such as “wear,” “in,” and “ride,” while full-weight fine-tuning overfits to the augmentation layout.
  • Negative prompting: Negative prompting can remove stitching artifacts introduced by Cut-Mix-Unmix for both SVD and full-weight models, but it does not always help.
  • Attend-and-Excite extension: Cut-Mix-Unmix better disentangles the visual features of a dog and cat when integrated with Attend-and-Excite.
  • Cross-attention regularization: Cross-attention regularization limits a special token’s attention using a mask derived from the coarse class token and an L2 loss.
  • Cross-attention regularization: Cross-attention regularization reduces overfitting to the background in comparisons using 800 fine-tuning steps, prior-preservation loss, and the same random seed.

D.2. Fine-Tuning with Fewer Steps

The fewer-step experiments evaluate fast adaptation with 100 fine-tuning steps and no prior-preservation loss, tuning learning rates to balance faithfulness and realism. SVDiff, LoRA, and DreamBooth generally perform similarly, with method-specific overfitting or underfitting in some cases.

  • Experimental setting: 100 fine-tuning steps without prior-preservation loss define the fast-adaptation setting, unlike the main experiments using 500–1000 steps with prior-preservation loss.
  • Attention analysis: Cross-attention regularization is reported as effective for reducing overfitting to the background.
  • Attention analysis: Cross-attention maps show that a fine-tuned dog token can attend to background areas.
  • Artifact analysis: Negative prompting helps remove stitching artifacts for both SVD and full-weight models.
  • Experimental setting: Learning rates were tuned for each method to balance faithfulness and realism in the fast-adaptation experiments.
  • Results: SVDiff, LoRA, and DreamBooth perform similarly overall; LoRA underfits one sculpture, DreamBooth overfits it, and SVDiff overfits another case.

E. Analysis on Spectral Shifts

The analysis examines how spectral-shift rank, scaling, inversion, and regularization affect reconstruction, editing, personalization, and model behavior. Results show trade-offs between compact control, detail capture, prompt alignment, and visual quality.

  • Rank of spectral shifts: Rank-1 spectral shifts can reconstruct subjects, but low rank may lose details in edited samples, especially for buildings and panda sculptures.Differences between reconstructed and edited samples are smaller for Teddybear, which is easier for the model to adapt to.
  • Correlations: Conceptually similar subjects have relatively high cosine similarity between their learned spectral shifts.The analysis averages similarities across all layers; examples include panda and No-Face sculptures, and Teddybear and Tortoise plushies.
  • Scaling spectral shifts and weight deltas: Scaling spectral shifts and full weight deltas changes personalized attribute strength, while excessive scaling can deviate from the prompt and create visual artifacts.The figure contrasts spectral-shift scaling with full weight-delta scaling using the same random seed; s = 2 is given as an example of excessive scale.
  • Single-image editing: DDIM inversion improves editing quality and alignment with input images for non-structural edits in the spectral-shift parameter space.The reported settings use η = 0 without inversion and η = 0.5 with inversion, with exceptions for selected edits.
  • Single-image editing: Compared with Instruct-Pix2Pix, the method better supports the illustrated structural edits, while Instruct-Pix2Pix tends to alter overall color schemes.The cited examples include emptying a room and producing a zoom-in view.
  • Cross-attention analysis: Without unmix regularization, the dog’s special token attends largely to the panda in cross-attention visualizations.The visualization is obtained using Prompt-to-Prompt.
Loading 2303.11305v4…