SVDiff: Compact Parameter Space for Diffusion Fine-Tuning

• Large parameter spaces in diffusion-model fine-tuning can cause overfitting or language drift, while learning multiple personalized concepts—especially similar ones—is difficult [57] [37].Source: §1 · p. 1
• A DreamBooth-based text-editing framework demonstrates single-image editing with spectral shifts, including when prior-preservation loss is not applicable [57].Source: §1 · p. 2
• Cut-Mix-Unmix augmentation, combined with spectral shifts, enables learning multiple personalized concepts even for semantically similar categories such as cats and dogs.Source: §1 · p. 2
• ≈2,200× fewer parameters than vanilla DreamBooth [57] on StableDiffusion [56] is achieved by fine-tuning weight-matrix singular values.The spectral-shift parameter space retains the weight matrix’s full representation power while remaining more compact than a low-rank constraint; its StableDiffusion representation is 1.7MB versus 3.66GB for a full-weight checkpoint.Source: §1 · pp. 1–2

Text-to-image diffusion models

• Text-guided image synthesis has advanced with diffusion models [4] [44] [52] [56] [59], and the paper’s main experiments use StableDiffusion [56], a latent diffusion model operating on compressed autoencoder features.The related-work discussion also notes transformer-based architectures [6] [7] [49] [73].Source: §2 · p. 2

Fine-tuning generative models for personalization

• SVDiff brings singular-value fine-tuning to few-shot diffusion personalization, optimizing all weight-matrix singular values rather than using LoRA’s low-rank adaptation.FSGAN [55] introduced singular-value fine-tuning in GANs, and NaviGAN [15] advanced it for unsupervised semantic-direction discovery.Source: §2 · p. 2

Diffusion-based image editing

• SVDiff presents a DreamBooth-based [57] single-image editing framework, while prior diffusion editing methods typically use inversion [65] and null-text optimization or image overfitting [85].The framework demonstrates SVDiff’s potential for single-image editing and mitigating overfitting.Source: §2 · p. 2

3.1. Preliminary

• StableDiffusion operates as an LDM that encodes images into latents for denoising, while FSGAN [55] motivates adapting reshaped weight-matrix singular values.The paper frames convolution kernels as reshaped 2-D matrices and illustrates their SVD.Source: §3.1 · pp. 2–3, Figure 2 · p. 3

3.2. Compact Parameter Space for Diffusion Finetuning

• SVDiff computes a cacheable SVD and optimizes spectral shifts rather than full weight matrices, reconstructing weights with ReLU-clamped shifted singular values.The shift δ is the difference between updated and original singular values.Source: §3.2 · p. 3
• Fine-tuning uses the diffusion model’s denoising loss with weighted prior preservation, but sets λ = 0 for single-image editing where prior preservation cannot be used.The prior-preservation term acts as a smoothing term.Source: §3.2 · p. 3
• Individually trained spectral shifts can be added or interpolated to combine models, enabling multi-subject generation, style mixing, and smooth transitions between styles.Interpolation weights the two shifts by α and 1 − α.Source: §3.2 · p. 3

3.3. Cut-Mix-Unmix for Multi-Subject Generation

• Cut-Mix-Unmix trains on manually constructed mixed image-prompt pairs, usually with probability 0.6, while cross-attention MSE regularization separates subjects and reduces stitching artifacts.Using augmentation with probability 1 can hinder subject differentiation; overfitting to mixed samples may cause stitching artifacts, which negative prompts can sometimes alleviate.Source: §3.3 · pp. 3–4, Figure 3 · p. 4

3.4. Single-Image Editing

• CoSINE fine-tunes a diffusion model with an image-prompt pair in spectral-shift parameter space, edits by changing the inference prompt, and reconstruction can be improved with DDIM inversion.Large structural changes may still require more noise, injected by setting η > 0 or perturbing zT.Source: §3.4 · pp. 4–5

4.1. Single-Subject Generation

• For single-subject generation, SVDiff performs similarly to DreamBooth while preserving subject identity better than Custom Diffusion in the examples shown.The reported comparisons used StableDiffusion implementations; Custom Diffusion tended to underfit the training images.Source: §4.1 · p. 5, Figure 5 · p. 6

4.2. Multi-Subject Generation

• Cut-Mix-Unmix, applied with probability 0.6 by pairing two randomly selected subjects, benefits multi-subject generation, especially for disentangling similar concepts.With three concepts, semantically separated pairs such as dog/building can work without the augmentation, but similar concepts such as dog/panda are difficult to disentangle; the augmentation also applies to full-weight fine-tuning.Source: §4.2 · p. 5, Figure 6 · p. 7
• SVDiff was favored over full-weight fine-tuning in 60.9% of 400 multi-subject image-pair comparisons (standard deviation: 6.9%).Each pair used the same random seed and was evaluated by 10 raters, who judged whether both objects appeared with a consistent background.Source: §4.2 · p. 5

4.3. Single Image Editing

• In single-image editing, SVDiff can perform desired edits when full-weight fine-tuning exhibits language drift, including object removal, pose changes, and zooming in.DDIM inversion improves alignment for non-structural edits with spectral shifts, but can worsen results with full-weight fine-tuning; without inversion, spectral shifts can sometimes produce over-creative results.Source: §4.3 · p. 6, Figure 7 · p. 8

4.4. Analysis and Ablation

• Among UNet parameter subsets, optimizing cross-attention generally preserves subject identity better than optimizing key and value projections, while 2D weights outperform UNet-CA on identity preservation.Optimizing the up-, down-, or mid-blocks alone is insufficient to maintain identity, though up-blocks appear to preserve it best.Source: §4.4 · p. 9
• Combining spectral shifts or full-weight deltas retains individual subject features but can blend styles for similar concepts, while both parameter types can interpolate between concepts.SVDiff’s shared eigenvector update directions may cause more interference when spectral shifts are summed; spectral shifts also support style transfer by changing the class word or adding “in style of,” and style mixing with geometry and appearance information from selected layers.Source: §4.4 · pp. 9–10, Figure 8 · p. 9, Figure 11 · p. 9

4.5. Comparison with LoRA

• For single-image editing, SVDiff provides a balance between faithfulness and realism, and its delta checkpoint is 1/2 to 1/3 the size of LoRA’s.LoRA tends to underfit in the reported comparison, but its adjustable rank may help when extensive fine-tuning or learning new concepts is needed.Source: §4.5 · p. 10

5. Conclusion and Limitation

• Spectral-shift fine-tuning achieves results similar to or better than full weight fine-tuning for both single- and multi-subject generation.Source: §5 · p. 10
• Cut-Mix-Unmix improves multi-subject generation quality, including cases where subjects belong to similar categories.Source: §5 · p. 10
• Spectral shift acts as a regularizer and enables single-image editing as an additional use case.Source: §5 · p. 10
• Cut-Mix-Unmix performance can decrease as the number of subjects increases.Source: §5 · p. 10
• Single-image editing may inadequately preserve the background.Source: §5 · p. 10
• Future work may combine spectral shifts with LoRA or develop training-free approaches for faster concept personalization.Source: §5 · p. 10

A. Implementation Details

• DreamBooth [57] and Custom Diffusion [37] are implemented in StableDiffusion with Diffusers, while the experiments build on StableDiffusion implementations of Imagen-based DreamBooth [59] [56].Source: A. Implementation Details · p. 14
• The custom LoRA [27] implementation fine-tunes 1-D weight kernels and uses rank-1 for 2-D and 4-D kernels, producing a 5.62MB delta checkpoint.This checkpoint is slightly larger than the official LoRA implementation.Source: A. Implementation Details · p. 14
• Figure 13's human-evaluation example shows two real images labeled A and B alongside one synthesized image labeled C.Source: Figure 13 · p. 14
• 1,000 times (for example, 10^-3) is the learning rate used for spectral shifts relative to full-weight fine-tuning.Source: A. Implementation Details · p. 14
• For undecomposed 1-D weights, the learning rate is either 10^-6 to prevent overfitting or higher for faster adaptation, depending on the stability-speed trade-off.Source: A. Implementation Details · p. 14

B.1. DDIM Inversion

• DDIM inversion improves editing quality and alignment with input images for SVD on non-structural edits.The comparison uses DDIM sampling with η = 0.5 and α = 0 when inversion is employed, except for edits (d), (f), and (h), which use η = 0.9 and α = 0.9.Source: §B.1 · p. 14
• In edits (b), (g), and (i), inversion worsens DreamBooth (“Full”) results; in edits (b) and (i), it worsens LoRA [27] results.For the chair example, a large amount of noise is needed to obtain the desired edits.Source: §B.1 · p. 14
• Without inversion, LoRA [27] tends to underfit the input image, as shown in edits (c), (d), (e), and (i).Without inversion, the comparison uses DDIM sampling with η = 0.Source: §B.1 · p. 14

B.2. Comparison with Other Methods

• The paper compares SVDiff with Instruct-Pix2Pix [10] in Fig. 22, where Instruct-Pix2Pix is marked “ip2p”.Source: §B.2 · p. 14
• The comparison is not entirely fair because Instruct-Pix2Pix does not require fine-tuning on individual images.Source: §B.2 · p. 14
• The paper identifies fast personalized adaptation without per-image fine-tuning as a direction for future work.Source: §B.2 · p. 14

C. Multi-Subject Generation

• Table 2 reports human evaluation results comparing “SVD” and “Full” across subject combinations, with 1000 ratings for each combination.Source: Table 2 · p. 14

C.1. User Study

• The study compared SVD and Full using side-by-side generated images, with 1,000 ratings collected for each of four subject combinations.Participants chose their preferred image or indicated that it was hard to decide.Source: §C.1 · p. 14
• Hard-to-decide responses were 4.1%, 1.2%, 2.1%, and 2.2% across the four subject combinations, respectively.Source: §C.1 · p. 15

C.2. Analysis of Cut-Mix-Unmix

• Cut-Mix-Unmix’s default left-right and up-down layouts generate meaningful relations, but “full” can overfit to the augmentation layout.The authors suggest random layouts, particularly with cross-attention regularization, could mitigate overfitting, but leave that study for future work.Source: §C.2 · p. 15
• Randomly splitting the left and right images produced results similar to a fixed 1:1 ratio.The authors identify random layouts, especially with cross-attention regularization, as a possible way to further mitigate overfitting, leaving this study for future work.Source: §C.2 · p. 15

C.3. Negative Prompt

• Negative prompting can remove Cut-Mix-Unmix stitching artifacts, but does not always help.The approach repurposes prior-preservation prompts as negative prompts and replaces the null-conditioned score in classifier-free guidance; the authors hypothesize this counters the model’s learned association between the prior prompt and stitching style.Source: §C.3 · p. 15

C.4. Extensions

• Cut-Mix-Unmix helps Attend-and-Excite better disentangle the dog’s and cat’s visual features.Attend-and-Excite already separates the subjects by design, though the cat’s color slightly leaks onto the dog; the method may also integrate with other attention-based methods.Source: §C.4 · p. 16, Figure 16 · p. 16

D.1. Cross-Attention Regularization

• A special token’s cross-attention can leak into unwanted background areas even in single-subject generation.The authors compare the “[V1]” token, whose attention leaks to the background, with “[V2],” whose attention does not.Source: §D.1 · p. 16
• Cross-attention regularization reduces overfitting to the background in single-subject generation.The method constrains a special token’s attention to the region indicated by a thresholded coarse-class-token attention map, using an L2 loss.Source: §D.1 · p. 16

D.2. Fine-Tuning with Fewer Steps

• Fast adaptation uses 100 fine-tuning steps without prior-preservation loss, unlike the main experiments’ 500–1,000 steps with that loss [57].Learning rates are tuned to balance faithfulness and realism [40].Source: §D.2 · pp. 16–17
• The tuned learning rates vary by method and weight dimensionality: SVDiff uses higher 2-D and 4-D rates than LoRA and DreamBooth.SVDiff: 1-D 2 × 10−3, 2-D and 4-D 5 × 10−3; LoRA [27]: 1-D 2 × 10−3, 2-D and 4-D 1 × 10−4; DreamBooth [57]: 1-D 1 × 10−3, 2-D and 4-D 5 × 10−6.Source: §D.2 · p. 17
• The methods perform similarly overall, but LoRA underfits and DreamBooth overfits on the “No-Face” sculpture, while SVDiff overfits on another subject, possibly due to its large learning rate.The fast-finetuning comparison is shown in Fig. 26.Source: §D.2 · p. 17

E.1. Rank

• With rank-1 spectral shifts, the model still reconstructs subjects, but low rank can limit details in edited samples, especially for the building and panda sculpture.The difference between reconstructed and edited samples is smaller for the Teddybear, potentially because the pretrained model already understands that concept.Source: §E.1 · p. 17, Figure 24 · p. 21

E.2. Correlations

• Similar subjects have relatively high average cosine similarity between their learned spectral shifts, including panda and No-Face sculptures and Teddybear and Tortoise plushies.The analysis averages cosine similarities across layers; diagonal entries compare runs with 1-D weight learning rates of 10−3 and 10−6.Source: §E.2 · pp. 17–18

E.3. Scaling

• Scaling spectral shifts or full weight deltas changes personalized attributes and feature strength, while an overly large scale such as s = 2 can cause prompt deviation and dark samples.The comparison uses the same random seed; the full-weight variant is W′ = W + s∆W, and the spectral-shift variant is Σδ′ = diag(ReLU(σ + sδ)).Source: §E.3 · p. 18, Figure 25 · p. 21

F. Image Attribution

• Teddybear, tortoise plushy, grey dog, and cat images are credited to Custom Diffusion [37], while panda and No-Face sculpture images were collected by the authors.These are among the image sources listed in the attribution section.Source: F. Image Attribution · p. 18
• DDIM inversion improves alignment with input images for non-structural edits using spectral shifts, but can harm “Full” and “LoRA” results, such as emptying a room or turning it purple.The figure caption specifies different sampler settings for inversion and non-inversion, with separate settings for edits (d), (f), and (h).Source: Figure 21 · p. 19
• Instruct-Pix2Pix [10] tends to alter the overall color scheme and struggles with significant or structural edits, including emptying a room and a zoom-in view.The comparison marks Instruct-Pix2Pix as “ip2p.”Source: Figure 22 · p. 20
• Without unmix regularization, the dog’s special token “sks” attends largely to the panda in cross-attention maps visualized with Prompt-to-Prompt [22].The figure analyzes the fine-tuned model without unmix regularization.Source: Figure 23 · p. 20