Source-linked AI summary
Controlling Text-to-Image Diffusion by Orthogonal Finetuning
Zeju Qiu, Weiyang Liu, Haiwen Feng, Yuxuan Xue, Yao Feng, Zhen Liu, Dan Zhang, Adrian Weller, Bernhard Schölkopf
TL;DR
The paper addresses how to control text-to-image diffusion models for downstream tasks while preserving their pretrained generative abilities. It introduces OFT, which preserves hyperspherical energy through orthogonal transformations, and COFT, which adds a stability constraint. Across subject-driven and controllable generation, the framework improves generation quality, convergence speed, stability, and sample efficiency, while adding no inference overhead.
Problem
Text guidance can be ambiguous for fine-grained control, creating a need to finetune diffusion models without losing pretrained generative performance.
Method
OFT adapts diffusion models with layer-shared orthogonal transformations that preserve hyperspherical energy, while COFT adds an angular-deviation constraint for stability.
Results
OFT improves generation quality, convergence speed, finetuning stability, and sample efficiency across subject-driven and controllable generation tasks.
Takeaways & Limitations
Preserving pairwise neuron angles provides a practical balance between finetuning flexibility and regularity, with no additional inference overhead.
Takeaways & Limitations
OFT’s Cayley parametrization involves a matrix inverse that limits scalability, while parameter efficiency depends on block-diagonal structure.
Abstract
from arXiv · showhide
Large text-to-image diffusion models have impressive capabilities in generating photorealistic images from text prompts. How to effectively guide or control these powerful models to perform different downstream tasks becomes an important open problem. To tackle this challenge, we introduce a principled finetuning method -- Orthogonal Finetuning (OFT), for adapting text-to-image diffusion models to downstream tasks. Unlike existing methods, OFT can provably preserve hyperspherical energy which characterizes the pairwise neuron relationship on the unit hypersphere. We find that this property is crucial for preserving the semantic generation ability of text-to-image diffusion models. To improve finetuning stability, we further propose Constrained Orthogonal Finetuning (COFT) which imposes an additional radius constraint to the hypersphere. Specifically, we consider two important finetuning text-to-image tasks: subject-driven generation where the goal is to generate subject-specific images given a few images of a subject and a text prompt, and controllable generation where the goal is to enable the model to take in additional control signals. We empirically show that our OFT framework outperforms existing methods in generation quality and convergence speed.
1 Introduction
Text guidance remains ambiguous for fine-grained control, motivating finetuning methods that adapt diffusion models without sacrificing pretrained generative performance. The paper proposes OFT, which preserves hyperspherical energy by learning layer-shared orthogonal transformations, and applies it to subject-driven and controllable generation.
- Text-to-image diffusion models can generate high-fidelity images, but text guidance may be insufficient for fine-grained, accurate control.
- Subject-driven generation creates images of a given subject in new contexts from a few subject images and a text prompt.
- Controllable generation conditions image synthesis on an additional signal, such as canny edges or segmentation maps, alongside a text prompt.
- Effective finetuning should reduce trainable parameters and epochs while preserving high-fidelity, diverse generation.
- Euclidean distance from pretrained weights alone may not capture semantic preservation, motivating structural measures of finetuned-versus-pretrained models.
- OFT preserves hyperspherical energy by applying shared orthogonal transformations to neurons, while COFT additionally constrains angular deviation for stability.
2 Related Work
Related work spans text-to-image diffusion, subject-driven generation, controllable generation, and parameter-efficient finetuning. OFT differs from LoRA by using layer-shared multiplicative orthogonal transformations and is reported to improve controllability with less data and no inference overhead.
- Text-to-image diffusion models: Text-to-image diffusion models have advanced through diffusion-based generation and vision-language representation learning.
- Subject-driven generation: Subject-driven methods use masks, inversion, or generator finetuning, but prior approaches may not preserve identity-related subject details well.
- Controllable generation: Controllable generation incorporates additional control signals into image synthesis, with ControlNet adapting pretrained diffusion models for this purpose.
- Controllable generation: OFT reportedly delivers better controllability with fewer training data and finetuning parameters, without additional test-time computational overhead.
- Model finetuning: LoRA uses low-rank additive updates, whereas OFT uses layer-shared orthogonal transformations to update neuron weights multiplicatively.
3 Orthogonal Finetuning
Orthogonal Finetuning adapts diffusion-model neurons by changing their directions through orthogonal transformations, preserving hyperspherical energy while enabling efficient and stable control. COFT adds an explicit deviation constraint, while block-diagonal and rescaled variants improve flexibility or parameter efficiency.
- Why Does Orthogonal Transformation Make Sense?: Angular information can almost perfectly recover input images, whereas neuron magnitude contains no useful information in the toy autoencoder experiment.This motivates finetuning neuron directions rather than magnitudes to modify generated-image semantics.
- Why Does Orthogonal Transformation Make Sense?: Orthogonal transformation offers a balance between flexibility and regularity for finetuning neuron directions.The transformation rotates or reflects all neurons in a layer while preserving their pairwise hyperspherical relationships.
- General Framework: OFT constrains finetuning by preserving hyperspherical energy, defined from pairwise normalized-neuron distances, and implements this through layer-shared orthogonal matrices.For a pretrained layer W0, OFT uses W = RW0 with R⊤R = RR⊤ = I and initializes R as the identity.
- Efficient Orthogonal Parameterization: Block-diagonal orthogonal matrices reduce parameter complexity from O(d^2) to O(d^2/r), or O(d^2/r^2) when blocks are shared, without sacrificing hyperspherical-energy preservation.When r = 1, the block-diagonal form recovers the unconstrained orthogonal matrix.
- Constrained Orthogonal Finetuning: COFT adds the radius constraint ∥R − I∥ ≤ ϵ to orthogonality, explicitly limiting deviation from pretrained weights and improving finetuning stability.Larger ϵ makes COFT resemble OFT; smaller ϵ makes it increasingly resemble the pretrained diffusion model.
- Re-scaled Orthogonal Finetuning: Rescaled OFT additionally learns positive per-neuron magnitude coefficients, increasing flexibility with negligible extra parameters while retaining orthogonal transformation.The experiments use original OFT to isolate the effect of orthogonal transformation, while the rescaled variant is reported as generally better in Appendix C.
4 Intriguing Insights and Discussions
The discussion positions OFT as a structured alternative that balances finetuning flexibility with regularity while preserving pretrained information and adding no inference overhead.
- Architecture applicability: For Transformers, experiments apply OFT to attention weights, while its block-diagonal structure also supports convolutional-layer finetuning.The paper reports a preliminary convolutional-layer study in Appendix D.
- Connection to LoRA: OFT preserves pretrained information by applying a full-rank orthogonal transform, unlike LoRA’s low-rank update.Its forward pass can be rewritten as a pretrained weight plus an analogous orthogonal-induced update.
- Why OFT converges faster: OFT converges faster because it changes neuron directions and optimizes neurons on a smooth hypersphere manifold.The paper links these design choices to effective semantic modification and a better optimization landscape.
- Why not minimize hyperspherical energy: Minimizing hyperspherical energy is unsuitable for finetuning because uniformly spacing neurons may destroy pretrained information.The paper distinguishes this objective from OFT’s goal of preserving hyperspherical energy.
- Trade-off between flexibility and regularity in finetuning: OFT balances flexibility and regularity by preserving pairwise neuron angles, whereas standard finetuning is unstable and can cause model collapse.Block-diagonal parameterization provides an additional form of orthogonal-matrix regularization.
- No additional inference overhead: OFT introduces no additional inference overhead because the learned orthogonal matrix can be absorbed into the pretrained weight matrix.Inference speed therefore remains the same as for the pretrained model.
5 Experiments and Results
Experiments evaluate OFT and COFT on subject-driven and controllable generation, finding stable convergence, strong semantic preservation, and accurate control compared with existing methods.
- Subject-driven Generation: Stable convergence: after 400 iterations, DreamBooth and OFT achieve good control, while LoRA fails to preserve subject identity.After 2000 iterations, DreamBooth begins generating collapsed images and LoRA generates yellow fur instead of a yellow shirt, whereas OFT and COFT remain stable.
- Subject-driven Generation: OFT and COFT remain effective with relatively large iteration counts, reducing the need to tune iteration numbers for different subjects.With a proper ϵ, COFT also makes both the learning rate and iteration number easier to set.
- Subject-driven Generation: OFT and COFT outperform DreamBooth and LoRA on DINO and CLIP-I while achieving slightly better or comparable CLIP-T and LPIPS performance.The comparison uses 30 random seeds per method to reduce randomness and reports better final performance together with convergence and stability.
- Subject-driven Generation: Qualitatively, OFT and COFT preserve subject identity and follow text prompts more accurately, whereas LoRA often loses identity and DreamBooth often misses the prompt.Examples use the same finetuned model for each method, with randomly selected outputs and best-validation-CLIP models.
- Controllable Generation: OFT and COFT converge faster than competing controllable-generation methods, with LoRA requiring 20 epochs to match OFT’s performance at epoch 8.OFT and COFT use a similar number of trainable parameters to LoRA and fewer than ControlNet; OFT also converges with 5% of ADE20K data in one example.
- Controllable Generation: Across Canny edge, segmentation-map, and landmark-to-face tasks, OFT and COFT provide stronger control consistency and qualitatively better images than the compared baselines.The evaluation uses task-specific consistency metrics, including IoU and F1 for Canny edges, accuracy metrics for segmentation, and mean ℓ2 landmark distance.
6 Concluding Remarks and Open Problems
The paper concludes that OFT improves controllability and stability with fewer finetuning parameters and no additional inference overhead, while identifying scalability, compositionality, and parameter-efficiency challenges.
- Concluding Remarks: OFT provides stronger controllability and finetuning stability with fewer finetuning parameters and no additional inference overhead.The authors describe these properties as supporting an efficient deployable model.
- Open Problems: Cayley parametrization requires a matrix inverse, limiting scalability despite block diagonal parametrization; differentiable acceleration remains open.The authors also question whether multiplying orthogonal matrices from multiple tasks preserves all downstream knowledge.
- Open Problems: Block diagonal structure improves parameter efficiency but introduces bias and limits flexibility, motivating less biased parameter-efficient designs.The paper identifies this as an important open problem.
A Experimental Details
The experiments evaluate OFT and COFT on subject-driven and controllable generation using settings aligned with DreamBooth and ControlNet. They measure controllability, generation similarity, and the effect of diagonal-block count on parameter efficiency and convergence.
- Evaluation setup: The evaluation covers subject-driven generation and controllable generation under the same task settings as DreamBooth and ControlNet.Baseline implementations were sourced from Diffusers and ControlNet repositories.
- Evaluation setup: Controllability is evaluated by measuring the difference between control signals and generated images, including canny-edge consistency for C2I.The evaluation applies the identical canny filter to generated images before comparison.
- Evaluation setup: FID quantifies similarity between image datasets using 2048-dimensional InceptionV3 features, with lower scores indicating greater similarity.The features come from the final average-pooling layer of an ImageNet-trained InceptionV3 network.
- Diagonal-block sensitivity: r controls trainable-parameter count; smaller r generally performs better, while larger r can improve efficiency and convergence on simple datasets.The authors use r = 4 across datasets and tasks as a robust trade-off.
C Experiments on Re-scaled OFT
Re-scaled OFT refines neuron magnitudes after or during orthogonal finetuning without changing their directions. This preserves hyperspherical energy while improving FID after lightweight post-stage fitting.
- Method: Re-scaled OFT finetunes neuron magnitudes while keeping their directions unchanged, using either joint or post-stage fitting.The method addresses potentially sub-optimal magnitudes after orthogonal transformations.
- Structural preservation: Because neuron directions remain unchanged, re-scaled OFT preserves hyperspherical energy and inherits OFT and COFT's structural-preservation advantage.The preservation follows from hyperspherical energy depending on neuron directions.
- Results: Magnitude fitting improves COFT's FID on the ADE20K segmentation-to-image task.The comparison uses validation images and control signals extracted from those images.
- Efficiency: Post-stage magnitude fitting adds one trainable parameter per modified layer and achieves its reported gain after one additional epoch.The parameter has shape (N × 1), where N is the number of neurons in the layer.
D Applying OFT to Convolution Layers
OFT and COFT can be applied to convolutional as well as attention linear layers. In controllable segmentation-to-image generation, convolution-only finetuning provides some control, while qualitative comparisons examine COFT, OFT, and layer choices.
- Convolution-layer adaptation: OFT is compatible with convolution layers by transforming convolutional neurons, extending beyond its attention-linear-layer application.Sharing diagonal-block parameters becomes interpretable in convolution layers.
- Convolution-layer adaptation: Convolution-only COFT finetuning achieves some control in segmentation-to-image generation using ResNet blocks of Stable Diffusion.The experiment includes quantitative results in Table 5 and qualitative results in Figure 11.
F More Qualitative Results
The appendix provides additional qualitative examples for subject-driven generation and several control-to-image tasks. These figures cover segmentation, Canny-edge, and landmark-to-face generation across varied text prompts.
- Canny-edge-to-image generation: Figures 19–21 show further OFT and COFT results for Canny-edge-to-image generation.The prompts cover rooms, buses, signs, trains, and food.
G More Controllable Generation Tasks
The paper presents qualitative comparisons for controllable generation across dense pose, sketch, and depth inputs. Additional figures provide further OFT and COFT results for these tasks.
- Dense pose to human body: Dense pose to human body generation is evaluated through qualitative comparisons among different methods.
- Dense pose to human body: Additional dense-pose results compare OFT and COFT on human-body generation.
- Sketch to image: Sketch-to-image generation is evaluated with qualitative comparisons among different methods and additional comparison results.
- Depth to image: Depth-to-image generation is evaluated with qualitative comparisons among different methods and additional comparison results.
H Human Evaluation
The human evaluation compares DreamBooth, LoRA, and OFT on subject-driven generation using participant judgments at two finetuning checkpoints. The section also reports qualitative adaptation results and failure cases for OFT and COFT.
- Human evaluation settings: 50 participants evaluated subject-driven generation using judgments of subject fidelity, text alignment, and overall image quality.The evaluation sampled seven subjects and used four prompts per subject, producing 28 subject-prompt combinations.
- Human evaluation settings: DreamBooth, LoRA, and OFT were compared at the 1000th and 10,000th finetuning iterations.The checkpoints represented typical best performance and extended-run finetuning stability, respectively.
- Human evaluation results: OFT was more favored at both evaluation checkpoints and delivered significantly better image quality and text-following ability than DreamBooth and LoRA after 10,000 iterations.
- Additional qualitative results: The paper includes qualitative results from directly finetuning Stable Diffusion with OFT on the Sketch Scene and WikiArt-caption datasets.
- Failure cases: The paper reports three failure cases each for subject-driven generation and controllable generation involving OFT and COFT.
- Failure cases: In subject-driven generation, OFT and COFT sometimes assign a text attribute to the wrong object, such as generating another red object instead of a red cat.
- Failure cases: In controllable generation, OFT and COFT sometimes hallucinate complicated structural details in large semantically uniform regions, preventing matches to the original segmentation maps.