Parameter-Efficient Orthogonal Finetuning via Butterfly Factorization

1 INTRODUCTION

• OFT preserves pairwise angles between neurons through a shared orthogonal transformation, but its high-dimensional matrices can require many trainable parameters.Its block-diagonal structure reduces parameters but limits expressivity and leaves the choice of a good sparsity pattern unresolved.Source: §1 · pp. 1–2
• The information-transmission framework treats dense-matrix construction as information flow across a grid-structured graph, guiding sparse factorizations that retain expressivity.The framework trades additional computation during training for fewer trainable parameters.Source: §1 · p. 2
• BOFT uses butterfly factorization to form a dense orthogonal matrix from O(log d) sparse matrices with O(d) non-zero entries each, requiring O(d log d) parameters.It subsumes OFT as a special case and provides a generalized orthogonal finetuning framework.Source: §1 · p. 2
• BOFT’s butterfly structure interpolates between full orthogonal matrices and identity matrices, with structural priors intended to balance expressivity and regularity.The authors argue that butterfly structures can recover classic linear transforms that OFT’s block-diagonal structure cannot.Source: §1 · p. 2
• Across vision and language adaptation applications, the authors report that BOFT outperforms current state-of-the-art methods by a considerable margin.They present this as evidence of BOFT’s parameter efficiency and generalization ability.Source: §1 · p. 2

Parameter-efficient finetuning (PEFT)

• LoRA updates pretrained weights additively with low-rank matrices, while OFT uses an orthogonal transformation and has generally required more trainable parameters.Prior OFT work reported stronger generalization and more stable training than LoRA for text-to-image diffusion models.Source: §2 · p. 3
• BOFT applies butterfly factorization to improve OFT’s parameter efficiency, addressing open questions about OFT’s use beyond controllable text-to-image generation.The paper applies orthogonal finetuning to broader adaptation tasks.Source: §2 · p. 3

Butterfly structures

• Prior work uses butterfly structures for fast linear transforms, neural-network training, kernel approximation, matrix-vector multiplication, and network transmission.The paper’s focus is using them to improve OFT’s parameter efficiency.Source: §2 · p. 3

3 AN INFORMATION TRANSMISSION VIEW ON ORTHOGONAL FINETUNING

• OFT reparameterizes a pretrained weight matrix by left-multiplying it with a learnable orthogonal matrix, preserving pairwise angles, while its block-diagonal form transforms dimension groups separately.The authors argue that grouping dimensions by index is undesirable and ask whether dense orthogonal matrices can be constructed without losing parameter efficiency.Source: §3 · p. 3
• The information-transmission view represents a product of sparse matrices as successive graph connections, where a nonzero matrix entry indicates a possible path for information.Dense connectivity requires every source node to reach every destination node across the factorization layers.Source: §3 · p. 4
• A suitable factorization seeks dense connectivity with as few trainable edges as possible, subject to orthogonality; OFT’s block-diagonal structure cannot construct a dense matrix.The framework distinguishes necessary, non-trainable edges from trainable ones.Source: §3 · p. 4
• For d = 4, the illustrated feasible factorization uses 10 edges, while the butterfly network achieves dense connectivity with 8.A single dense orthogonal matrix would have d² − d trainable edges.Source: §3 · pp. 4–5

4 ORTHOGONAL PARAMETERIZATION BY BUTTERFLY FACTORIZATION

• BOFT constructs its orthogonal matrix as a product of orthogonal butterfly components, with each component built by arranging parameterized orthogonal blocks in butterfly patterns.The component patterns are permutations of a block-diagonal pattern, and the paper uses Cayley transforms or 2D rotations to ensure orthogonality.Source: §4 · p. 5
• BOFT(1,b) reduces to block-diagonal OFT with block size b, while BOFT(1,d) recovers OFT with a full orthogonal matrix.The full butterfly depth can yield a dense orthogonal matrix.Source: §4 · p. 5
• BOFT uses 1/2(b−1)dm effective trainable parameters for a d×n linear layer, compared with O(d²) for full OFT and O(bd) for block-diagonal OFT.With m = log d and b = 2, BOFT uses O(d log d) parameters and can produce a dense matrix.Source: §4 · p. 5
• BOFT initializes its butterfly components as identity matrices so finetuning starts from the pretrained model.In the Cayley transform, this corresponds to initializing the skew-symmetric matrix to zero.Source: §4 · p. 6
• BOFT’s multiplicative dropout randomly replaces selected butterfly components and diagonal blocks with identity matrices.The method is proposed because conventional dropout does not naturally fit BOFT’s multiplicative weight update.Source: §4 · p. 6

Expressivity of BOFT

• BOFT(9,2) has better expressiveness than BOFT(1,16) with far fewer parameters, and BOFT(6,4) has similar approximation error to BOFT(2,16) with far fewer parameters.The reported simulation approximates a random 512 × 512 orthogonal matrix; smaller b and larger m are generally more parameter-efficient.Source: §5 · p. 6
• With m1=m2=log d, b1=b2=2, and l=d−1, the generalized BOFT composition can represent the entire orthogonal group, but greater expressivity does not always improve finetuning performance.The paper identifies a trade-off between expressivity and regularity as key to finetuning generalizability.Source: §5 · p. 6

Spectral properties

• OFT and BOFT preserve the pretrained weight matrix’s spectral norm under left multiplication by an orthogonal matrix, which the paper relates to training stability and generalization.The spectral norm is the largest singular value; the passage says this preservation has been shown to benefit stability and generalization [58] [90].Source: §5 · p. 6

Orthogonal finetuning as learning bilinear similarity

• BOFT can be viewed as learning an orthogonality-constrained bilinear similarity matrix, connecting it to distance metric learning and bilinear forms.Source: §5 · p. 6

Inductive bias and generalization in BOFT

• Because BOFT’s structured subset of the orthogonal group constrains the hypothesis class, it naturally induces an inductive bias.Source: §5 · p. 6

Comparison to butterfly-based sparse training

• Unlike butterfly-based sparse training that reparameterizes weights directly, BOFT transforms weights with layer-shared matrices for finetuning.The cited comparison describes prior approaches as training from scratch, with one cited work projecting pretrained weights onto a butterfly-matrix variant before optimization [13] [14] [15].Source: §5 · p. 7

6.1 ADAPTATION OF LARGE LANGUAGE MODELS (LLMS)

• BOFT excels on GSM8K and MATH, achieving over 10% relative improvement over LoRA on MATH with half as many trainable parameters and outperforming OFT at the same effective block number.The authors attribute the OFT comparison to a generalizable inductive bias from the butterfly structure.Source: Table 3 · p. 7, §6.1 · p. 7

6.2 ADAPTATION OF VISION FOUNDATION MODELS

• BOFT with m = 4, b = 4 achieves the best overall VTAB-1K accuracy, with performance marginally below full finetuning on only three tasks.The three exceptions are dSpr-Ori (−0.7%), Caltech101 (−0.3%), and sNORB-Ele (−0.1%).Source: Table 4 · p. 8, §6.2 · p. 8
• BOFT-SAM matches HQ-SAM using only 3% of its trainable parameters and retains SAM’s inference speed.BOFT is applied to the mask decoder’s linear layers while the rest of SAM remains frozen; unlike HQ-SAM, BOFT-SAM adds no inference modules.Source: Table 5 · p. 8, §6.2 · p. 9

6.3 CONTROLLING TEXT-TO-IMAGE DIFFUSION MODELS

• With fewer than 3M parameters, BOFT with r = 32, m = 2 performs significantly better than LoRA and OFT with block number 16 on controllable generation.For denser orthogonal matrices, BOFT with r = 8, m = 4 achieves the best performance and significantly outperforms LoRA with a similar parameter count.Source: Table 6 · p. 9, §6.3 · p. 9
• Increasing the number of butterfly components m at fixed r = 32 improves control performance and generally leads to faster, more stable convergence.The ablation varies m from 0, equivalent to OFT at block number 32, to 6, yielding a dense orthogonal matrix.Source: Figure 8 · p. 9, §6.3 · p. 9
• In subject-driven generation, BOFT combines strong subject-identity preservation with accurate text-prompt following, while OFT favors identity and LoRA shows better prompt following.The authors report that BOFT generally captures more intrinsic identity characteristics, producing images they describe as more plausible in subject identity.Source: §6.3 · pp. 9–10, Figure 9 · p. 10
• Setting trained butterfly components to identity without retraining produces plausible images that smoothly interpolate from landmark-controlled outputs toward uncontrolled Stable Diffusion outputs.Figure 10 shows the component-by-component interpolation for BOFT with m = 5 and r = 16.Source: Figure 10 · p. 10, §6.3 · p. 10

7 CONCLUDING REMARKS AND LIMITATIONS

• BOFT parameterizes a dense orthogonal matrix as a product of sparse orthogonal matrices, using a graphical information-transmission framework to identify butterfly structure as an effective factorization.The authors report empirical effectiveness in finetuning large language models, vision models, and text-to-image generative models.Source: §7 · pp. 10–11
• BOFT’s training runtime is slightly larger than OFT’s, and improving it remains open; after finetuning, its learned matrices can be merged into pretrained weights without additional inference latency.The authors also leave open whether butterfly networks are the most efficient information-transmission structures.Source: §7 · p. 11

A.2 TRANSFER LEARNING ON VTAB-1K

• The VTAB-1K appendix uses DINOv2-large and matches training setups across methods, with best final test Top-1 accuracy reported after the final epoch.The setup uses 30 epochs, batch size 64, AdamW, and a cosine learning-rate schedule; GLoRA is also tested for 100 epochs.Source: §A.2 · p. 19, Table 8 · p. 19

A.3 EXPERIMENTAL DETAILS IN LLAMA FINETUNING

• The Llama appendix specifies separate language-understanding and math-question-answering settings and evaluates MMLU zero- and five-shot, and math QA zero-shot.The two tasks use a batch size of 64 and two training epochs; their learning rates are 1e-4 and 8e-4, respectively.Source: §A.3 · pp. 19–20

A.4 EXPERIMENTAL DETAILS IN SAM FINETUNING

• For SAM finetuning, the appendix follows HQ-SAM’s training and evaluation settings, using a 0.0005 learning rate and keeping SAM frozen during finetuning.The stated setup also uses AdamW, cosine annealing, weight decay 0.01, and multiplicative dropout 0.005.Source: §A.4 · p. 20

A.5 EXPERIMENTAL DETAILS IN CONTROLNET AND DREAMBOOTH

• ControlNet experiments use a lightweight eight-convolution image encoder and train it with attached UNet PEFT layers on ADE20K S2I and CelebV-HQ landmark-to-face tasks.Both tasks run for 20 epochs with batch size 16; the reported setup uses a 1e-5 learning rate.Source: §A.5 · p. 20
• DreamBooth finetunes UNet cross-attention layers rather than the text transformer, using 2000 steps and prior-preservation training with 200 generated class images.The appendix points to further subject-driven generation examples in Figures 9–15 and ControlNet results in Figures 16–17.Source: §A.5 · p. 20
• On S2I, BOFT’s best performance across parameter budgets from 2M to 20M is similar to OFT’s, but BOFT achieves better control performance with fewer parameters.The authors characterize S2I as generally easier than landmark-to-face generation and requiring less finetuning flexibility.Source: Table 9 · p. 20, §A.5 · p. 20

B MORE QUALITATIVE RESULTS ON HIGH-QUALITY SEGMENTATION

• Figure 11 provides an additional qualitative comparison of mask prediction between SAM and BOFT-SAM.Source: Figure 11 · p. 21

C MORE QUALITATIVE RESULTS IN SUBJECT-DRIVEN GENERATION

• Figures 12–15 present qualitative comparisons of subject-driven generation.Source: Figure 12 · p. 22, Figure 13 · p. 23, Figure 14 · p. 24

D MORE QUALITATIVE RESULTS IN CONTROLLABLE GENERATION

• Figures 16–17 show additional qualitative results for landmark-to-face and segmentation map-to-image generation.Source: Figure 16 · p. 26, Figure 17 · p. 27

• BOFT returns 4 change for the cookie purchase (Llama-2: $6), while both answers give 240 monthly sheets for Miguel.Source: E MATHEMATICAL QUESTION-ANSWERING CASE… · p. 28
• BOFT answers the friends problem with 120 (Llama-2: 100) and the sprint problem with 540 (Llama-2: 180).Source: E MATHEMATICAL QUESTION-ANSWERING CASE… · p. 28
• BOFT gives 21 for Andrea’s savings (Llama-2: 11) and $110 for beef cost (Llama-2: $137.50).Source: E MATHEMATICAL QUESTION-ANSWERING CASE… · p. 28

F PROOF OF THEOREM 1

• BOFT is more expressive than OFT at the same block size for m>1, reduces to OFT at m=1, and RExt can represent any orthogonal matrix.Source: F PROOF OF THEOREM 1 · p. 29

G MATHEMATICAL PROPERTIES OF BUTTERFLY MATRICES

• Butterfly matrices are orthogonal-matrix subsets whose additional inductive biases further regularize finetuned models.Source: G MATHEMATICAL PROPERTIES OF BUTTERFLY… · p. 30

G.1 BALANCED ENTRY-WISE LEARNING RATE

• Products of all-one block butterfly components retain balanced entry-wise learning rates, but identity initialization need not make component elements equal, and general sparse factorizations lack this guarantee.Source: §G.1 · p. 30

G.2 AN ALTERNATIVE DEFINITION OF ORTHOGONAL BUTTERFLY MATRICES

• The appendix defines orthogonal butterfly matrices recursively from generalized rotations, distinguishes simple from non-simple forms, and notes that the main paper uses non-simple diagonal butterflies.Source: §G.2 · pp. 30–31

G.3 TOPOLOGICAL PROPERTIES

• OB(N) and OBs(N) are compact spaces in SO(N) homeomorphic to quotients of tori, and OBs(N) is also a compact abelian subgroup.Source: §G.3 · p. 31

G.4 INPUT SENSITIVITY

• The stated perturbation bounds show that the map from butterfly parameters θ to OB(θ) is Lipschitz continuous.Source: §G.4 · p. 32

G.5 RANDOM ORTHOGONAL BUTTERFLY MATRICES

• Random orthogonal butterfly matrices are generated from random generalized rotations using an independent sequence of matrix pairs.Source: §G.5 · p. 32

H ILLUSTRATION OF BLOCK BUTTERFLY MATRICES

• Figure 18 illustrates block butterfly components and matrices, including a 16-dimensional matrix with block size 1 and an 8-dimensional matrix with block size 2.Source: Figure 18 · p. 33, H ILLUSTRATION OF BLOCK BUTTERFLY… · p. 33

• The comparison obtains 10 finetuned 1024 × 1024 dense orthogonal matrices using 10 random seeds.Source: I MORE RESULTS ON EXPRESSIVENESS OF BOFT · p. 34
• The expressiveness evaluation follows the same experiment as Figure 5, with Figure 19 presenting the dense-matrix comparison.Source: I MORE RESULTS ON EXPRESSIVENESS OF BOFT · p. 34, Figure 19 · p. 34
• BOFT yields better expressiveness under the same number of parameters in an experiment using 10 learned dense orthogonal matrices from DINOv2 finetuning.Source: Figure 19 · p. 34, I MORE RESULTS ON EXPRESSIVENESS OF BOFT · p. 34