PaperlayerView exact revision on arXiv

Source-linked AI summary

SuperLoRA: Parameter-Efficient Unified Adaptation of Multi-Layer Attention Modules

Xiangyu Chen, Jing Liu, Ye Wang, Pu Perry Wang, Matthew Brand, Guanghui Wang, Toshiaki Koike-Akino

arXiv:2403.11887v1cs.CVcs.AIcs.LG

TL;DR

Large models make resource- and data-efficient fine-tuning important, while existing LoRA variants address low-rank adaptation in different ways. SuperLoRA unifies and extends those variants with flexible tensor and grouping choices; the paper reports fewer parameters than LoRA across transfer-learning tasks. Its tensor core can, however, cause parameter growth at larger ranks.

  • Problem

    Large models increase resource consumption and data requirements, while LoRA variants address the low-rank constraint of matrix factorization in different ways.

    Source: §1 · pp. 1–2

  • Method

    SuperLoRA unifies and extends LoRA variants with configurable grouping, tensor reshaping and factorization, and fixed projection for weight updates.

    Source: §1 · p. 2, LoRTA · p. 6

  • Results

    SuperLoRA demonstrates parameter efficiency in low- and extremely-low-parameter transfer learning regimes, using 3 to 10 folds fewer parameters than LoRA.

    Source: §5 · p. 14

  • Takeaways & Limitations

    The framework offers adjustable parameter requirements for fine-tuning across low- and extremely-low-parameter regimes.

    Source: §5 · p. 14

    The Tucker core tensor can cause parameter counts to grow exponentially with rank r as the tensor order M increases.

    Source: LoRTA · p. 6

Abstract

from arXiv · show

Low-rank adaptation (LoRA) and its variants are widely employed in fine-tuning large models, including large language models for natural language processing and diffusion models for computer vision. This paper proposes a generalized framework called SuperLoRA that unifies and extends different LoRA variants, which can be realized under different hyper-parameter settings. Introducing grouping, folding, shuffling, projecting, and tensor factoring, SuperLoRA offers high flexibility compared with other LoRA variants and demonstrates superior performance for transfer learning tasks especially in the extremely few-parameter regimes.

1 Introduction

Large models create resource and data demands that motivate parameter-efficient fine-tuning, including LoRA. SuperLoRA unifies and extends LoRA variants with configurable updates, and the paper reports parameter reductions in two transfer-learning tasks.

1 Introduction

  • SuperLoRA unifies and extends LoRA variants, including LoHA, LoKr, and LoTR, under different hyperparameter settings.The framework also introduces LoNKr and LoRTA as new variants.Source: §1 · p. 2
  • Grouping, tensor reshaping, and fixed projection extend the framework's options for assigning and compressing weight updates.The introduction describes reshaping updates into multidimensional tensors and splitting them into arbitrary numbers of groups.Source: §1 · p. 2
  • SuperLoRA is proposed to adapt weights across layers jointly, with adjustable parameter amounts through projected tensor rank decomposition.The schematic describes fine-tuning multi-layer attention modules at once.Source: §1 · p. 2, Figure 1 · p. 2
  • For transfer learning in image classification and image generation, the paper reports that parameters can be reduced by 3 to 10 folds across these two tasks.Source: §1 · p. 2

2 Related Work

Related work covers parameter-efficient fine-tuning and LoRA's low-rank assumptions, alongside variants that alter the factorization or jointly adapt model updates. SuperLoRA is distinguished by its flexible grouping and tensor-decomposition choices.

2 Related Work

  • Parameter-efficient fine-tuning freezes base weights and trains adapter parameters, while LoRA approximates weight corrections through low-rank decomposition.These approaches are presented as useful for reducing fine-tuning resource and data demands.Source: §2 · pp. 2–3
  • LoHA, KronA, and LoKr alter how weight-update splits are combined, while LoDA introduces nonlinearity.LoHA uses a Hadamard product, KronA a Kronecker product, and LoKr extends KronA to convolutional layers.Source: §2 · p. 3
  • LoTR jointly approximates model-wide updates while preserving weight geometry; SuperLoRA instead permits arbitrary grouping and couples Tucker decomposition with fixed projection.With reshaping, LoRTA can be applied to any weight shape.Source: §2 · pp. 3–4
  • A filter-atom method works on convolution kernels separately, whereas SuperLoRA considers the model jointly and uses fastfood projection for atom coefficients.The passage characterizes fastfood projection as faster and simpler than matrix factorization.Source: §2 · p. 4
  • Local LoRA reduces fine-tuning memory by processing model groups sequentially; SuperLoRA instead focuses on assigning LoRA structure within groups.The passage describes SuperLoRA as a viable extension of Local LoRA.Source: §2 · p. 4

3 Methodology

LoRA represents each weight update as a low-rank product added to frozen weights. SuperLoRA generalizes this approach by jointly adapting weights across layers and adding adjustable grouping, tensor decomposition, and projection choices.

3.1 Low-Rank Adaptation (LoRA)

  • LoRA approximates each weight update as a product of two low-rank matrices, then adds that update to the frozen weight matrix.With rank r smaller than the matrix dimensions, LoRA uses (din + dout)r parameters per weight matrix rather than dindout for dense fine-tuning.Source: §3.1 · p. 4

3.2 SuperLoRA

  • SuperLoRA concatenates updates across layers, divides them into adjustable groups, and assigns each group's update to corresponding weight-matrix elements.The described targets include query and value projection weights across attention modules.Source: §3.2 · p. 5
  • SuperLoRA can reshape grouped updates as high-order tensors and use Tucker decomposition, combining a core tensor with mode-specific plane factors.The framework also allows groups to combine dense fine-tuning, LoRA, and high-order Tucker decomposition choices.Source: LoRTA · pp. 5–6, §3.2 · p. 5
  • A projection function can map a smaller LoRA-module update to a larger grouped update; its fastfood form uses fixed random transforms and needs no extra parameters with a predetermined seed.A nonlinear function such as tanhshrink may be added, and projection weights are reproduced from a known seed.Source: LoRTA · pp. 6–7
  • Shuffling uses a random permutation without compression, while LoNKr extends LoKr by combining more than two sub-LoRA splits through Kronecker products.When K = 2, LoNKr reduces to LoKr with additional flexibility; adjustable group size allows adapting multiple attention modules at once.Source: §3.2 · p. 5, LoRTA · p. 7

4 Empirical Experiments

The experiments evaluate SuperLoRA on image classification and image generation transfer tasks, comparing accuracy or image quality against trainable parameter counts. They also visualize generated images across parameter regimes.

4.1 Classification transfer task

  • 3–4× parameter efficiency at the same CIFAR100 accuracy is reported for group-wise SuperLoRA versus weight-wise LoRA.Figure 4 compares classification accuracy and parameter counts.Source: Figure 4 · p. 8, Results · p. 8
  • 10-fold fewer parameters give 3D LoRTA comparable 0.9 accuracy to the lowest-rank LoRA; it also improves accuracy by about 1% at comparable parameter counts.The ImageNet1k-to-CIFAR10 transfer experiment reports similar 3–10-fold improved parameter efficiency.Source: Figure 4 · p. 8, Results · pp. 8–9

4.2 Image generation transfer task

  • At low parameter counts, group1 has the best IS; reshaping also improves IS and reduces parameters versus corresponding non-reshaped groups.At middle parameter counts, groups G=8 and 12 show slightly better IS than LoRA.Source: Figure 5 · p. 10, Settings · p. 10
  • Below 5,000 parameters, LoNKr has more data points and better IS; its least-parameter setting drops from 500 to 150 versus LoKr.More splits provide more choices in low-parameter regimes, especially for group-wise LoNKr.Source: Figure 6 · p. 11, LoKr vs. LoNKr · p. 11
  • Under 80 parameters, group-wise 5D LoRTA produces a result; for LoRA-comparable IS, 3D LoRTA uses 2 × 10^3 versus LoRA’s 10^4 parameters.That comparable-IS setting is a 5-fold parameter reduction.Source: Figure 6 · p. 11, LoKr vs. LoNKr · p. 11
  • About 30 parameters suffice with projection, versus 10,000 for original LoRA; shuffling does not harm IS and improves it for SuperLoRA (2D) in most cases.Linear projection performs better than nonlinear projection; linearv2 is better in higher-parameter areas, while linear works better in lower-parameter areas.Source: Figure 7 · p. 12, Projection effect · p. 12, Shuffling effect · p. 12

4.3 Visualization

  • At high parameter counts, generated images are close to MNIST images; at middle counts, colorful backgrounds and missing digits occur occasionally.At low and extremely low counts, more missing digits and colorful backgrounds appear, though some SVHN-to-MNIST transfer is realized with as few as 31 parameters.Source: Figure 8 · p. 13, Figure 9 · p. 13, §4.3 · p. 13

5 Conclusion

SuperLoRA unifies and extends LoRA variants, including LoKr and LoTR, and offers flexible hyperparameters for transfer learning. The experiments report promising parameter efficiency, while future work includes projection functions and broader model and task applications.

5 Conclusion

  • SuperLoRA generalizes and extends LoRA variants, including LoKr and LoTR, and introduces LoNKr and LoRTA.Its hyperparameters include factorization rank, projection function and ratio, group count, tensor order, and Kronecker splits.Source: §5 · p. 14
  • Across two transfer-learning experiment types, SuperLoRA shows promising parameter efficiency and reduces parameters by 3–10-fold versus LoRA.Source: §5 · p. 14
  • Future work includes studying projection functions for greater efficiency in extremely low-parameter regimes and applying the method to other tasks and large models such as LLMs.Source: §5 · p. 14

A.1 Illustration of ViT model in detail

The classification experiments use an ImageNet21k-pretrained ViT-base and fine-tune only query and value projection layers in its self-attention modules.

A.1 Illustration of ViT model in detail

  • The ViT-base model has 12 layers and dimension 768.Source: §A.1 · p. 18
  • Only query and value projection layers in the self-attention modules are fine-tuned.Source: §A.1 · p. 18
  • The ViT-base model has 86.6M total parameters.Source: §A.1 · p. 18

A.2 Illustration of diffusion model in detail

The image-generation experiments use a classifier-free diffusion model with a U-Net containing 21 attention modules, and fine-tune query and value projections in those modules.

A.2 Illustration of diffusion model in detail

  • The U-Net contains 21 attention modules.Source: §A.2 · p. 18
  • The attention modules have either 64 or 128 input and output channels.Source: §A.2 · p. 18
  • The U-Net base model has 10.42M parameters, including 300 class-embedding parameters.Source: §A.2 · p. 18

A.3 Illustration of grouping mechanism

The grouping mechanism ranges from separate weight-wise updates to joint adaptation across attention modules. Reshaping, arbitrary group boundaries, projection, and tensor folding relax the original weight geometry.

A.3 Illustration of grouping mechanism

  • Weight-wise grouping represents each value or query correction separately with a rank-r decomposition.Layer-wise grouping instead adapts both projections within each layer jointly.Source: §A.3 · p. 18
  • Reshaping stacked weights into a regular shape avoids inefficient decomposition of unbalanced arrays and allows groups to cross weight boundaries.Projection can shuffle and mix elements; LoRTA extends folding to arbitrary-dimensional tensors using Tucker decomposition.Source: §A.3 · p. 19

A.4 Geometric analysis

The geometric analysis compares SuperLoRA variants with dense fine-tuning using singular-vector similarities and Euclidean distance. SuperLoRA shows lower distance, especially with reshaping, while retaining some subspace similarity.

A.4 Geometric analysis

  • SuperLoRA with reshaping reached 0.9218 accuracy with 165,572 parameters, versus LoRA's 0.9113 with 147,456 parameters.SuperLoRA without reshaping reached 0.9170 with 115,200 parameters; reshaping plus projection reached 0.9213 with 138,372 parameters.Source: §A.4 · p. 19
  • SuperLoRA's Euclidean distance from dense fine-tuning decreases significantly, especially with reshaping, while query weights show higher right-singular similarity than LoRA.The analysis uses the top five principal singular vectors and considers query and value projections across 12 ViT attention modules.Source: §A.4 · p. 20, Figure 14 · p. 21
  • Reshaped SuperLoRA tends to preserve higher right-singular similarity, whereas LoRA tends to preserve higher left-singular similarity.With reshaping, right-singular similarities are much larger for value weights than query weights.Source: §A.4 · p. 21

A.5 Grouping effect on SuperLoRA (1D, dense, with projection)

The number of groups affects projection size and accuracy; jointly updating attention modules performs better than using a projection matrix that is too small.

A.5 Grouping effect on SuperLoRA (1D, dense, with projection)

  • One group achieves the best overall accuracy; four or eight groups are comparable to a smaller projection ratio, while 12 groups sharply reduces accuracy.The projection matrix is shared across groups, so more groups make it smaller.Source: §A.5 · p. 21

A.6 Linear vs. nonlinear projection

Appendix experiments compare projection variants and transfer-learning settings, including grouping, reshaping, and tensor-factorization choices. Results vary by setting, with several reported advantages in accuracy or parameter range and caveats about metric behavior.

A.6 Linear vs. nonlinear projection

  • With the same number of parameters, linear projection outperforms nonlinear variants in most tested cases.The nonlinear variants apply tanhshrink after fixed linear projection.Source: §A.6 · pp. 21–22, Figure 17 · p. 23

A.7 Transfer learning from ImageNet1k to CIFAR10

  • ImageNet1k-to-CIFAR10 transfer reaches accuracy close to 0.99, and SuperLoRA outperforms LoRA in accuracy and parameter range.Reshaped SuperLoRA uses at least threefold fewer parameters than LoRA; projected SuperLoRA improves accuracy by about 1% over the lowest-rank LoRA at around 0.97 accuracy, while LoRTA can reduce parameters tenfold at comparable accuracy.Source: §A.7 · p. 24, Figure 18 · p. 24

A.8 Transfer learning from SVHN to MNIST

A.9 Transfer learning from MNIST to SVHN

  • For MNIST-to-SVHN transfer, some metrics have ill-posed cases, while SuperLoRA extends LoRA into the low-parameter regime and performs better occasionally on several metrics.Reshaping performs better than non-reshaping in most cases; LoNKr extends LoKr to a lower-parameter regime with better MSID, and LoRTA reduces required parameters from 10^4 to 10^2 compared with LoRA.Source: Grouping effect · p. 26, Reshaping effect · p. 26, SuperLoRA (LoNKr) · p. 27

A.10 Effect of groups in LoNKr and LoRTA

  • In the low-parameter regime, fewer groups benefit LoNKr, while LoRTA prefers fewer groups; the methods are comparable for LoNKr in the high-parameter regime.The group effects are shown for LoNKr and LoRTA in Figures 27 and 28.Source: §A.10 · p. 27, Figure 27 · p. 32, Figure 28 · p. 33

A.11 Effect of split K in LoNKr

Related papers

Loading 2403.11887v1…