PaperlayerView exact revision on arXiv

Source-linked AI summary

One-for-All: Generalized LoRA for Parameter-Efficient Fine-tuning

Arnav Chavan, Zhuang Liu, Deepak Gupta, Eric Xing, Zhiqiang Shen

arXiv:2306.07967v2cs.LGcs.AIcs.CV

TL;DR

Static fine-tuning may not accommodate the distinct distributions of diverse downstream datasets. GLoRA unifies adaptation across weights, features, and input tokens; it surpasses previous state-of-the-art PEFT methods in VTAB-1K average accuracy.

  • Problem

    Static fine-tuning may not accommodate downstream datasets with distinct distributions, limiting its capacity to adapt across them.

    Source: §1 · p. 1

  • Method

    GLoRA uses a unified supernet and layer-wise support tensors to adapt weights, features, and input tokens, with evolutionary search optimizing the configuration.

    Source: §1 · p. 2, §2.2 · p. 4

  • Results

    On VTAB-1K, GLoRA surpasses previous state-of-the-art PEFT methods in average accuracy and advances parameter-efficient transfer-learning results by up to 2.9%.

    Source: §1 · p. 2, §3.1 · p. 6

  • Takeaways & Limitations

    Structural re-parameterization lets GLoRA avoid extra inference parameters or FLOPs, and its weights can be loaded into deployed models without system modifications.

    Source: §4 · p. 8

    The paper identifies constraints in prior PEFT methods: some add inference cost, while LoRA is constrained to weight-space adaptation.

    Source: §2.1 · pp. 3–4

Abstract

from arXiv · show

We present Generalized LoRA (GLoRA), an advanced approach for universal parameter-efficient fine-tuning tasks. Enhancing Low-Rank Adaptation (LoRA), GLoRA employs a generalized prompt module to optimize pre-trained model weights and adjust intermediate activations, providing more flexibility and capability across diverse tasks and datasets. Moreover, GLoRA facilitates efficient parameter adaptation by employing a scalable, modular, layer-wise structure search that learns individual adapter of each layer. Originating from a unified mathematical formulation, GLoRA exhibits strong transfer learning, few-shot learning and domain generalization abilities, as it adapts to new tasks through not only weights but also additional dimensions like activations. Comprehensive experiments demonstrate that GLoRA outperforms all previous methods in natural, specialized, and structured vision benchmarks, achieving superior accuracy with fewer parameters and computations. The proposed method on LLaMA-1 and LLaMA-2 also show considerable enhancements compared to the original LoRA in the language domain. Furthermore, our structural re-parameterization design ensures that GLoRA incurs no extra inference cost, rendering it a practical solution for resource-limited applications. Code and models are available at: https://github.com/Arnav0400/ViT-Slim/tree/master/GLoRA.

1 INTRODUCTION

Heterogeneous downstream data can make static fine-tuning insufficient, while adapting large pretrained models motivates parameter-efficient methods. GLoRA generalizes LoRA through a unified approach to adapting weights, features, and input tokens, and the paper reports strong results across vision and language benchmarks.

1 INTRODUCTION

  • Distinct downstream datasets can differ in distribution and composition, and static fine-tuning may not adapt sufficiently to those variations.The paper situates this problem in the context of large pretrained models and parameter-efficient fine-tuning.Source: §1 · p. 1
  • A unified formulation spans weights, features, and input tokens, while evolutionary search over a supernet yields configurations without manual hyperparameter tuning.After training, extra parameters are fused into nearby projection weights, so the method adds no inference cost.Source: §1 · p. 2
  • GLoRA significantly outperforms prior methods across heterogeneous vision benchmarks and reports gains across vision and language evaluations with few-shot and domain-generalization tasks.The paper reports these results while requiring few extra trainable parameters and no additional inference cost.Source: §1 · p. 2
  • GLoRA extends LoRA with a generalized per-layer prompt design that can rescale and shift intermediate activations as well as adapt model weights.This broadens adaptation beyond LoRA's weight-focused approach.Source: §1 · p. 2

2 GLORA

GLoRA unifies layer-wise adaptation of weights and features, addressing limitations of methods restricted to one space or costly at inference. It combines selectable support tensors with re-parameterization and evolutionary search.

2.1 PREVIOUS SOLUTIONS WITH LIMITATIONS

  • Existing methods leave gaps: some add inference cost, while weight-only or feature-only adaptation cannot flexibly learn both weight and feature changes.VPT-Deep and AdaptFormer are described as adding inference cost; SSF is limited to feature adaptation.Source: §2.1 · pp. 3–4

2.2 A UNIFIED FORMULATION OF ONE-FOR-ALL

  • GLoRA expresses each adapted layer through trainable support tensors A–E that modify weights, inputs, prompts, and bias, with scalar, vector, low-rank, or absent forms.A multi-path supernet samples layer-wise configurations for optimization.Source: §2.2 · p. 4

2.3 STRUCTURAL RE-PARAMETERIZATION DESIGN AND INFERENCE EFFICIENCY ANALYSIS

  • GLoRA re-parameterizes learned parameters into existing weights and bias, imposing no additional computational burden during inference.The design eliminates nonlinearity between adjacent transformations to permit parameter absorption.Source: §2.3 · p. 5

2.4 EVOLUTIONARY SEARCH FOR OPTIMAL LAYER-WISE CONFIGURATIONS

  • Evolutionary search selects heterogeneous per-layer configurations and avoids manual hyperparameter tuning, although the search may increase training time.The authors describe this as an implicit search over the unified support-tensor design.Source: §2.4 · p. 5

2.5 GLORA WITH HIGHER CAPACITY

  • The complete GLoRA supernet has greater model capacity than a randomly sampled subnet because the subnet’s hypothesis space is contained within the supernet’s.The paper relates greater VC dimension to greater model flexibility and capability.Source: §2.5 · p. 5

3 EXPERIMENTS

Experiments evaluate GLoRA across vision, few-shot, domain-generalization, and language benchmarks. Results report strong performance across these settings, including state-of-the-art VTAB-1K results and gains over LoRA in language and out-of-domain evaluation.

3 EXPERIMENTS

  • Evaluation covers 19 VTAB-1K tasks, five visual few-shot datasets at 1–16 shots, four ImageNet out-of-domain datasets, and four Open LLM Leaderboard datasets.Vision experiments use ViT-B; language experiments use LLaMA-1-7B and LLaMA-2-7B. The trained supernet is searched on a validation set, and the selected subnet is evaluated on the test set.Source: §3 · pp. 5–6

3.1 RESULTS ON VTAB-1K

  • GLoRA pushes VTAB-1K state of the art by up to 2.9%, with best results on 14 of 19 datasets and its smallest model surpassing existing approaches.It remains competitive on the other VTAB-1K datasets.Source: Table 1 · p. 6, §3.1 · p. 6

3.3 FEW-SHOT LEARNING

  • Across most few-shot visual datasets, GLoRA outperforms existing methods with similar parameter counts; gains are more pronounced at higher shots but remain significant at lower shots.Flowers102 results are similar across methods, and GLoRA’s average accuracy on Food101 is on par with NOAH.Source: Figure 2 · p. 7, §3.3 · p. 7

3.2 RESULTS ON LARGE LANGUAGE MODELS

  • On LLaMA-1 and LLaMA-2, searched GLoRA models consistently outperform pretrained LLMs and corresponding LoRA variants on the Open LLM Leaderboard.The experiments tune attention layers only and use Alpaca and ShareGPT data.Source: §3.2 · p. 7, Table 2 · p. 7

3.4 DOMAIN GENERALIZATION

  • With 16-shot ImageNet tuning, GLoRA reaches 78.3% versus 83.97% for the fully scaled model; versus LoRA, performance improves by up to 100% on ImageNet-A and 50% on ImageNet-Sketch.The evaluation tests the tuned model on out-of-domain datasets.Source: Table 3 · p. 8, §3.4 · p. 8

4 ANALYSIS AND DISCUSSION

The analysis describes GLoRA’s parameter distribution and layerwise support-tensor configurations. Its structural re-parameterization is reported to preserve inference efficiency, including for VPT-Deep-level prompts.

Computational Cost

  • GLoRA adds no inference parameters or FLOPs; its structural re-parameterization also supports VPT-Deep-level prompts without computational overhead.Weights can be loaded directly into deployed base models without manual system modifications.Source: §4 · p. 8
  • On VTAB-1K, the 0.86M GLoRA parameter distribution is broken down by layer type, distinguishing MHSA linear layers from MLP linear layers.Q-K-V and Projection are in MHSA; FC1 and FC2 are in MLP.Source: Figure 3 · p. 8
  • The layerwise support-tensor configuration on VTAB-1K shows that some layers use no adaptations, while A and B have more adaptations without a distinct adaptation-type pattern.Even a scalar can serve as a support tensor, supporting parameter efficiency while adapting every linear layer.Source: Figure 4 · p. 9, §4 · p. 9

5 RELATED WORK

Related work positions LoRA as a cross-modal PEFT method with inference efficiency, while noting flexibility limits in other approaches and the motivation to extend LoRA’s adaptability.

  • LoRA transfers across modalities and tasks through direct weight-tensor adaptation, independent of model type or configuration (Hu et al.).Source: §5 · p. 9
  • Unlike Adapters and Prompt tuning, LoRA adds no inference parameters or latency through structural re-parameterization (Houlsby et al., 2019; Chen et al., 2022; Jia et al., 2022).Source: §5 · p. 9
  • RepAdapter requires architecture-specific manual design, SSF is limited to activation-space adaptation, and FacT decomposes LoRA matrices; the authors argue under 1M parameters is efficient for single-GPU fine-tuning.Source: §5 · p. 9

6 CONCLUSION

GLoRA is presented as an adaptable parameter-efficient fine-tuning approach for large pretrained vision and language models. Experiments support its superiority over existing PEFT techniques, while the conclusion identifies future directions for generalized low-rank adaptation, hybrid approaches, and search refinement.

6 CONCLUSION

  • GLoRA enhances fine-tuning and transfer learning for large-scale pretrained models through generalized low-rank adaptation and re-parameterization.The conclusion describes the approach as resource-efficient and practical for real-world applications.Source: §6 · p. 9
  • Experiments across diverse tasks and datasets show GLoRA's superiority over existing PEFT techniques, with scalability and adaptability.The conclusion also reports reductions in parameters and computation required for fine-tuning.Source: §6 · p. 9
  • Ablation studies provide insights into GLoRA's components and their relative importance.Source: §6 · p. 9
  • The conclusion points to generalized low-rank adaptation, hybrid approaches, and improved search and optimization algorithms as future work.Source: §6 · p. 9

A HYPERPARAMETERS

The appendix describes limited manual hyperparameter search for GLoRA and gives training settings for vision and language experiments. It also outlines evolutionary search for selecting layer-wise subnet configurations.

A HYPERPARAMETERS

  • GLoRA requires minimal hyperparameter adjustment apart from optimizer hyperparameters, and the experiments use AdamW.The stated practice follows prior studies.Source: A HYPERPARAMETERS · p. 14
  • Vision supernet learning-rate search considers 1e−4 and 5e−4, with dataset-specific rates reported in Table 5.Few-shot learning uses 5e−4 and ImageNet uses 1e−4.Source: A HYPERPARAMETERS · p. 14, Table 5 · p. 14
  • Language experiments use a 2e−5 learning rate, cosine annealing, and batch size 32; GLoRA trains for 15 epochs versus LoRA's 3.The authors attribute the extra GLoRA epochs to its supernet structure and contrast this with manual hyperparameter tuning for LoRA.Source: A HYPERPARAMETERS · p. 14

B EVOLUTIONARY SEARCH

  • Evolutionary search evaluates subnet validation accuracy, selects the top K as parents, and generates subsequent subnets through crossover and mutation.Source: B EVOLUTIONARY SEARCH · p. 14
  • Search begins with 50 randomly sampled subnets, then generates 50 per step for 20 vision epochs or 5 language epochs, without tuning evolution hyperparameters.Crossover and mutation probabilities are each 0.2; the authors note performance might improve with evolution-hyperparameter tuning.Source: B EVOLUTIONARY SEARCH · p. 14

C HIERARCHICAL TRANSFORMER

On the Swin-B backbone, GLoRA is reported to outperform existing works on VTAB-1K. The appendix also identifies manual design choices and hyperparameters as a subject of comparison across methods.

C HIERARCHICAL TRANSFORMER

  • On Swin-B, GLoRA uses dataset-specific learning-rate search as for ViT-B and is added to the reduction linear layer for uniformity, outperforming all existing works across tasks and datasets by a fair margin.Source: C HIERARCHICAL TRANSFORMER · p. 14
  • GLoRA outperforms existing works by a fair margin on VTAB-1K with Swin-B pretrained on ImageNet-21K.The authors apply GLoRA to Swin's reduction linear layer and report that it adapts across layers and architecture configurations.Source: C HIERARCHICAL TRANSFORMER · p. 14, Table 6 · p. 15
  • Table 7 inventories manual design choices and hyperparameters in existing works.Source: Table 7 · p. 15

D TRAINING TIME

GLoRA's automated search entails extra training time for supernet training and evolutionary search, while the described support-tensor choices create a large layer-wise subnet space. The appendix characterizes these tensors by their dimensions and roles in adapting a linear layer.

D TRAINING TIME

  • GLoRA incurs increased training time for supernet training and evolutionary search, while replacing manual design-choice search with automated evolutionary search.The authors note manual search can inflate total training time for a dataset because it spans many design choices.Source: D TRAINING TIME · p. 15

E SEARCH SPACE

  • ViT-B's 48 layers and 432 configurations per linear layer yield 20,736 possible subnets; allowing ranks 8 and 4 yields 82,944.The passage attributes comparable performance across subnets, even when not all are explored, to weight entanglement.Source: E SEARCH SPACE · p. 15

F SUPPORT TENSOR

  • For a linear layer mapping d1 to d2 dimensions, A can be a full matrix, vector, or scalar, corresponding to LoRA, vector, or scalar operations.When A is a full matrix, LoRA uses matrices Ad and Au for its low-rank representation.Source: F SUPPORT TENSOR · p. 15

G FINE-TUNED EMBEDDING VISUALIZATION

The section compares ViT-B features adapted by GLoRA and FacT on SVHN. GLoRA features form more segregated, broader, and more concentrated clusters.

G FINE-TUNED EMBEDDING VISUALIZATION

H MORE RESULTS ON FEW-SHOT LEARNING DATASETS

GLoRA performs consistently better than the listed baseline methods across five few-shot datasets and varying numbers of training examples per class.

H MORE RESULTS ON FEW-SHOT LEARNING DATASETS

Loading 2306.07967v2…