Source-linked AI summary
VeRA: Vector-based Random Matrix Adaptation
Dawid J. Kopiczko, Tijmen Blankevoort, Yuki M. Asano
TL;DR
Large language models make personalized and multi-task finetuning storage-intensive, while existing LoRA adaptations still require substantial trainable state. VeRA shares frozen random matrices across layers and learns small scaling vectors instead, achieving large parameter reductions with comparable downstream performance and no added inference-time cost.
Problem
Personalized and multi-task finetuning can require storing many adapted model checkpoints, while LoRA still introduces substantial memory overhead.
Method
VeRA freezes a single pair of random matrices shared across layers and uses trainable scaling vectors for layer-wise adaptation.
Results
Ten-fold fewer parameters preserve the same performance on GLUE for RoBERTalarge and image classification, while E2E achieves a three-fold reduction.
Takeaways & Limitations
VeRA is suited to frequent swapping of many finetuned models, allowing numerous versions to reside in limited single-GPU memory.
Takeaways & Limitations
The GLUE experiments omit MNLI and QQP because of time and budget constraints, and therefore omit the MNLI trick for MRPC, RTE, and STS-B.
Abstract
from arXiv · showhide
Low-rank adapation (LoRA) is a popular method that reduces the number of trainable parameters when finetuning large language models, but still faces acute storage challenges when scaling to even larger models or deploying numerous per-user or per-task adapted models. In this work, we present Vector-based Random Matrix Adaptation (VeRA), which significantly reduces the number of trainable parameters compared to LoRA, yet maintains the same performance. It achieves this by using a single pair of low-rank matrices shared across all layers and learning small scaling vectors instead. We demonstrate its effectiveness on the GLUE and E2E benchmarks, image classification tasks, and show its application in instruction-tuning of 7B and 13B language models.
1 INTRODUCTION
Large-model personalization creates severe storage demands, while LoRA still leaves substantial memory overhead. VeRA addresses this by adapting shared frozen random matrices with trainable scaling vectors, reducing trainable parameters without added inference time and with comparable results.
- Motivation: Personalized assistants may require storing many finetuned checkpoints, causing storage needs to escalate across users and tasks.The paper highlights cloud-based assistants that continuously adapt to individual behavior and feedback.
- Motivation: 288MB for one LoRA-adapted GPT-3 model would scale to 275TB for one million finetuned weights.The example assumes rank 16 applied to GPT-3 query and value layers, stored in single precision.
- Motivation: Lower intrinsic dimensionality than LoRA’s typical parameter usage suggests room for more efficient adaptation methods.Prior studies reported intrinsic dimensions much lower than the trainable parameters used in LoRA.
- Proposed approach: VeRA adapts a pair of frozen random matrices shared across layers using trainable scaling vectors, minimizing introduced trainable parameters.The approach is intended to let more adapted model versions fit within a single GPU’s limited memory.
- Contributions: VeRA reduces trainable parameters relative to LoRA while yielding comparable results and no additional inference-time cost.The method merges its adaptations into the original weights, and the paper evaluates it across GLUE, E2E, instruction-following, and image classification tasks.
- Contributions: The study includes an ablation analysis of VeRA’s individual components and their effects on performance.The ablation examines how component choices influence results.
2 RELATED WORK
Related work motivates VeRA through two observations: existing parameter-efficient methods still use more trainable parameters than intrinsic dimensionality may require, and frozen random projections can retain useful model capacity. The paper positions VeRA as an alternative route to substantial parameter reduction.
- Parameter-efficient adaptation: LoRA reduces finetuning costs with low-rank updates, while AdaLoRA dynamically reallocates rank by pruning less important components.AdaLoRA targets parameter allocation across layers using an importance metric.
- Parameter efficiency: RoBERTabase has reported intrinsic dimension d901 of 896, compared with 0.3M trainable parameters used by LoRA.The intrinsic-dimension definition is the smallest dimension reaching 90% of the full training metric.
- Parameter efficiency: Prior work suggests further parameter reduction may be possible, even while tolerating marginal performance degradation.This motivates approaches different from dynamic rank allocation.
- Random models and projections: Randomly initialized networks and random projections have supported high-performing subnetworks or effective training with few parameters.The cited studies provide empirical and theoretical motivation for using frozen random matrices in finetuning.
- VeRA positioning: VeRA’s schematic contrasts trainable LoRA matrices with frozen, shared matrices adapted by scaling vectors, while both can merge updates into W without added latency.The figure presents the method as a parameter-reduced adaptation of LoRA.
3 METHOD
VeRA reparameterizes LoRA-style updates by sharing frozen random matrices across layers and learning layer-specific scaling vectors. This reduces storage and trainable-parameter growth while preserving the ability to merge adaptations into the original weights.
- 3 METHOD: VeRA freezes one randomly initialized matrix pair shared across adapted layers and learns scaling vectors for layer-wise adaptation.The scaling vectors and frozen matrices can be merged into the original weights, eliminating additional inference latency.
- LoRA formulation: LoRA constrains the update to BA, with trainable matrices A and B and bottleneck rank r.A has shape m × r and B has shape r × n, where r is much smaller than the input dimensions.
- VeRA formulation: VeRA replaces trainable A and B with frozen random shared matrices and trainable diagonal scaling vectors Λb and Λd.The scaling vectors can scale or disable rows and columns, enabling layer-wise adaptation.
- 3.2 PARAMETER COUNT: VeRA’s trainable-parameter count is Ltuned × (dmodel + r), compared with LoRA’s 2 × Ltuned × dmodel × r.At r = 1, VeRA requires approximately half as many trainable parameters as LoRA, and its count grows much more slowly with rank.
- Memory footprint: Frozen random matrices need not be stored because they can be regenerated from an RNG seed, leaving only the trained vectors and one seed.The resulting trained-adjustment memory footprint is compared with LoRA in Table 1 for RoBERTabase, RoBERTalarge, and GPT-3.
- Initialization: Kaiming initialization is used for VeRA’s frozen matrices, while b starts at zero and d starts at one in the illustrated setup.Alternative uniform matrix and non-zero d initializations are also explored experimentally.
4 EXPERIMENTS
The experiments evaluate VeRA against LoRA and other parameter-efficient baselines across language, instruction-following, and image-classification tasks, while also probing rank scaling and component choices. Across these settings, VeRA achieves comparable or better performance with substantially fewer trainable parameters.
- GLUE benchmark: VeRA performs competitively with LoRA on GLUE for RoBERTa-base and RoBERTa-large while using an order of magnitude fewer parameters.The GLUE evaluation applies VeRA to query and value projections and fully trains the classification head.
- E2E benchmark: VeRA outperforms LoRA on E2E with 3 and 4 times fewer trainable parameters for GPT-2 Medium and Large, respectively.Results are reported from the final training epoch under the benchmark setup described for each model.
- Instruction tuning: VeRA closely matches LoRA in instruction tuning despite a 100x reduction in trainable parameters, using 1.6M and 2.4M parameters for 7B and 13B models.The corresponding LoRA configurations use 159.9M and 250.3M trainable parameters.
- Rank scaling: At equal parameter counts on RTE, higher-rank VeRA outperforms standard LoRA by 4 accuracy percentage points.Figure 2 compares the methods while varying their ranks.
- Ablation study: Ablations show that removing either scaling vector compromises performance, Kaiming initialization outperforms uniform-range initialization, and d initialization at 10^-1 or 10^-7 outperforms 1.0.Sharing random matrices preserves mean performance on RTE and STS-B, with slight gains from unique matrices on MRPC and CoLA.
5 CONCLUSION
VeRA substantially reduces trainable parameters relative to LoRA while achieving similar or better downstream performance across language and vision tasks. It is especially suited to serving many personalized finetuned models, though its applicability beyond Transformer-based language and vision models remains open.
- Ten-fold fewer parameters preserve performance on GLUE for RoBERTalarge and on image classification, while E2E requires three-fold fewer parameters.
- VeRA supports frequent swapping of numerous finetuned models because its small scaling vectors allow many versions to reside in one GPU’s limited memory.
- The study covers language and vision models with Transformer architecture; applicability to other architectures and domains remains future work.
A HYPERPARAMETERS
The appendices document hyperparameter configurations for GLUE, instruction tuning, E2E, and ViT image classification. GLUE settings reuse several LoRA configurations, while the available compute limited hyperparameter searches.
- GLUE configurations vary by model size, with optimizer, warmup ratio, and learning-rate schedule taken from Hu et al. (2022).
- Compute limitations prevented full GLUE grid searches; only learning rates and epoch counts were evaluated, while other settings relied on LoRA configurations.
- The appendices provide hyperparameter configurations for instruction tuning and the E2E benchmark.
- ViT image-classification configurations use separate learning rates for the classification head and the remaining model parameters.
B RELATIVE PERFORMANCE GAIN.
Relative-performance analysis on RTE with RoBERTalarge measures performance gains per 1K trainable parameters against a classification-head-only baseline. VeRA provides the highest normalized gain, and higher-rank VeRA can outperform LoRA at equal parameter counts.
- The analysis focuses on RTE with the RoBERTalarge model and normalizes gains by additional trainable parameters.
- VeRA yields the highest performance gain per 1K trainable parameters relative to the classification-head-only baseline.
C IMPACT ON TRAINING TIME AND MEMORY USAGE
Compared with LoRA at rank 64 for LLaMA 7B instruction tuning, VeRA modestly increases training time while reducing GPU memory usage. The memory reduction reflects avoiding optimizer states and gradients for shared random matrices.
- The comparison evaluates LoRA and VeRA using the same rank of 64 on the instruction-tuning dataset.
- 7.4% lower GPU memory usage is observed with VeRA than LoRA during rank-64 LLaMA 7B instruction tuning.
- 1.8% higher training time results from VeRA’s additional vector multiplications in the forward pass.
D SIMILARITIES OF TRAINED WEIGHTS
The study compares LoRA and VeRA weight differences across adapted layers using cosine similarity, finding greater similarity in later layers. This supports the view that VeRA can approximate LoRA-trained weights.
- At rank 64, the study flattened LoRA and VeRA weight differences across query layers and calculated their cosine similarity.LoRA differences multiplied two low-rank matrices, while VeRA additionally multiplied by scaling vectors.
- Later layers show notably higher similarity between trained LoRA and VeRA weights.The authors relate this pattern to earlier findings that adaptation is greatest in later layers.
- The similarity results support VeRA’s ability to approximate weights trained with LoRA.
E EXPRESSIVITY OF VERA
An expressivity experiment fits random 10x10 matrices with LoRA and VeRA under matched trainable-parameter budgets. Both perform equally well, while VeRA permits substantially lower parameterizations.
- The experiment evaluates LoRA and VeRA by fitting random square 10x10 matrices.
- For a given number of trainable parameters, LoRA and VeRA perform equally well.The comparison is reported through MSE loss on the matrix-fitting task.
- VeRA allows much lower parameterization, including below LoRA’s rank 1.
F INSTRUCTION-TUNING WITH VICUNA EVAL
The instruction-tuning evaluation reports results for a Llama 7B model fine-tuned on cleaned Alpaca data and compares VeRA- and LoRA-fine-tuned models using GPT-4 scores. The supplied passages identify the evaluation setup and reported comparison dimensions, but not numerical outcomes.
- The evaluated model is a Llama 7B model fine-tuned on a 10K subset of the cleaned Alpaca dataset.
- The evaluation uses Vicuna Eval, described as a predecessor of MT-Bench.
- The comparison table reports GPT-4 average answer scores alongside trainable-parameter counts and theoretical single-precision storage requirements.