PaperlayerView exact revision on arXiv

Source-linked AI summary

LoRETTA: Low-Rank Economic Tensor-Train Adaptation for Ultra-Low-Parameter Fine-Tuning of Large Language Models

Yifan Yang, Jiajun Zhou, Ngai Wong, Zheng Zhang

arXiv:2402.11417v1cs.CLcs.AIcs.LG

TL;DR

Existing PEFT methods can still require many trainable parameters as LLMs grow. LoRETTA applies tensor-train decomposition through tensorized adapters and weight parameterization. On LLaMA-2 models, the paper reports better performance than other PEFT methods with fewer trainable parameters, alongside improved efficiency, multi-task learning, and anti-overfitting outcomes.

  • Problem

    Existing PEFT methods remain limited by high trainable-parameter counts as LLMs grow, and ultra-low-parameter tuning that matches or exceeds full-model fine-tuning remains an open question.

    Source: §1 · pp. 1–2

  • Method

    LoRETTA uses tensor-train decomposition in tensorized adapters and weight parameterization with small tensor factors for fine-tuning LLMs.

    Source: §1 · p. 2, §3.1 · p. 3

  • Results

    On LLaMA-2 models, LoRETTA outperforms other PEFT methods with fewer trainable parameters, and experiments report benefits for computation and memory demands, storage, multi-task learning, and overfitting.

    Source: §5 · p. 8

  • Takeaways & Limitations

    The experiments support LoRETTA's use for natural-language understanding and generation while reducing computation, memory, and storage demands.

    Source: §5 · p. 8

    The scarcity of benchmark datasets limits more comprehensive experiments, and the paper notes that modifications remain necessary for its LLaMA experiments.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Various parameter-efficient fine-tuning (PEFT) techniques have been proposed to enable computationally efficient fine-tuning while maintaining model performance. However, existing PEFT methods are still limited by the growing number of trainable parameters with the rapid deployment of Large Language Models (LLMs). To address this challenge, we present LoRETTA, an ultra-parameter-efficient framework that significantly reduces trainable parameters through tensor-train decomposition. Specifically, we propose two methods, named {LoRETTA}$_{adp}$ and {LoRETTA}$_{rep}$. The former employs tensorized adapters, offering a high-performance yet lightweight approach for the fine-tuning of LLMs. The latter emphasizes fine-tuning via weight parameterization with a set of small tensor factors. LoRETTA achieves comparable or better performance than most widely used PEFT methods with up to $100\times$ fewer parameters on the LLaMA-2-7B models. Furthermore, empirical results demonstrate that the proposed method effectively improves training efficiency, enjoys better multi-task learning performance, and enhances the anti-overfitting capability. Plug-and-play codes built upon the Huggingface framework and PEFT library will be released.

1 Introduction

Growing LLMs make full-model fine-tuning infeasible, while existing PEFT methods can still require many trainable parameters. LoRETTA uses tensor-train decomposition in two variants to target parameter-efficient fine-tuning, with reported performance and evaluation spanning multiple settings and concerns.

1 Introduction

  • Existing PEFT methods still impose substantial parameter costs: LoRA updates over 16 million parameters on LLaMA-2-70B, motivating more economical approaches.The introduction asks whether ultra-low-parameter PEFT can match or exceed full-model fine-tuning.Source: §1 · pp. 1–2
  • LoRETTA uses tensor-train format and reports up to 100× fewer trainable parameters than widely used methods such as Adapters and LoRA on LLaMA-2.Its two variants are tensorized adapters and weight parameterization with small tensor factors.Source: §1 · p. 2
  • LoRETTA is reported to perform comparably to other widely used PEFT methods across model scales, tasks, and setups, particularly on large-model generation tasks.The introduction also describes LoRETTAadp as outperforming PEFT methods at equivalent trainable-parameter sizes.Source: §1 · p. 2
  • The evaluation compares PEFT methods on storage and computation efficiency, anti-overfitting, multi-task forgetting, and performance across setups.Source: §1 · p. 2

2 Background

The background reviews PEFT approaches and tensor compression, motivating LoRETTA as a way to reduce the added trainable structures while retaining broad applicability. It also describes prior tensorized fine-tuning approaches and the task-dependent classifier in the LoRETTAadp architecture.

2.1 Parameter-Efficient Fine-Tuning

  • BitFit reduces trainable parameters by tuning biases, but its performance drops and its use is restricted on bias-free models such as LLaMA; LoRETTA is presented as broadly applicable to language models.Source: §2.1 · p. 2

2.2 Tensor-based Model Compression

  • Prior tensorized fine-tuning approaches either require over 10% of model parameters or rely on a ViT-sized stacked tensor that reaches 7 billion parameters for LLaMA-2-7B.The section frames high-rank pretrained weights as a barrier to applying tensorized approaches to LLM fine-tuning.Source: §2.2 · p. 2
  • In LoRETTAadp, the optional tensorized classifier is trainable for classification tasks and frozen for language-modeling tasks.Source: Figure 2 · p. 3

3 LoRETTA Method

LoRETTA reduces the size of injected fine-tuning modules by representing weights or updates with tensor-train factors. It provides tensorized adapters and a weight-reparameterization variant.

3.1 Tensorized TT Layer

  • LoRETTArep reshapes linear-layer weights into tensors and trains small TT factors instead of storing and training the full weight matrices.The tensorized representation uses factors with dimensions set by the TT ranks and tensor modes.Source: §3.1 · p. 3

3.2 Lightweight Tensorized Adapters

  • With hidden size 768 and bottleneck size 64, LoRETTAadp uses 1.2K parameters versus about 98K for conventional adapter weight matrices.The example assumes tensor shapes [8, 8, 8, 8, 8, 8] and constant TT rank 5.Source: §3.2 · p. 4
  • LoRETTAadp places two tensorized linear layers and an activation in adapters following attention and feed-forward sublayers.The method also investigates training layer normalization and the last layer; it tensorizes the classification last layer and freezes the language-model head.Source: §3.2 · p. 4

3.3 TT Reparameterization

  • LoRETTArep reparameterizes weight updates with tensor factors; one adapter uses 1K parameters versus LoRA’s 12K in the stated configuration.The comparison uses hidden size 768, tensor rank 5, and LoRA rank 8.Source: §3.3 · p. 4
  • LoRETTArep initializes all tensor factors from a Gaussian distribution, then uses tensor reconstruction during the first training step to mitigate initialization noise.The stated motivation is to avoid zero-gradient issues from setting a factor to zero.Source: §3.3 · pp. 4–5

4 Experiment

Experiments compare LoRETTA with full fine-tuning and common PEFT baselines across BERT-family and LLaMA-2 models. They report competitive task performance alongside results on parameter and memory efficiency, overfitting, and multi-task learning.

4 Experiment

  • The evaluation covers BERT-family and differently scaled LLaMA-2 models against full fine-tuning and PEFT baselines, with most methods using the same learning rate, batch size, and training epochs.Experiments use AdamW and NVIDIA Tesla V100-16GB and A100-40GB GPUs.Source: §4 · p. 5

4.1 GLUE Experiments on the BERT Family

  • On GLUE, LoRETTAadp outperforms LoRA on 4 of 8 tasks despite LoRA having 3× more trainable parameters, while attaining similar average performance with a difference of nearly 0.5%.Source: §4.1 · p. 5

4.2 Large-Scale Language Models

  • On LLaMA-2-7B low-resource tasks, LoRETTA performs better or on par with widely used PEFT methods, with up to 100× fewer trainable parameters.LoRETTAadp shows superior performance across most tasks; LoRETTArep achieves comparable performance with up to 100× fewer parameters than Adapters.Source: §4.2 · p. 6, Table 2 · p. 7
  • On LLaMA-2-13B and 70B, LoRETTAadp outperforms LoRA on 5 of 6 tasks; on 70B it uses nearly 12 million fewer parameters with over 1% accuracy improvement.The experiments include reasoning and generation tasks, and LoRETTAadp performs strongly on generation tasks such as SQuAD and DROP.Source: §4.2 · pp. 6–7, Table 3 · p. 7

4.3 Over-fitting and Multi-Task Learning

  • LoRETTAadp and LoRETTArep show more stable evaluation-loss curves and better handling of overfitting than LoRA and Adapters.LoRA and Adapters’ evaluation loss escalates rapidly beyond a point; the authors attribute LoRETTA’s behavior to fewer trainable parameters retaining pretrained information.Source: Figure 4 · p. 6, §4.3 · p. 7
  • After sequential training on SST-2, MRPC, and QNLI, LoRETTAadp and LoRETTArep achieve higher average test accuracy and retain more information from previous training.The experiment tests validation performance after training on the three datasets in sequence.Source: §4.3 · p. 7, Table 4 · p. 7

4.4 Memory Performance

  • LoRETTArep stores trainable parameters in around 1MB; on LLaMA-2, LoRETTArep and LoRETTAadp reduce storage by 57.4× and 9.8×, respectively, while LoRETTA reduces memory-copying time and training FLOPs.The FLOPs comparison is for three epochs of LLaMA-2 fine-tuning on SST-2.Source: §4.4 · p. 8, Figure 5 · p. 8, Table 5 · p. 8

4.5 Tensor Rank Analysis and Ablation Study

  • Rank sensitivity varies by task: SST-2 is not rank-sensitive, while QNLI accuracy drops at extra-small ranks; the tensorized classifier matches the regular classifier with fewer parameters, and layernorm matters on some tasks.The authors note that smaller ranks may reduce trainable parameters under tight hardware constraints.Source: §4.5 · p. 8, Table 6 · p. 8

5 Conclusion

LoRETTA is an ultra-parameter-efficient fine-tuning method that uses fewer trainable parameters than other PEFT methods on LLaMA-2 models. The authors report benefits for computation, memory, multi-task learning, and overfitting.

5 Conclusion

  • LoRETTA outperforms other PEFT methods with fewer trainable parameters on LLaMA-2 models.The conclusion describes its performance across both natural language understanding and generation tasks.Source: §5 · p. 8
  • The authors report that low trainable-parameter counts can facilitate computation and memory demands, reduce storage, and improve handling of multi-task learning and overfitting.These are outcomes reported from the paper’s experiments.Source: §5 · p. 8
  • Future work could combine LoRETTA with FlashAttention and quantization to further improve computation efficiency.Source: §5 · p. 8

Limitations

The authors identify experiment runtime and limited benchmark availability as constraints, and note that effectiveness beyond NLP remains unverified.

Limitations

  • The breadth of PEFT methods, models, and tasks makes training time-consuming.Source: Limitations · p. 9
  • Scarce benchmark datasets limit the authors’ ability to conduct more comprehensive experiments.Source: Limitations · p. 9
  • Future directions include memory-efficient fine-tuning on devices with limited memory bandwidth, zeroth-order training, and multi-task learning.Source: Limitations · p. 9
  • Effectiveness in ASR, ViT training, transfer learning, and deep reinforcement learning still requires experimental verification.The paper reports on NLP and presents these other fields as possible adaptations.Source: Limitations · p. 9

Ethics Statement

LoRETTA is presented as a cost-effective approach with a minimal memory footprint, but prolonged training—especially on multiple GPUs—can pose environmental challenges.

Ethics Statement

  • LoRETTA operates with a minimal memory footprint.The statement frames this as a cost-effective solution.Source: Ethics Statement · p. 9
  • The authors say LoRETTA alleviates data-center burdens and reduces CO2 emissions.Source: Ethics Statement · p. 9
  • Prolonged training, particularly with multiple GPUs, can pose environmental challenges; the authors aim to develop more efficient training methods.They also cite preserving computational power with ecological considerations in mind.Source: Ethics Statement · p. 9

A.1 Dataset Setup

The experiments use GLUE, SuperGLUE, and question-answering generation tasks, with a low-resource setting for SuperGLUE and generation evaluations.

A.1 Dataset Setup

  • GLUE covers understanding tasks including sentiment, acceptability, similarity and paraphrase, and reasoning, with metrics summarized in Table 7.Source: §A.1 · p. 13, Table 7 · p. 13
  • SuperGLUE classification and multiple-choice tasks, plus SQuAD and DROP, are evaluated with 1,000 training, 500 validation, and 1,000 test examples.The authors use prompt settings from Malladi et al. (2023); metrics are summarized in Table 8.Source: §A.1 · p. 13
  • The low-resource evaluation setting applies to both SuperGLUE and generation datasets.Source: §A.1 · p. 13

A.2 Baselines

The baselines include full fine-tuning and several PEFT approaches, while the appendix specifies experiment settings and tensor shapes for LoRETTA.

A.2 Baselines

  • Full fine-tuning initializes the model with pretrained weights and biases and updates all parameters, while adapter experiments evaluated bottleneck sizes such as 8 and 64.Source: §A.2 · p. 13
  • LoRA adds trainable rank-decomposition matrices in parallel to weight matrices and is mainly applied to query and value layers in most experiments.Source: §A.2 · p. 13
  • Prefix tuning adds trainable representations as keys and values at each layer; prompt tuning trains a small set of input-prompt parameters.The experiments report that prefix length m = 8 works satisfactorily across most tasks.Source: §A.2 · p. 13
  • BitFit trains only bias vectors and is tested only with BERT-based models because LLaMA linear layers lack bias terms.Source: §A.2 · p. 13

A.3 Hyperparameters

  • For GLUE, methods generally use nearly the same learning rates and batch sizes, and tasks train for 10 to 20 epochs; LLaMA-2 experiments run for 3 epochs.Full-model fine-tuning is excepted because it does not converge under the large learning rate; the authors note more learning steps may improve LoRETTA.Source: §A.3 · p. 14

A.4 Additional Detail of TT-format

  • TT tensor shapes are chosen for each model’s hidden size and bottleneck setup, with specific shape definitions needed for other hidden sizes.The implementations include common hidden sizes from 768 to 8192; Table 9 summarizes shape settings.Source: §A.4 · p. 14, Table 9 · p. 14

B Ablation Study on Classifier and Layernorm

The ablation finds that a tensorized classifier can match a regular classifier with substantially fewer last-layer parameters, while layernorm also plays an important role.

B Ablation Study on Classifier and Layernorm

Loading 2402.11417v1…