PaperlayerView exact revision on arXiv

Source-linked AI summary

DoRA: Weight-Decomposed Low-Rank Adaptation

Shih-Yang Liu, Chien-Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang-Ting Cheng, Min-Hung Chen

arXiv:2402.09353v6cs.CLcs.CV

TL;DR

LoRA reduces fine-tuning cost but often trails full fine-tuning, leaving a capacity gap whose underlying causes remain underexplored. DoRA decomposes weights into magnitude and direction, using LoRA for directional adaptation, and consistently outperforms LoRA across tasks and architectures without added inference latency.

  • Problem

    LoRA reduces trainable parameters and fine-tuning cost, but its capacity remains below full fine-tuning and the underlying causes of this gap are underexplored.

  • Method

    DoRA decomposes pretrained weights into magnitude and direction, fine-tuning both while using LoRA for efficient directional adaptation.

  • Results

    DoRA consistently outperforms LoRA across language and vision-language tasks, including a +3.7% improvement on LLaMA-7B commonsense reasoning.

  • Takeaways & Limitations

    DoRA closely resembles full fine-tuning's learning capacity while adding no inference latency over LoRA.

Abstract

from arXiv · show

Among the widely used parameter-efficient fine-tuning (PEFT) methods, LoRA and its variants have gained considerable popularity because of avoiding additional inference costs. However, there still often exists an accuracy gap between these methods and full fine-tuning (FT). In this work, we first introduce a novel weight decomposition analysis to investigate the inherent differences between FT and LoRA. Aiming to resemble the learning capacity of FT from the findings, we propose Weight-Decomposed Low-Rank Adaptation (DoRA). DoRA decomposes the pre-trained weight into two components, magnitude and direction, for fine-tuning, specifically employing LoRA for directional updates to efficiently minimize the number of trainable parameters. By employing \ours, we enhance both the learning capacity and training stability of LoRA while avoiding any additional inference overhead. \ours~consistently outperforms LoRA on fine-tuning LLaMA, LLaVA, and VL-BART on various downstream tasks, such as commonsense reasoning, visual instruction tuning, and image/video-text understanding. Code is available at https://github.com/NVlabs/DoRA.

1. Introduction

The introduction identifies the prohibitive cost of full fine-tuning and the capacity gap between LoRA and full fine-tuning. It presents DoRA, which decomposes weights into magnitude and direction, uses LoRA for directional updates, and consistently outperforms LoRA without additional inference overhead.

  • Method: DoRA decomposes pretrained weights into magnitude and direction, applying LoRA to directional updates to approach full fine-tuning capacity without additional inference latency over LoRA.The method is designed to improve LoRA’s learning capacity and training stability while avoiding extra inference costs.
  • Motivation: Full fine-tuning becomes prohibitively expensive as model and dataset sizes increase, motivating parameter-efficient fine-tuning methods.Full fine-tuning retrains all model parameters, whereas PEFT methods update only a minimal number.
  • Motivation: LoRA is popular for its simplicity and efficacy but retains a capacity gap relative to full fine-tuning.The introduction attributes this gap to LoRA’s limited number of trainable parameters while noting that the underlying differences require further exploration.
  • Analysis: The proposed weight decomposition analysis reparameterizes model weights into magnitude and directional components and compares update patterns from LoRA and full fine-tuning.The analysis is motivated by Weight Normalization and examines how each method changes magnitude and direction.
  • Results: DoRA consistently outperforms LoRA across commonsense reasoning, visual instruction tuning, and image/video-text understanding tasks.Reported gains include +3.7/+1.0 on LLaMA-7B/13B, +2.9 on LLaMA2-7B, +4.4 on LLaMA3-8B, +0.6 on LLaVA-7B, and +0.9/+1.9 on VL-BART.

2. Related Works

Related PEFT methods reduce fine-tuning costs by training only a small subset of parameters, while differing in whether they modify the model’s architecture, input, or weights. LoRA-based methods use low-rank updates that can be merged before inference, avoiding additional inference burden.

  • Parameter-Efficient Fine-Tuning: PEFT methods reduce large-model fine-tuning costs by training a relatively small subset of parameters for downstream adaptation.Existing methods are divided into three categories.
  • Adapter- and Prompt-Based Methods: Adapter-based methods add trainable modules to a frozen backbone, while prompt-based methods fine-tune additional soft input tokens.Prompt-based approaches can be sensitive to initialization, and both categories increase inference latency compared with the baseline model.
  • LoRA-Based Methods: LoRA and its variants approximate fine-tuning weight changes with low-rank matrices that merge into pretrained weights before inference, avoiding extra inference burden.Variants include SVD-based pruning of less significant singular values and low-rank Hadamard products for federated learning.

3. Pattern Analysis of LoRA and FT

This section describes LoRA’s low-rank weight updates and analyzes how LoRA and full fine-tuning differ in magnitude and direction changes. The analysis finds that LoRA exhibits a consistent positive relationship between directional and magnitude changes across intermediate steps.

  • LoRA formulation: LoRA represents the update ΔW as BA, where B and A are low-rank matrices and r ≪ min(d, k).The pre-trained weight W0 remains static while the low-rank update is trained.
  • LoRA formulation: A is Kaiming-initialized and B is initialized to zero, making ΔW = BA zero at training start.This initialization leaves the pre-trained weights unchanged initially while enabling subsequent updates.
  • Deployment: Merging the learned update with W0 before deployment lets LoRA and related variants avoid extra inference latency relative to the original model.Both the merged and original weights retain dimensionality Rd×k.
  • Weight decomposition analysis: The analysis decomposes weights into magnitude vectors and directional matrices, comparing W0, WFT, and merged WLoRA in VL-BART query/value matrices.The case study uses VL-BART fine-tuned on four image-text tasks, with LoRA applied only to query/value matrices in self-attention.
  • Analysis Results: LoRA shows a consistent positive slope across intermediate training steps, indicating a proportional relationship between directional and magnitude changes.Figure 2 examines query weight matrices across layers and training steps; the appendix reports analogous value-matrix visualizations.

4. Method

DoRA decomposes pretrained weights into magnitude and direction, applying LoRA to directional updates while fine-tuning both components. This decomposition improves optimization and produces learning patterns closer to full fine-tuning without additional inference latency, while a gradient-graph modification substantially reduces training memory with negligible accuracy change.

  • 4.1 DoRA: DoRA decomposes pretrained weights into magnitude and directional components, then applies LoRA to the directional component for efficient fine-tuning.Both magnitude and direction are fine-tuned, while the larger directional component is represented using two low-rank matrices.
  • 4.1 DoRA: The low-rank matrices are initialized so W′ equals W0 before fine-tuning, and DoRA weights can be merged before inference without additional latency.This preserves the pretrained initialization and avoids inference overhead.
  • 4.2 Optimization and Learning Pattern: DoRA and full fine-tuning exhibit negative magnitude–direction slopes, unlike LoRA, indicating learning behavior closer to full fine-tuning.The comparison is based on regression lines for directional and magnitude changes across query-weight matrices.
  • 4.2 Optimization and Learning Pattern: -0.62 and -0.31 are the magnitude–direction correlation values for FT and DoRA, respectively, whereas LoRA’s correlation is 0.83.The authors associate DoRA’s pattern with stronger learning capacity because it permits substantial directional changes with relatively small magnitude changes, or vice versa.
  • 4.3 Memory-Efficient Optimization: 24.4% and 12.4% are the training-memory reductions from detaching the normalization term for LLaMA and VL-BART, respectively.Accuracy remains unchanged for VL-BART and differs by only 0.2 from unmodified DoRA on LLaMA.

5. Experiments

Experiments evaluate DoRA across language, image, and video tasks against LoRA, full fine-tuning, and other PEFT baselines. DoRA consistently improves accuracy while retaining parameter-efficient training, including on commonsense reasoning, VL-BART multimodal tasks, and LLaVA visual instruction tuning.

  • 5. Experiments: DoRA is evaluated across language, image, and video domains, including commonsense reasoning, VL-BART image/video-text understanding, and LLaVA visual instruction tuning.The experiments compare DoRA with LoRA and additional PEFT or full-finetuning baselines across these task families.
  • Commonsense reasoning: DoRA consistently surpasses baseline methods on LLaMA-7B/13B, LLaMA2-7B, and LLaMA3-8B commonsense reasoning tasks.The comparison includes Prompt learning, Series adapter, Parallel adapter, LoRA, and ChatGPT’s zero-shot accuracy.
  • Commonsense reasoning: 3.7%: DoRA further enhances LLaMA-7B accuracy over LoRA and exceeds ChatGPT’s accuracy levels.For LLaMA-13B, DoRA improves over LoRA by 1% and achieves comparable accuracy to the Parallel adapter.
  • Multimodal fine-tuning: DoRA uniformly surpasses LoRA in VL-BART image/video-text task accuracy while maintaining a similar count of trainable parameters.The evaluation compares DoRA with LoRA and full fine-tuning across image-text and video-text tasks.
  • Visual instruction tuning: DoRA demonstrates superior performance over both LoRA and full fine-tuning on LLaVA-1.5-7B visual instruction tuning tasks.The passage notes that LoRA’s average accuracy already surpasses full fine-tuning, limiting DoRA’s improvement over LoRA relative to settings where full fine-tuning is stronger.
  • Rank robustness: 77.96%: DoRA retains this accuracy for r = 8, compared with LoRA’s 40.74%; for r = 4, DoRA achieves 61.89% versus LoRA’s 39.49%.These results demonstrate DoRA’s resilience and superior performance across rank settings.

6. Broader Impacts

The section extends DoRA’s evaluation to quantized fine-tuning and text-to-image generation. It examines QDoRA against QLoRA and FT on Orca-Math, and compares DoRA with LoRA for SDXL personalization.

  • Quantized Fine-Tuning: Figure 6 compares QDoRA, QLoRA, and FT for LLaMA2-7B and LLaMA3-8B on Orca-Math.Reported reference conditions also include zero-shot, few-shot, and FT with post-training quantization to BnB NF4.
  • Quantized Fine-Tuning: QDoRA substitutes DoRA for LoRA within QLoRA and uses FSDP for model splitting and parallel multi-GPU training.This addresses the memory demands of fine-tuning quantized pretrained models.
  • Quantized Fine-Tuning: Experiments fine-tuned LLaMA2-7B and LLaMA3-8B on Orca-Math using QDoRA, QLoRA, and FT.The training set contained 100k samples, with 500 reserved for evaluation using exact match.
  • Text-to-Image Generation: The SDXL personalization study follows DreamBooth with identical LoRA and DoRA hyperparameters and matched generation seeds.It uses challenging 3D icons and Lego sets datasets, with generated images shown in appendix Figures 10 and 11.
  • Text-to-Image Generation: DoRA achieves significantly better personalization than LoRA for text-to-image generation on the evaluated SDXL datasets.The comparison uses the 3D icons and Lego sets datasets under matched sample seeds.

7. Conclusion

The work analyzes distinct learning patterns between LoRA and full fine-tuning, then introduces DoRA, a LoRA-compatible method designed to more closely resemble full fine-tuning and consistently outperform LoRA.

  • 7. Conclusion: Weight decomposition analysis reveals distinct learning patterns between LoRA and full fine-tuning.
  • 7. Conclusion: DoRA is compatible with LoRA and its variants and exhibits learning behavior closer to full fine-tuning.
  • 7. Conclusion: DoRA consistently outperforms LoRA across various fine-tuning tasks and model architectures, including commonsense reasoning and visual instruction tuning.

A. Appendix · A.1. Weight decomposition analysis on the value weight matrix

Figure 7 analyzes magnitude and directional changes in value weight matrices for FT, LoRA, and DoRA across training steps and layers. The analysis finds that DoRA exhibits learning behaviors closely mirroring FT across modules.

  • A.1. Weight decomposition analysis on the value weight matrix: The analysis tracks value-matrix changes across different training steps.These intermediate steps are included in Figure 7’s comparison of FT, LoRA, and DoRA.
  • A.1. Weight decomposition analysis on the value weight matrix: The analysis tracks value-matrix changes across different layers.Figure 7 presents the magnitude and directional changes for the three methods across layers.
  • A.1. Weight decomposition analysis on the value weight matrix: The value-matrix patterns are similar to those observed in the query weight matrix.The passage explicitly relates Figure 7’s findings to the query weight matrix shown in Figure 2.
  • A.1. Weight decomposition analysis on the value weight matrix: DoRA displays learning behaviors that closely mirror FT across various modules.This conclusion follows from the observed magnitude and direction patterns in the value weight matrix.
  • A.1. Weight decomposition analysis on the value weight matrix: Figure 7 compares magnitude changes in the value weight matrices for FT, LoRA, and DoRA.The comparison spans different layers and intermediate training steps.
  • A.1. Weight decomposition analysis on the value weight matrix: Figure 7 compares directional changes in the value weight matrices for FT, LoRA, and DoRA.The comparison spans different layers and intermediate training steps.

A.2. Ablation study for the modification to reduce DoRA training cost

The proposed modification reduces DoRA’s training memory cost while largely preserving accuracy across LLaMA, VL-BART, commonsense reasoning, and image-text understanding tasks. It lowers memory use substantially, with minimal reported accuracy change.

  • Memory reduction: 24.4% training memory reduction occurs for DoRA fine-tuning LLaMA with the modification.The passage reports an approximately 24.4% reduction compared with DoRA without the modification.
  • Memory reduction: 12.4% training memory reduction occurs for DoRA fine-tuning VL-BART with the modification.The passage reports an approximately 12.4% reduction compared with DoRA without the modification.
  • Accuracy preservation: VL-BART accuracy remains unchanged, while LLaMA accuracy differs by only 0.2 between modified and unmodified DoRA.The passage characterizes the LLaMA accuracy difference as negligible.

A.3. Hyperparameters

This section reports hyperparameter configurations for DoRA across language, vision-language, and visual instruction tuning experiments. It also compares DoRA with LoRA and DVoRA in selected fine-tuning settings.

  • Commonsense reasoning: Table 8 specifies DoRA hyperparameter configurations for LLaMA-7B/13B, LLaMA2-7B, and LLaMA3-8B on commonsense reasoning tasks.The configurations cover multiple LLaMA model sizes and commonsense reasoning evaluations.
  • Image/video-text understanding: Table 9 specifies DoRA hyperparameter configurations for fine-tuning VL-BART on image/video-text tasks.The configurations target both image-text and video-text understanding settings.
  • Visual instruction tuning: Table 10 reports hyperparameter configurations for DoRA and LoRA when fine-tuning LLaVA-1.5-7B on visual instruction tuning datasets.The table enables configuration comparison between DoRA and LoRA in visual instruction tuning.
  • Alpaca fine-tuning: Table 11 reports hyperparameter configurations for DoRA and DVoRA when fine-tuning LLaMA-7B and LLaMA2-7B on the cleaned Alpaca dataset.The comparison covers two LLaMA model variants and the cleaned Alpaca fine-tuning setting.

A.4. Magnitude and Direction difference between DoRA/LoRA fine-tuned weight and the pre-triained weight of LLaMA2-7B for the commonsesne reasoning tasks … A.9. Text-to-Image Generation

Across commonsense reasoning, vision-language evaluation, instruction tuning, rank robustness, and text-to-image personalization, DoRA generally improves adaptation over LoRA while requiring smaller weight changes or fewer trainable parameters. The examples also report stronger response quality for DVoRA than VeRA and better SDXL personalization than LoRA under identical configurations.

  • A.4. Magnitude and Direction difference between DoRA/LoRA fine-tuned weight and the pre-triained weight of LLaMA2-7B for the commonsesne reasoning tasks: DoRA fine-tuned LLaMA2-7B weights deviate less than LoRA weights from the pre-trained model in both magnitude and direction.This supports the finding that effective downstream adaptation can require only modest changes to a robust foundation model.
  • A.5. Visual instruction tuning evaluation result: DoRA consistently outmatches FT and achieves higher average accuracy than LoRA across seven LLaVA-1.5-7B vision-language benchmarks.The evaluation uses visual instruction tuning data.
  • A.6. Instruction-tuning evaluation examples: DVoRA responses are generally more accurate and well-organized than VeRA responses on sampled MT-Bench questions.For Question 24, DVoRA uses a familiar superhero story, while VeRA begins with an uncertain tone.
  • A.7. Robustness towards different numbers of instruction tunning examples: DoRA and DVoRA consistently outperform LoRA and VeRA across instruction-tuning sample sizes of 1000, 4000, 7000, and 10000.The comparison evaluates LLaMA-7B on MT-Bench using varying numbers of Alpaca training samples.
  • A.8. Robustness of DoRA towards different rank: DoRA consistently outperforms LoRA across rank settings, with the performance gap widening as rank decreases.The results indicate improved LoRA learning capacity and better accuracy with fewer trainable parameters.
  • A.9. Text-to-Image Generation: DoRA achieves considerably better SDXL personalization than LoRA on the 3D Icon1 and Lego2 training sets under identical DreamBooth configurations.DoRA outputs more closely match the training targets.
Loading 2402.09353v6…