PaperlayerView exact revision on arXiv

Source-linked AI summary

A Riemannian Geometry for Low-rank Adaptation

Shoichiro Takeda, Shin'ya Yamaguchi, Satoshi Suzuki, Yasunori Akagi

arXiv:2610.08049v1cs.LG

TL;DR

Full fine-tuning of large foundation models is costly, and LoRA can leave a gap to full fine-tuning at the weight level. The paper introduces a LoRA-tailored Riemannian metric whose preconditioning better approximates full fine-tuning in theory and in language and vision experiments.

  • Problem

    Full fine-tuning of large foundation models requires high computational and storage costs, while LoRA can retain a gap to full fine-tuning at the weight level.

    Source: §1 · pp. 1–2

  • Method

    The paper designs a Riemannian metric tailored to LoRA that induces new preconditioning.

    Source: §1 · p. 2

  • Results

    The preconditioning makes LoRA's allowed first-order weight update closest to the full fine-tuning gradient and its updated matrix closer in Frobenius norm than conventional or absent preconditioning, with language and vision experiments validating effectiveness and efficiency.

    Source: §1 · p. 2, §6 · p. 7

  • Takeaways & Limitations

    The theoretical comparisons and experiments indicate that the proposed preconditioning makes LoRA better approximate full fine-tuning.

    Source: §1 · p. 2, §9 · p. 10

    The method is inspired by a first-order approximation of full fine-tuning updates, so each update is assumed sufficiently small and higher-order effects remain for future study.

    Source: §8 · p. 10

Abstract

from arXiv · show

Low-rank adaptation (LoRA) is widely used as a parameter-efficient fine-tuning technique for pre-trained deep neural networks, which approximates the weight update via full fine-tuning by a low-rank matrix $BA^\top$. This parameterization leads to the equivalence relation $(B, A) \sim (BG^{-1}, AG^\top)$ for any invertible matrix $G$ because $BA^\top = BG^{-1}(AG^\top)^\top$ and thus both pairs yield the same loss value. This relation induces a quotient manifold where matrices $(BG^{-1}, AG^\top)$ for all $G$ are identified, eliminating redundant directions along which the loss value remains unchanged. To respect the geometry of this manifold, the original search space is endowed with a Riemannian metric that is invariant under the equivalence relation. Such a metric induces preconditioning at each gradient step and ensures that each weight update via LoRA changes the loss value, leading to efficient optimization. In this paper, we propose a new Riemannian metric that is specifically tailored to LoRA to close the gap to full fine-tuning at the weight level. We theoretically show that LoRA with our preconditioning induced by this metric satisfies the following two properties at each iteration: (i) The weight update follows the direction closest to the gradient of full fine-tuning within the subspace of first-order weight changes allowed by the LoRA parameterization. (ii) The updated weight matrix is closer in Frobenius norm to that of full fine-tuning than the updated weight matrices of LoRA with conventional preconditioning and without preconditioning. These theoretical insights suggest that our preconditioning makes LoRA better approximate full fine-tuning, thereby leading to more efficient optimization. Experiments show the effectiveness and efficiency of our preconditioning for LoRA on fine-tuning tasks with language and vision domains.

1 Introduction

Full fine-tuning is costly for large foundation models, while LoRA reduces trainable parameters but can lag full fine-tuning. This paper tailors a Riemannian metric to LoRA and shows theoretically and experimentally that its preconditioning better approximates full fine-tuning.

1 Introduction

  • Full fine-tuning of large foundation models is impractical because each task-specific model requires high computational and storage costs.LoRA is presented as a parameter-efficient alternative, though it can retain a performance gap to full fine-tuning.Source: §1 · pp. 1–2
  • LoRA approximates the full fine-tuning weight update with a low-rank factorization, reducing trainable parameters and computational and storage costs.The factors are B and A, and choosing rank r much smaller than the weight dimensions yields the stated savings.Source: §1 · pp. 1–2
  • LoRA's equivalence relation identifies factor pairs that yield the same loss, motivating an invariant Riemannian metric on the quotient manifold.Such a metric induces preconditioning at each gradient step; conventional work uses a metric originating in low-rank matrix completion.Source: §1 · p. 2
  • The proposed LoRA-tailored preconditioning makes each weight update closest to the full fine-tuning gradient within LoRA's allowed first-order changes and yields an updated matrix closer in Frobenius norm.The comparison is against conventional preconditioning and no preconditioning; experiments cover language and vision fine-tuning tasks.Source: §1 · p. 2

2 Related Works

Related work spans LoRA architecture, initialization, and optimization strategies. This paper follows standard two-factor LoRA and invariant-metric preconditioning while leaving combinations with several existing strategies for future work.

Modeling (Architecture)

  • Architecture variants include three-factor factorization, adaptive ranks, mixture-of-experts, multiple LoRA modules, and nonlinear mappings, while this paper uses standard two-factor LoRA.Expansion to advanced modelings is left for future work.Source: §2 · p. 3

Initialization

  • Initialization strategies use pretrained-weight singular vectors or one-step full fine-tuning gradients, whereas this paper follows the standard initialization.Combining advanced initialization with the proposed method is left for future work.Source: §2 · p. 3

Optimization

  • Optimization work includes gradient alignment, complementary subspace regularization, Stiefel-manifold constraints, and invariant-metric preconditioning.This paper follows prior Riemannian preconditioning for LoRA but designs a metric tailored to LoRA.Source: §2 · p. 3

Others

  • Other LoRA variants adjust factor learning rates, separate weight magnitude and direction, train modules sequentially, or quantize pretrained models.Combining these studies with the proposed method is left for future work.Source: §2 · p. 3

3 Preliminary

The preliminaries establish notation and the Riemannian optimization framework used to formulate LoRA updates on a quotient manifold. Invariant metrics make the quotient optimization well-defined through horizontal tangent representatives.

3.1 Notations

  • The notation section defines real numbers, identity and zero matrices, the Frobenius norm, and the general linear group GL(r).GL(r) consists of invertible r × r real matrices.Source: §3.1 · p. 3

3.2 Riemannian Optimization

  • Riemannian optimization equips each manifold tangent space with an inner product, defines its gradient through directional derivatives, and updates by a retraction with step size α.The paper reviews this framework as the basis for its optimization method.Source: §3.2 · pp. 3–4
  • On a quotient manifold, tangent directions split into vertical and metric-orthogonal horizontal spaces, and an invariant metric defines the quotient metric using horizontal lifts.For an invariant objective and equivalence-preserving retraction, the update corresponds to optimization on the quotient.Source: §3.2 · p. 4

4 Riemannian Preconditioning for LoRA

LoRA's factorization has equivalent representations that leave the loss unchanged, so prior work uses a quotient manifold and an invariant metric to remove redundant directions. The resulting Riemannian gradient update respects this geometry.

4 Riemannian Preconditioning for LoRA

  • The equivalence relation identifies factor pairs whose low-rank products yield the same loss, eliminating redundant directions in the LoRA search space.The quotient manifold represents each such set of pairs as one equivalence class.Source: §4 · p. 4
  • Prior LoRA preconditioning endows the search space with the invariant metric adopted from classical low-rank matrix completion.This is the established metric that the paper's later method seeks to tailor to LoRA.Source: §4 · pp. 4–5
  • Solving for the Riemannian gradient with that metric and applying an equivalence-preserving retraction gives a gradient-descent update that respects quotient geometry.The factor gradients are Euclidean gradients with respect to B and A; the resulting update can also be interpreted as preconditioned plain gradient descent.Source: §4 · p. 5

5 Proposed Method: Riemannian Preconditioning for LoRA with New Metric

The section introduces a LoRA-tailored invariant metric and derives preconditioning that respects quotient geometry while retaining the conventional asymptotic implementation cost.

5 Proposed Method: Riemannian Preconditioning for LoRA with New Metric

  • The proposed metric is invariant under LoRA’s equivalence relation, so it defines geometry consistently across equivalent factor pairs.The metric also induces the horizontal space used for quotient-manifold optimization.Source: §5 · p. 5
  • The resulting Riemannian gradient halves directions along col(B) and col(A) while preserving directions in their orthogonal complements.The projections onto these column spaces and their complements guide the preconditioning.Source: §5 · p. 5

5.1 An Efficient Implementation of Our Preconditioning

  • The preconditioning can be implemented without forming an m × m matrix, with total time O((m + n)r^2 + r^3), simplifying to O((m + n)r^2).The same complexity applies to the corresponding preconditioning for ∇AL.Source: §5.1 · p. 6

6 Theoretical Insights

The section compares LoRA updates with full fine-tuning under small-step assumptions. The proposed preconditioning yields the closest allowed first-order direction and an updated weight matrix closer in Frobenius norm to full fine-tuning than the conventional or unpreconditioned alternatives.

6 Theoretical Insights

  • The analysis assumes a sufficiently small step size, scaling factor s = 1, no dropout, no weight decay, and ignores terms of order O(α^2).It uses the chain rule to express the factor gradients through the full weight gradient.Source: §6 · p. 6
  • The conventional preconditioned update projects the full gradient onto the column spaces of B and A, while the proposed update additionally incorporates PBt(∇WtL)PAt.The latter matches an additional term in the decomposition of the full-fine-tuning update.Source: §6 · p. 6
  • The proposed preconditioning follows the full-fine-tuning gradient direction closest within LoRA’s allowed subspace of first-order weight changes.The comparison uses the orthogonal projection of the full gradient onto that subspace.Source: §6 · p. 7
  • The updated weight matrix is closer in Frobenius norm to full fine-tuning than the updates using conventional preconditioning or no preconditioning.The paper also reports empirical validation of this proximity result in Sec. F.Source: §6 · p. 7

7 Experiments

The experiments compare LoRA preconditioning methods across language generation, image classification, and commonsense reasoning, and report results for training time as well. The proposed preconditioning performs strongly across the task-specific comparisons, with training times reported as almost the same across methods.

7.1 Methods for Comparison

  • The comparisons apply no preconditioning, conventional preconditioning, and the proposed preconditioning to SGD and AdamW, tuning each method's learning rate on validation data.The authors used a train/validation/test split, unlike prior works described as using only train and test sets; experiments ran on a single H100 GPU.Source: §7.1 · p. 7, §7 · p. 7

7.2 Natural Language Generation

  • For E2E generation, the study fine-tunes GPT-2 medium with rank 4 LoRA in attention query and value matrices, evaluating five metrics on the test set.The setup uses five epochs and a linear learning-rate schedule.Source: §7.2 · p. 8
  • Our AdamW method achieves the highest scores on four E2E generation metrics and the second-highest on the fifth.The study fine-tunes GPT-2 medium on E2E and reports BLEU, NIST, METEOR, ROUGE-L, and CIDEr scores.Source: §7.2 · p. 8, Table 1 · p. 8

7.3 Image Classification

  • For image classification, the study fine-tunes CLIP ViT-B/32 with rank 16 LoRA in attention query and value matrices, testing AdamW methods only.Learning rates are tuned separately on each dataset's validation set; top-1 test accuracy is measured.Source: §7.3 · p. 8
  • Our AdamW method achieves either the highest or second-highest top-1 test accuracy across seven image-classification datasets, with the highest average.The comparison uses CLIP ViT-B/32 and reports accuracy across the seven datasets.Source: §7.3 · p. 8, Table 2 · p. 8

7.4 Commonsense Reasoning

  • For commonsense reasoning, the study fine-tunes LLaMA2-7B with Adam methods and evaluates top-1 test accuracy across eight datasets.LoRA is inserted into attention query, key, and value projections and feed-forward up and down projections.Source: §7.4 · p. 9
  • Our AdamW method achieves either the highest or second-highest top-1 test accuracy across eight commonsense-reasoning datasets, with the highest average.The experiment fine-tunes LLaMA2-7B on the benchmark datasets.Source: §7.4 · p. 9, Table 3 · p. 9

7.5 Empirical Validation: Training Time Comparison

  • Training times were almost the same across AdamW methods with different preconditioning, despite some differences attributed to constant time factors.The comparison reports means and standard deviations across validation learning rates for CLIP ViT-B/32 on seven image-classification datasets.Source: §7.5 · p. 9, Figure 1 · p. 9

8 Discussions and Limitations

The discussion identifies scope boundaries and future work, including optimizer coverage, factorization and initialization choices, the first-order approximation, and further evaluation.

8 Discussions and Limitations

  • Experiments tested the proposed preconditioning with SGD and AdamW; extending it to other optimizers remains future work.The authors describe extension to other sophisticated optimizers as a promising direction.Source: §8 · p. 9
  • The method uses standard two-factor LoRA; combining it with three-factor factorization remains future work.The authors also identify combining their method with other modeling approaches as an open direction.Source: §8 · p. 10
  • Combining the method with state-of-the-art LoRA initialization strategies is nontrivial because the approach may break those strategies' concepts.The authors leave exploring this combination to future work.Source: §8 · p. 10
  • The analysis uses a first-order approximation that assumes sufficiently small updates; higher-order effects remain to be considered.The authors say they will design more efficient preconditioning approaches in future work.Source: §8 · p. 10
  • Further assessment across varied models and tasks, including image generation with qualitative visual evaluation, remains future work.The authors frame this as a direction for assessing practical utility.Source: §8 · p. 10

9 Conclusions

The proposed LoRA-specific Riemannian metric is intended to narrow the weight-level gap to full fine-tuning; the appendices establish its invariance and characterize the allowed first-order weight changes.

9 Conclusions

  • The paper concludes that its LoRA-tailored metric better approximates full fine-tuning than conventional or absent preconditioning, with effectiveness and efficiency shown in language and vision experiments.Source: §9 · p. 10

A.1 Proof of Lemma 1

  • The proof derives the horizontal space associated with the LoRA equivalence relation and shows that the metric is invariant under that relation.Source: §A.1 · p. 18

A.2 Proof of Proposition 1

  • Equivalent factor pairs have corresponding horizontal lifts, and the metric assigns them the same geometry, establishing invariance under the LoRA equivalence relation.Source: §A.2 · p. 18

A.3 Proof of Proposition 2

  • The first-order weight changes allowed by LoRA form a subspace, and the proof identifies its Frobenius-orthogonal projection using the column-space projections for A and B.Source: §A.3 · p. 19

A.4 Proof of Proposition 3

The proof notes that a trace term can have either sign, leaving the ordering of the compared quantities undetermined in general.

A.4 Proof of Proposition 3

  • Because the second trace term may be positive or negative, its sign—and therefore the ordering—cannot be determined in general.Source: §A.4 · p. 20

B Derivations

The appendix derives the proposed gradients and weight update, documents dataset splits and hyperparameter selection, and relates the preconditioning to LoRA-Pro while distinguishing it from a Riemannian metric on the original space.

B Derivations

  • The derivations use the chain rule to express factor gradients through the weight gradient and the LoRA weight representation.Source: B Derivations · p. 20

B.1 Deviation of (15)

  • The metric yields Riemannian gradients for A and B that lie in the horizontal space associated with the LoRA quotient geometry.Source: §B.1 · pp. 20–21

B.3 Derivation of (20)

  • For the proposed preconditioning, the weight update is approximated as W_t − αP_Bt(∇W_tL) − α(∇W_tL)P_At + αP_Bt(∇W_tL)P_At, with higher-order terms O(α²).Source: §B.3 · p. 21

C.1 Natural Language Generation

  • The E2E natural-language-generation dataset contains 51,426 meaning representations and is split into 42,061 training, 4,672 validation, and 4,693 test instances.Source: §C.1 · p. 22

C.2 Image Classification

  • For image classification, the seven datasets use train, validation, and test splits, with dataset-specific splitting procedures described for EuroSAT and RESISC45.Source: §C.2 · p. 22, Table 4 · p. 22

C.3 Commonsense Reasoning

  • Commonsense reasoning uses 170,420 combined training samples, split into 160,420 training and 10,000 validation samples before evaluation on each dataset's test set.Source: §C.3 · p. 22, Table 5 · p. 22

D Details of Hyperparameters

E Relationship with LoRA-Pro [56]

F Empirical Validation of Proposition 3

A repeated single-step experiment provides some empirical evidence for Proposition 3, though the authors note that broader experimental settings would be desirable.

F Empirical Validation of Proposition 3

Related papers

Loading 2610.08049v1…