PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond Low-Rank Parameterization: Narrowing the Gap Between LoRA and Full Fine-Tuning via Gradient Decomposition

Yihao Ouyang, Shiwei Li, Haozhao Wang, Xiandi Luo, Zhuoqi Hu, Jinglun Yu, Yichen Li, Ruixuan Li

arXiv:2609.37027v1cs.AI

TL;DR

LoRA can remain behind full fine-tuning, while its first-order weight-space directions are limited by its current parameterization. GDLoRA decomposes the full weight gradient, applies its normal component to base weights, and retains standard AdamW for LoRA factors. Across evaluated task families, it improves over LoRA and narrows the gap to FFT without additional optimizer-state memory under matched configurations.

  • Problem

    LoRA can remain behind full fine-tuning, and its first-order weight-space updates are restricted to directions accessible through the current parameterization.

    Source: §1 · pp. 1–2

  • Method

    GDLoRA characterizes the accessible space as the current LoRA tangent space, then applies the orthogonal normal component of the full weight gradient to base weights while retaining AdamW for LoRA factors.

    Source: §1 · p. 2

  • Results

    Across natural language understanding, mathematical and commonsense reasoning, and image classification, GDLoRA improves average scores by 2-4% relative to LoRA and narrows the gap to FFT.

    Source: §1 · p. 2, §5 · p. 10

  • Takeaways & Limitations

    Under matched adapter and optimizer configurations, normal-gradient updates complement LoRA without increasing its optimizer-state memory budget.

    Source: §1 · p. 2

    The usual rank-r matrix-manifold tangent-space equivalence applies when both factors have rank r; the accessible-space definition also covers rank-deficient factors.

    Source: §3.2 · p. 4

Abstract

from arXiv · show

Low-Rank Adaptation (LoRA) is a widely used approach to parameter-efficient fine-tuning (PEFT), yet a performance gap can remain relative to full fine-tuning (FFT). Many LoRA variants improve the initialization or optimization of low-rank factors. At each training step, however, their first-order weight-space directions are constrained by the current parameterization. We characterize the corresponding LoRA-accessible gradient space and show that it coincides with the tangent space induced by the current LoRA parameterization. This characterization yields an orthogonal decomposition of the full weight gradient at the current model parameters. We term the component orthogonal to this space the normal gradient. Based on this decomposition, we propose GDLoRA (Gradient-Decomposed Low-Rank Adaptation). GDLoRA reconstructs the full weight gradient from forward activations and backward signals, extracts its normal component, and directly updates the base weights with this component, while retaining standard AdamW optimization for the LoRA factors. GDLoRA incorporates complementary normal gradients without increasing standard LoRA's optimizer-state memory budget under matched adapter and optimizer configurations. Experiments on natural language understanding, mathematical reasoning, commonsense reasoning, and image classification show that GDLoRA consistently improves over LoRA and narrows the performance gap to FFT. The code is available at https://anonymous.4open.science/r/GDLoRA.

1 INTRODUCTION

GDLoRA addresses LoRA’s remaining performance gap to full fine-tuning by adding gradient information beyond the directions available through its current parameterization. It isolates the normal component of the full weight gradient for base-weight updates while retaining standard LoRA optimization, and improves average scores over LoRA without additional optimizer-state memory under matched configurations.

1 INTRODUCTION

  • LoRA variants improve initialization or gradient alignment, but their first-order updates remain constrained by directions accessible through the current parameterization.Fira instead reuses SVD projection bases across steps, which may become less aligned as the leading gradient subspace drifts; refreshing bases more often adds computation.Source: §1 · pp. 1–2
  • The LoRA-accessible gradient space coincides with the tangent space induced by the current parameterization, making the orthogonal normal gradient a complementary local descent direction.When nonzero, the normal component is a local descent direction for the current batch loss with adapters and other weights held fixed.Source: §1 · p. 2
  • GDLoRA extracts the normal component from reconstructed full weight gradients, updates base weights with it, and retains standard AdamW optimization for LoRA factors without new base-weight momentum or variance buffers.The normal branch uses token-count normalization; both update branches use quantities computed from the same pre-update parameters.Source: §1 · p. 2
  • GDLoRA improves average scores by 2-4% relative to LoRA across four task families, approaching FFT without additional optimizer-state memory under matched configurations.The reported tasks are natural language understanding, mathematical reasoning, commonsense reasoning, and image classification.Source: §1 · p. 2

2 RELATED WORK

Related work spans parameter-efficient adapters, prompts, and LoRA, alongside variants that change initialization, optimization, or adaptation capacity. GDLoRA retains standard LoRA optimization and derives projection bases from current adapters rather than reusing SVD bases.

2 RELATED WORK

  • PEFT reduces adaptation costs by optimizing a small parameter subset while keeping most pretrained weights frozen; adapters add modules, prompts add learnable tokens, and LoRA adds low-rank matrices.LoRA keeps base weights frozen, reducing optimized parameters and associated optimizer-state memory.Source: §2 · pp. 2–3
  • LoRA variants alter initialization, gradient alignment, learning rates, scaling, rank allocation, or weight decomposition to improve adaptation.Examples include PiSSA, LoRA-GA, LoRA-One, LoRA+, rsLoRA, AdaLoRA, and DoRA.Source: §2 · p. 3
  • GaLore and Fira use low-rank gradient projections with reused SVD bases, whereas GDLoRA derives projection bases from current adapters each step and updates base weights along the normal gradient.Fira also adds scaled residual updates for full-rank training; reused bases trade subspace alignment against refresh costs.Source: §2 · p. 3

3 METHOD

GDLoRA recovers the full weight gradient, separates directions accessible to the current LoRA factors from their orthogonal complement, and uses that complement to update the base weights while retaining AdamW for the adapters.

3.1 FULL GRADIENT RECOVERY

  • GDLoRA reconstructs the full weight gradient from forward activations and backward signals, then divides the normal-branch gradient by the number of input positions.This normalization preserves the gradient direction; adapter gradients remain unchanged.Source: §3.1 · p. 3, Figure 1 · p. 4

3.2 GRADIENT DECOMPOSITION

  • The paper defines accessible directions at the current factors, a definition that remains valid when the factors are rank-deficient.When both factors have rank r, this space coincides with the usual tangent space of the rank-r matrix manifold at sBA.Source: §3.2 · p. 4
  • The LoRA-accessible gradient space equals the tangent space induced by the current parameterization, including for rank-deficient factors.The orthogonal complement is the normal gradient; when nonzero, a sufficiently small step along its negative decreases the current batch loss with adapters and other weights fixed.Source: §3.2 · pp. 4–5
  • The normal gradient is obtained by orthogonally decomposing the full gradient using the current LoRA tangent space and its orthonormal column- and row-space bases.The operator mapping the full gradient to the LoRA-induced gradient is generally not an orthogonal projector, so subtracting it does not generally yield an orthogonal residual.Source: §3.2 · p. 5
  • The normal component provides a local descent direction orthogonal to the current first-order adapter space when adapters and other weights are fixed.The interpretation treats the normal gradient as fixed at the current parameters and applies to a sufficiently small step when the component is nonzero.Source: §3.2 · p. 5

3.3 PARAMETER UPDATE

  • GDLoRA applies standard AdamW to the adapter factors and directly updates the base weights with the normal gradient.Both branches use gradients and projection bases computed from the same pre-update parameters.Source: Figure 1 · p. 4, §3.3 · p. 6
  • The normal-gradient branch adds no base-weight momentum or variance buffers, preserving LoRA’s optimizer-state budget under matched configurations.At the current iterate, its update is orthogonal to all first-order weight-space directions induced by the LoRA factors.Source: §3.3 · p. 6

4 EXPERIMENTS

Across language understanding, mathematical and commonsense reasoning, and image classification, GDLoRA improves on LoRA and narrows the performance gap to full fine-tuning. Ablations examine its gradient component, learning-rate sensitivity, and accuracy–memory trade-off.

4.1 NATURAL LANGUAGE UNDERSTANDING

  • GDLoRA reaches average scores of 85.26 and 88.71 on RoBERTa-Base and RoBERTa-Large, exceeding LoRA by 1.82 and 2.07 points, respectively.It outperforms LoRA and Fira in all 12 model–task combinations and closely matches FFT averages of 85.20 and 88.63.Source: §4.1 · p. 7

4.2 MATHEMATICAL REASONING

  • GDLoRA averages 70.01 accuracy on six mathematical reasoning tasks, 2.50 points above LoRA and 0.62 points below FFT, closing approximately 80.1% of LoRA’s FFT gap.It scores above FFT on MultiArith and AQuA and ranks second on the other four tasks.Source: §4.2 · p. 7, Table 2 · p. 7

4.3 COMMONSENSE REASONING

  • GDLoRA averages 85.15 on Gemma-7B commonsense reasoning, 1.75 points above LoRA and 0.55 points below FFT.It scores higher than Fira on all eight tasks and has the strongest aggregate performance among non-FFT methods.Source: §4.3 · p. 7, Table 3 · p. 8

4.4 IMAGE CLASSIFICATION

  • GDLoRA averages 86.79 and 89.20 on ViT-Base and ViT-Large, respectively, exceeding rank-8 LoRA by 2.07 and 2.46 points.It has the highest average among evaluated methods on both backbones, with results comparable to FFT averages of 86.43 and 89.08.Source: §4.4 · p. 8, Table 4 · p. 8

4.5 ABLATION STUDIES AND ANALYSIS

  • On GLUE, GDLoRA improves average scores over LoRA by 1.82 and 2.07 points on RoBERTa-Base and RoBERTa-Large, respectively, and outperforms LoRA+Full.LoRA+Tangent reduces performance on RoBERTa-Base and yields only a marginal improvement on RoBERTa-Large; the observed advantage is consistent with complementary normal-gradient information.Source: §4.5 · pp. 8–9
  • Across tested Math10K base-update learning rates, GDLoRA’s mean accuracy varies from 69.46 to 70.01, with gains over LoRA from 1.95 to 2.50 points.The highest accuracy occurs at ηn = 0.5, and the lowest GDLoRA mean exceeds the best LoRA+Full mean of 68.93.Source: §4.5 · p. 9
  • On Math10K, GDLoRA achieves 70.01 average accuracy with 21.6 GB memory, 2.50 points above LoRA using 1.2 GB additional memory.It trails FFT by 0.62 points while reducing memory from 61.79 GB to 21.6 GB.Source: §4.5 · p. 9, Figure 2 · p. 9

5 CONCLUSION

GDLoRA complements LoRA with gradient information outside the current parameterization’s first-order directions. Experiments show consistent improvements over LoRA and a reduced gap to FFT while preserving LoRA’s optimizer-state memory budget under matched configurations.

5 CONCLUSION

  • GDLoRA adds gradient information beyond the first-order directions accessible through LoRA’s current parameterization.It characterizes that accessible space as the induced tangent space, extracts the orthogonal normal component, and directly applies it to base weights while retaining AdamW for adapters.Source: §5 · p. 9
  • Under matched adapter and optimizer configurations, GDLoRA preserves LoRA’s optimizer-state memory budget.The normal-gradient branch updates base weights without additional base-weight momentum or variance buffers.Source: §5 · pp. 9–10
  • Across language understanding, mathematical and commonsense reasoning, and image classification, GDLoRA consistently improves over LoRA and narrows the performance gap to FFT.Ablation studies support the effectiveness of normal-gradient updates in complementing low-rank adaptation.Source: §5 · p. 10

A GRADIENT SUBSPACE DRIFT AND PROJECTION REFRESH COSTS

The section examines gradient-subspace drift and the accuracy–runtime cost of refreshing Fira projection bases, then contrasts that approach with GDLoRA’s adapter-derived bases. It also compares the methods’ leading per-layer computation.

A GRADIENT SUBSPACE DRIFT AND PROJECTION REFRESH COSTS

  • Leading rank-8 gradient subspaces vary across the recorded 50-step trajectory and across six selected attention projections spanning layers 0, 15, and 31.The measurements use pairwise subspace similarities during Llama-3-8B fine-tuning on Math10K; variation within the window means even 50-step refreshes need not eliminate mismatch.Source: A GRADIENT SUBSPACE DRIFT AND… · p. 12, Figure 4 · p. 14
  • On Math10K, native Fira’s accuracy rises from 65.21% to 67.35% as refresh intervals shorten from 200 to 50 steps, while training time rises from 30 to 47 minutes.GDLoRA reaches 70.01% in 21 minutes; even the 50-step Fira setting is 2.66 points lower and takes 2.24× as long.Source: A GRADIENT SUBSPACE DRIFT AND… · p. 13, Figure 3 · p. 13
  • GDLoRA recomputes its projection bases directly from current LoRA factors at every step, rather than repeatedly computing an SVD of the full weight gradient.The cost is O(dr2), compared with O(d3) for a dense gradient SVD per refresh; for r ≪ d, recomputing adapter-derived bases at every step is practical.Source: A GRADIENT SUBSPACE DRIFT AND… · p. 14, B COMPUTATIONAL COMPARISON WITH FIRA · p. 15

B COMPUTATIONAL COMPARISON WITH FIRA

  • The computational comparison counts full weight-gradient computation for both methods and contrasts GDLoRA’s thin-factor QR basis construction with Fira’s full-gradient SVD, amortized over refresh interval T.GDLoRA’s two-sided projection requires more projection FLOPs than Fira’s one-sided formulation.Source: B COMPUTATIONAL COMPARISON WITH FIRA · p. 14

C PROOF OF THE LORA-ACCESSIBLE GRADIENT SPACE CHARACTERIZATION

The proof characterizes the first-order directions induced by LoRA factor perturbations and identifies their orthogonal complement. It establishes the claimed characterization through the kernel and range of a self-adjoint operator.

C PROOF OF THE LORA-ACCESSIBLE GRADIENT SPACE CHARACTERIZATION

  • Differentiating Fs(A, B) = sBA yields parameter-induced directions of the form UK + LV^T when the factor perturbations vary freely.For full-row-rank factors, the maps from perturbations to K and L are surjective; a rank-zero factor contributes no corresponding term.Source: C PROOF OF THE LORA-ACCESSIBLE… · p. 15
  • The orthogonal complement of those directions consists of matrices H satisfying U^T H = 0 and HV = 0.The proof obtains this condition by requiring the inner product with UK + LV^T to vanish for independently varying K and L.Source: C PROOF OF THE LORA-ACCESSIBLE… · pp. 15–16
  • The proof identifies the operator’s range as the orthogonal complement of its kernel, completing the characterization of the accessible space.It invokes the finite-dimensional self-adjoint-operator result and combines it with the basis representation.Source: C PROOF OF THE LORA-ACCESSIBLE… · p. 16

D ORTHOGONAL PROJECTION ONTO THE NORMAL SPACE

The section constructs the normal component of the full gradient as the orthogonal complement to the LoRA tangent space. The remaining component lies in the tangent space and is the projection onto the LoRA-accessible gradient space.

D ORTHOGONAL PROJECTION ONTO THE NORMAL SPACE

  • The normal space consists of matrices Z satisfying U^T Z = 0 and ZV = 0.These conditions characterize the orthogonal complement of the tangent space {UK + LV^T}.Source: D ORTHOGONAL PROJECTION ONTO THE… · p. 16
  • G⊥ is the orthogonal projection of G onto the complement of the LoRA tangent space, and G = G∥ + G⊥ is the unique orthogonal decomposition.The text establishes G⊥ ∈ T⊥ and that every accessible gradient is orthogonal to it.Source: D ORTHOGONAL PROJECTION ONTO THE… · p. 16
  • Every LoRA-accessible gradient is orthogonal to G⊥.The section identifies the accessible gradient space with the LoRA tangent space.Source: D ORTHOGONAL PROJECTION ONTO THE… · p. 16

E FINITE-STEP ANALYSIS OF THE TWO UPDATE BRANCHES

The analysis separates finite adapter updates from direct normal compensation: factor updates can create some normal displacement through their bilinear remainder, but reachability and matching remain constrained. Under a sufficient condition, direct compensation improves normal matching for a fixed candidate factor update.

E.1 NORMAL DISPLACEMENT OF A FINITE ADAPTER UPDATE

  • A finite LoRA update’s normal component arises entirely from its bilinear remainder, while its first-order term lies in the current tangent space.The orthogonality property holds for arbitrary factor increments, including those produced by AdamW.Source: §E.1 · p. 17

E.2 REACHABLE NORMAL DISPLACEMENTS UNDER A FACTOR-STEP BUDGET

  • Within fixed factor coordinates, reachable normal displacements have rank at most r and nuclear norm at most quadratic in the factor-step budget ε.A nonzero spectral tail cannot be matched by an adapter normal displacement, even without a step-size constraint; the budget imposes additional error, and these are best-attainable errors rather than optimizer predictions.Source: §E.2 · p. 18
  • The reachability result concerns the normal projection only; its construction can also produce a tangent component.It characterizes admissible factor increments, not the subset selected by a particular optimizer, and the Euclidean factor-coordinate budget need not be invariant to factor rescaling.Source: §E.2 · p. 18

E.3 EFFECT OF DIRECT NORMAL COMPENSATION

  • For a fixed candidate factor update, direct normal compensation improves normal matching under a sufficient condition, not a necessary one.The condition depends on the alignment of the candidate normal displacement with G⊥; the comparison is to a current full-gradient step, not an FFT AdamW update or complete training trajectory.Source: §E.3 · pp. 18–19

F.1 COMPARISON WITH ADDITIONAL PEFT METHODS

On six mathematical reasoning tasks, GDLoRA outperforms the compared PEFT methods on average. Its advantage over LoRA persists across the evaluated adapter ranks, though individual task scores are not always monotonic.

F.1 COMPARISON WITH ADDITIONAL PEFT METHODS

  • 70.01% average accuracy: GDLoRA exceeds LoRA by 2.50 percentage points and MiSS by 2.43 points across six mathematical reasoning tasks.It leads on MultiArith, SingleEq, SVAMP, and GSM8K, while IA3 and FourierFT lead on AddSub and AQuA, respectively.Source: §F.1 · p. 19, Table 8 · p. 20

F.2 EFFECT OF ADAPTER RANK

  • 70.01%, 71.96%, and 72.81% average accuracies: GDLoRA exceeds same-rank LoRA by 2.50, 2.29, and 2.01 percentage points at ranks 8, 16, and 32, respectively.GDLoRA at rank 8 outperforms LoRA at rank 16, and GDLoRA at rank 16 outperforms LoRA at rank 32; rank-32 GDLoRA leads on five tasks, while AQuA peaks at rank 16. Individual task scores do not always improve monotonically.Source: §F.2 · pp. 19–20, Table 9 · p. 20

G EXPERIMENTAL DETAILS

Experiments cover language understanding, mathematical and commonsense reasoning, and image classification, using established model and dataset settings. The study compares FFT and several PEFT approaches under matched adapter configurations and reports results averaged across three random seeds.

G EXPERIMENTAL DETAILS

  • The evaluation covers six GLUE tasks with RoBERTa-Base and RoBERTa-Large, six mathematical and eight commonsense reasoning tasks with Llama-3-8B and Gemma-7B, and eight image-classification datasets with ViT-Base and ViT-Large.The reasoning models are fine-tuned on Math10K and Commonsense170K; rank-8 adapters target query and value projections for GLUE and query, key, and value projections for both reasoning settings.Source: G EXPERIMENTAL DETAILS · p. 20
  • Baselines include FFT, standard LoRA, Fira, and five LoRA variants; GDLoRA adds direct normal-gradient updates to the base weights beyond directions accessible through the current LoRA parameterization.Within each comparison, LoRA-based methods use the same adapter rank and target modules as GDLoRA.Source: G EXPERIMENTAL DETAILS · pp. 20–21
  • Language-model learning rates are independently selected for each method from the same candidate set, while other training and optimization settings are held consistent apart from the main learning rate.Fira refreshes its SVD-based gradient projection bases every 50 optimizer steps, except in the refresh-interval study.Source: G EXPERIMENTAL DETAILS · p. 21
  • Experiments use eight NVIDIA A100 GPUs, and reported results are means and standard deviations across three random seeds.The reported GPU counts refer to individual training runs.Source: G EXPERIMENTAL DETAILS · p. 21

H MAGNITUDE CONTROL AND LOCAL DESCENT OF THE NORMAL UPDATE

The normalized normal-gradient step has a magnitude bound and can strictly decrease the current loss under stated smoothness and step-size conditions. The section also specifies fixed-count microbatch accumulation and numerical-basis limitations.

H MAGNITUDE CONTROL AND LOCAL DESCENT OF THE NORMAL UPDATE

  • With ∥h_i∥F ≤ C, the normalized normal step is bounded by ηnC and strictly lowers loss when G⊥ ≠ 0 and 0 < ηn < 2N/β.The descent condition assumes a β-Lipschitz gradient on a neighborhood containing the update segment. Dividing by N removes explicit linear dependence on the position count from the magnitude bound, without requiring independent token contributions; the result does not guarantee descent of the joint update or convergence.Source: H MAGNITUDE CONTROL AND LOCAL DESCENT… · pp. 22–23

H.1 MICROBATCH ACCUMULATION WITH A FIXED INPUT COUNT

  • With fixed input counts across microbatches, the normal branch applies one common positive scale to the accumulated gradient, preserving its direction and relative microbatch weighting.Both branches use derivatives of the same accumulated objective, with no further division by the number of microbatches; the residual is projected at pre-update parameters, then applied to base weights after the adapter AdamW step.Source: §H.1 · p. 23

H.2 NUMERICAL BASES AND ZERO-RANK FACTORS

  • Thresholded unpivoted QR handles zero initialization, but may miss the complete column space for nonzero rank-deficient factors.The exact tangent-space statements assume bases of the true column spaces, whereas the implementation projects using retained QR bases.Source: §H.2 · p. 24

Related papers

Loading 2609.37027v1…