PaperlayerView exact revision on arXiv

Source-linked AI summary

AdaPreLoRA: Adafactor Preconditioned Low-Rank Adaptation

Ziyun Liu, Fengmiao Bian, Jian-Feng Cai

arXiv:2605.08734v1cs.LGcs.AIcs.CL

TL;DR

LoRA’s rank-deficient factorization makes mapping a preconditioned weight-space direction back to factors non-unique, leaving gradient-aware preconditioning underexplored at LoRA memory cost. AdaPreLoRA resolves this with an Adafactor W-space preconditioner and an H_t-weighted closed-form factor selection, achieving competitive or improved results across language and diffusion tasks with LoRA-level memory. The paper’s broader extensions remain open for settings such as mixture-of-experts, quantized backbones, and diffusion transformers.

  • Problem

    LoRA’s rank-deficient generator makes J_G^*F_tJ_G singular, so preconditioned W-space directions cannot be uniquely mapped back to factor-space updates; gradient-aware solutions within O((m+n)r) memory remain underexplored.

  • Method

    AdaPreLoRA adopts an Adafactor diagonal Kronecker preconditioner on W and selects the H_t-weighted imbalance-minimizing element from the factor-space solution family.

  • Results

    AdaPreLoRA is competitive with or improves over representative LoRA optimizers across GPT-2, Mistral-7B, Qwen2-7B, and diffusion personalization while matching LoRA-level peak memory.

  • Takeaways & Limitations

    The method provides a gradient-statistics-aware LoRA update whose W-space direction is the closest LoRA approximation to the preconditioned direction under the H_t-weighted norm.

  • Takeaways & Limitations

    Systematic extensions to mixture-of-experts, quantized backbones, and diffusion transformers are left for future work, with diffusion transformers potentially requiring relaxed gradient-statistics assumptions.

Abstract

from arXiv · show

Low-Rank Adaptation (LoRA) reparameterizes a weight update as a product of two low-rank factors, but the Jacobian $J_{G}$ of the generator mapping the factors to the weight matrix is rank-deficient, so the factor-space preconditioner $J_{G}^* {F}_t J_{G}$ induced by any ${W}$-space preconditioner ${F}_t$ is singular, and consequently the standard chain rule cannot be uniquely inverted to map a preconditioned ${W}$-space direction back to a factor-space update. We cast existing LoRA optimizers in a unified framework parameterized by two choices: (i) which invertible surrogate for $J_{G}^* {F}_t J_{G}$ to use, and (ii) which ${F}_t$ on ${W}$ to use. Existing methods occupy four families along these axes: factor-space adaptive updates, block-diagonal surrogates for $J_{G}^* J_{G}$, Frobenius-residual pseudoinverse methods, and Riemannian manifold constraint. Within this design space, a gradient-statistics-aware ${F}_t$ paired with a closed-form factor-space solve at ${O}((m+n)r)$ memory remains underexplored. We propose \textbf{AdaPreLoRA}, which fills this gap by adopting the Adafactor diagonal Kronecker preconditioner ${H}_t$ on ${W}$ and selecting from the resulting factor-space solution family the element minimizing an ${H}_t$-weighted imbalance between the two factor contributions; by construction, the resulting factor update is the closest LoRA approximation to the preconditioned ${W}$-space direction under the ${H}_t$-weighted norm. Across GPT-2 (E2E), Mistral-7B and Qwen2-7B (GLUE, ARC, GSM8K), and diffusion-model personalization, AdaPreLoRA is competitive with or improves over a representative set of LoRA optimizers while keeping peak GPU memory at the LoRA optimizer level.

1 Introduction

LoRA optimizers must address the singular factor-space preconditioner induced by mapping low-rank factors to weights. AdaPreLoRA combines an Adafactor W-space preconditioner with a closed-form, H_t-weighted factor-space selection and achieves competitive performance with LoRA-level memory.

  • LoRA reduces trainable parameters and optimizer state from O(mn) to O((m+n)r) by representing updates as BA.
  • Because the generator Jacobian is rank-deficient, J_G^*F_tJ_G is singular and cannot be uniquely inverted into a factor-space update.
  • AdaPreLoRA uses an Adafactor diagonal Kronecker preconditioner and selects the factor update minimizing H_t-weighted imbalance between the two factor contributions.
  • By construction, its W-update is the closest LoRA-expressible point to the Adafactor-preconditioned direction under the H_t-weighted norm.
  • Across language, reasoning, and diffusion benchmarks, AdaPreLoRA matches or outperforms representative LoRA optimizers while retaining LoRA-level peak memory.
  • Existing optimizers differ in their W-space preconditioner and their rule for selecting an invertible surrogate or solution from the affine factor-space family.

2 Background and Related Work

LoRA links low-rank factor updates to weight-space updates through a rank-deficient Jacobian, making the induced factor-space preconditioner singular and factor updates non-unique. Existing optimizers differ in how they replace or approximate this singular system and which weight-space preconditioner they use.

  • LoRA freezes W0 and represents the weight update as W = BA with r ≪ min(m, n), reducing trainable parameters and optimizer state to O((m + n)r).
  • The Jacobian JG connects factor-space and W-space directions, and its non-trivial kernel makes the factor-space pullback JG*FtJG singular for any Ft.
  • When Bt and At have full rank, ker(JG) has dimension r^2, so distinct factor updates can induce the same W-update.
  • Generalized inverses replace the unavailable inverse but yield different factor updates, shifting the issue from non-existence to non-uniqueness.
  • Existing optimizers are organized by their surrogate for JG*FtJG and their W-space preconditioner Ft, spanning factor-space, block-diagonal, Frobenius-residual, and manifold-based approaches.
  • Adafactor is the only listed candidate combining gradient-statistics-based preconditioning with O(m + n) W-space memory, whereas Shampoo and K-FAC require O(m^2 + n^2) memory.

3 The Proposed Algorithms

AdaPreLoRA solves the singular factor-space problem using an Adafactor preconditioner and an Ht-weighted criterion that selects a unique balanced factor update. Its projected W-update is geometrically aligned with the preconditioned direction while retaining LoRA-scale optimizer memory.

  • 3.1 The Adafactor Preconditioner Ht = L1/2: AdaPreLoRA uses the Adafactor diagonal Kronecker preconditioner Ht on W and selects a unique factor-space solution by minimizing Ht-weighted imbalance.
  • 3.2 Solving the Linear System on Factor Space: Figure 1 compares AdaPreLoRA, LoRA-Pro, and Riemannian Preconditioned LoRA under the Ft-weighted inner product, with AdaPreLoRA realizing the corresponding weighted projection.
  • 3.1 The Adafactor Preconditioner Ht = L1/2: Adafactor stores row and column second-moment estimates, reducing the preconditioner memory cost to O(m + n).
  • 3.2 Solving the Linear System on Factor Space: Because JG*HtJG is singular, AdaPreLoRA solves an equivalent least-squares problem rather than directly inverting the operator.
  • 3.2 Solving the Linear System on Factor Space: All minimizers induce the same Ht-orthogonal projection of the preconditioned direction onto the LoRA tangent space, while differing by kernel offsets.
  • 3.2 Solving the Linear System on Factor Space: The Ht-balance criterion resolves the r^2-dimensional ambiguity and yields a closed-form factor update with O((m + n)r) optimizer state.

4 Experimental Results

AdaPreLoRA is evaluated across language, reasoning, mathematics, generation, and image tasks, where it generally matches or exceeds representative LoRA optimizers while preserving low memory use.

  • GPT-2 controlled study: AdaPreLoRA achieves the best or tied-best score on every GPT-2 E2E metric at rank 4 across both model sizes and optimizer families.The advantage is largest for SGD-based methods and persists for AdamW-based methods.
  • GPT-2 controlled study: The GPT-2 gains transfer to the DART generation benchmark, with the same optimizer ordering confirmed at ranks 16 and 64.The DART results are reported for GPT-2 small with rank 4.
  • 7B-scale language models: AdaPreLoRA achieves the best accuracy in six of seven 7B settings and the lowest reported variance on Mistral-7B RTE.It is second-best on Mistral-7B CoLA, 0.1 point behind Scaled AdamW.
  • 7B-scale language models: AdaPreLoRA-AdamW matches Scaled AdamW at 26.0 GB peak memory, while LoRA-Pro AdamW requires 50.4 GB and AdaPreLoRA-SGD uses 21.5 GB.LoRA-Pro’s larger footprint comes from materializing full-weight gradients and moments.
  • Diffusion personalization: On diffusion personalization, AdaPreLoRA obtains the lowest FID at every scaling-optimizer combination and the best CLIP at scaling 1.0.At scaling 0.7, it remains competitive with the best baseline.

5 Conclusion

The paper organizes LoRA optimizers by their surrogate for the singular factor-space operator and their W-space preconditioner, then instantiates an underexplored low-memory design point with AdaPreLoRA.

  • Conclusion: The framework classifies LoRA optimizers by the invertible surrogate for J_G^* F_t J_G and the W-space preconditioner F_t.These two axes expose how existing methods differ in both factor-space inversion and gradient-statistics handling.
  • Conclusion: AdaPreLoRA combines an Adafactor diagonal Kronecker preconditioner with an H_t-balance criterion that selects a unique factor update.The resulting update has a closed-form expression and is the closest LoRA approximation under the H_t-weighted norm.
  • Conclusion: Across GPT-2, 7B language models, and diffusion personalization, AdaPreLoRA is competitive with or improves over representative LoRA optimizers while retaining LoRA-level peak GPU memory.The conclusion covers GPT-2, Mistral-7B, Qwen2-7B, and Mix-of-Show.
  • Limitations and future work: Future work includes alternative choices for both framework axes, mixture-of-experts adapters, QLoRA, and diffusion transformers.For diffusion transformers, the single H_t summary of per-step gradient statistics may need relaxation.

B Proof of Theoretical Results

The appendix derives the Jacobian of the low-rank generator and its adjoint, then computes their composition explicitly for arbitrary factor-space directions.

  • Jacobian definition: The low-rank generator maps factor pairs [B, A] to the matrix product BA, with J_G and J_G^* denoting its Jacobian and adjoint.B and A have dimensions R^{m×r} and R^{r×n}.
  • Composition: Composing the Jacobian and adjoint yields [PAA^T + BQA^T, B^TPA + B^TBQ].This explicit expression is the stated result for J_G^*J_G applied to [P, Q].
  • Notation: The notation table distinguishes calligraphic linear operators from bold matrices used for factors and related quantities.Examples include F_t and H_t as operators, and B_t, A_t, L_t, and R_t as matrices.
  • Jacobian computation: For a factor-space direction [P, Q], the Jacobian produces the W-space update PA + BQ.This follows by differentiating the product BA along differentiable factor curves.
  • Adjoint computation: The adjoint J_G^* maps a W-space matrix back to the paired factor spaces under the Frobenius inner product.The proof characterizes the adjoint through equality of inner products for arbitrary [P, Q] and C.

B.2 Proof of Proposition B.2 (Kernel of JG)

The proposition characterizes the Jacobian kernel arising from factorization redundancy and derives the associated rank. It also connects the Jacobian image with the tangent-space and weighted-projection formulations used for LoRA updates.

  • Kernel characterization: For full-column-rank B_t and full-row-rank A_t, every kernel element has the form [B_tX, −XA_t] for X ∈ R^{r×r}.The proof establishes both inclusion directions using one-sided pseudoinverses.
  • Rank consequence: By rank–nullity, rank(J_G) = (m+n)r−r^2, while the kernel complement has dimension r^2 in the factor space.The factor-space domain has dimension (m+n)r.
  • Kernel dimension: The map X ↦ [B_tX, −XA_t] is injective, so ker(J_G) has dimension r^2.Injectivity follows from the column-rank condition on B_t.
  • Tangent-space interpretation: The Jacobian image is the linear subspace T_t of expressible W-updates and coincides with the tangent space of the rank-r manifold when W_t has rank r.This identifies the main-text subspace view with the manifold viewpoint.
  • Weighted projection: Under an H_t-induced inner product, the common W-update across factor solutions is the H_t-orthogonal projection of the preconditioned direction onto T_t.When H_t=I, this reduces to the Frobenius projection used by LoRA-Pro.
  • Factor selection: AdaPreLoRA solves the factor update in two stages: parameterize the affine solution set by X_t, then minimize the H_t-weighted imbalance between factor contributions.Substitution of the minimizing X_t gives the closed form in Theorem 3.2.

D Computational and Memory Complexity Analysis of SoLoRA

The SoLoRA complexity analysis decomposes each update into gradient-statistics computations and factor-update operations. It reports linear memory in the matrix dimensions and a per-iteration cost dominated by the dense gradient term.

  • Statistics computation: Computing row and column statistics from G_t costs O(mn), while forming L_t and R_t square roots costs O(m) and O(n).The statistics are obtained by summing squared gradient elements along rows and columns.
  • Computational complexity: The total per-iteration computational complexity is O(mn + (m+n)r^2 + r^3).The complexities for ΔA_t and ΔB_t are symmetric and combine to this bound.
  • Memory complexity: The algorithm’s stated memory complexity is O(m+n), requiring storage of the vectors l_t and r_t each iteration.The analysis identifies these vectors as the additional stored statistics.

E Supplementary Experiments of GPT-2 Fine-tuning

The supplementary GPT-2 experiments document the training and inference configurations and optimizer learning rates used for reproducibility. They cover GPT-2 models of various sizes and distinguish medium-sized rank-4 settings.

  • Experimental configuration: Tables 8 and 9 provide the fine-tuning parameters and optimizer learning rates used in the GPT-2 experiments.The stated purpose is to support reproducibility, verification, and comparison.
  • Model settings: The experiments include GPT-2 models of various sizes.The configuration description explicitly states that multiple model sizes were tested.
  • Rank notation: “Rank 4 (M)” denotes a medium-sized model using LoRA with rank 4.The notation is defined in the supplementary configuration description.

E.2 Cross-rank ablation on GPT-2 small

The GPT-2 small cross-rank ablation evaluates ranks 16 and 64 using the main-text training and optimizer settings. AdaPreLoRA achieves the best or tied-best score on essentially every reported metric, with narrower but positive gains at higher rank.

  • Cross-rank setup: Ranks r∈{16,64} are evaluated on GPT-2 small fine-tuned on E2E using the settings from Tables 8 and 9.The comparison extends the main-text r=4 experiments.
  • Results: AdaPreLoRA achieves the best or tied-best score on essentially every metric across both ranks and optimizer families.The result is reported for the cross-rank comparisons.
  • Rank effect: At higher rank, AdaPreLoRA’s gains over Scaled GD/AdamW and LoRA-Pro narrow but remain positive.The passage relates this pattern to the continued advantage of the gradient-statistics-aware projection when factorization is less rank-constrained.

E.3 Training Efficiency Comparison

AdaPreLoRA’s training and inference efficiency is evaluated against other optimizers on GPT-2 small fine-tuned on E2E.

  • The comparison reports total training and inference time for all algorithms on GPT-2 small with rank 64.The model is fine-tuned on E2E.

F Supplementary Experiments of Diffusion Model Fine-tuning

The diffusion-model personalization experiments compare optimizer outputs across prompts, optimizer families, and LoRA scaling factors. AdaPreLoRA is reported to produce higher-quality generations, particularly for character faces, prompted scenes, and specified objects.

  • Diffusion-model personalization: The Mix-of-Show evaluation fine-tunes only LoRA components in the text encoder and U-Net of a Stable Diffusion backbone.Embedding-vector tuning is disabled.
  • Evaluation metrics: CLIP measures generated-image alignment with text prompts on a 0–100 scale, with higher scores preferred, while FID compares generated images against reference images, with lower scores preferred.Aggregate scores use LoRA scaling factors 0.7 and 1.0.
  • Qualitative comparisons: At scaling factor 1.0, Figures 2 and 3 compare AdamW-based optimizers on Harry Potter and Hermione Granger prompts, while Figures 4 and 5 compare SGD-based optimizers.The figures use the per-optimizer learning rates listed in Table 12.
  • Qualitative comparisons: AdaPreLoRA’s AdamW-based outputs better preserve prompted characters, actions, scenes, and facial quality in Figures 2 and 3.Figure 2 highlights Harry Potter, walking, and Mount Fuji; Figure 3 highlights Hermione Granger’s face and the beach scene.
  • Qualitative comparisons: AdaPreLoRA’s SGD-based and lower-scaling outputs are reported as higher quality, especially for Harry Potter’s and Hermione Granger’s faces and prompted objects or scenes.Figures 4 and 5 use SGD-based optimizers; Figures 6 and 7 use AdamW-based optimizers at scaling factor 0.7.
Loading 2605.08734v1…