PaperlayerView exact revision on arXiv

Source-linked AI summary

Singular Geometry of Low-Rank Adaptation

Kamillo Ferry, Anthea Monod, Keisuke Yano

arXiv:2610.32854v1stat.MLcs.LG

TL;DR

LoRA’s low-rank factorization reduces trainable parameters but creates an identifiability problem, since many factor pairs can represent the same update. The paper analyzes LoRA under local linearization using forward-backward covariance to identify observable update directions, and studies the resulting local and global singular geometry. LoRA recovers the reduced-rank regression RLCT under suitable nondegeneracy, while experiments find substantial layer variation and show ambient covariance degeneracy alone does not determine LoRA complexity.

  • Problem

    LoRA’s low-rank factorization reduces trainable parameters but creates an identifiability problem, since many factor pairs can represent the same update.

    Source: §1 · p. 2

  • Method

    The paper analyzes LoRA under local linearization using forward-backward covariance to identify observable update directions, and studies the resulting local and global singular geometry.

    Source: §1 · p. 2

  • Results

    LoRA recovers the reduced-rank regression RLCT under suitable nondegeneracy, while experiments find substantial layer variation and show ambient covariance degeneracy alone does not determine LoRA complexity.

    Source: §1 · p. 4, §7 · p. 35

  • Takeaways & Limitations

    Modules with the same rank and parameter count can have different learning complexities depending on their location in the network.

    Source: §7 · p. 35

    The theoretical study considers one layer at a time, leaving interactions among simultaneously adapted layers for future study.

    Source: §7 · p. 36

Abstract

from arXiv · show

Low-rank adaptation (LoRA) is now a standard approach to parameter-efficient fine-tuning of large neural networks. Its low-rank factorization is inherently non-identifiable, making LoRA a singular statistical model whose learning complexity cannot be described by parameter dimension alone. We study this complexity using the real log canonical threshold (RLCT) from singular learning theory and how it depends on both the low-rank parameterization and the pretrained network. Under a local linearization, LoRA induces a low-rank trace regression model whose Kullback--Leibler geometry is controlled by a forward-backward covariance, constructed from activations and backpropagated sensitivities at the adapted layer. This covariance determines which low-rank perturbations are statistically observable. When it is nondegenerate along the relevant low-rank directions, we recover the classical learning complexity of reduced-rank regression; when its kernel meets these directions, additional statistically indistinguishable parameters change the singular geometry. We characterize this geometry locally through observable tangent directions and a residual singularity, and globally through affine linear sections of determinantal varieties. The global learning complexity need not be determined at a ground-truth parameter, as singularities on another rank stratum can dominate. We also introduce a curvature-corrected multi-temperature estimator of the RLCT. Experiments on multilayer perceptron and transformer architectures reveal strongly layer-dependent forward-backward geometry and show that ambient covariance degeneracy does not imply degeneracy along the low-rank directions relevant to adaptation.

1 Introduction

LoRA’s non-identifiable low-rank factorization makes its learning complexity depend on singular geometry, not just parameter count. This paper characterizes how that geometry interacts with the pretrained network and when LoRA’s RLCT agrees with reduced-rank regression.

1 Introduction

  • The framework links low-rank factorization geometry to pretrained-network geometry through a covariance that determines which update directions are statistically observable.The covariance is constructed from layer activations and backpropagated sensitivities.Source: §1 · p. 2
  • When covariance degeneracy intersects low-rank directions, the local and global singular geometry matters, and singularities away from the ground truth can determine global complexity.The global analysis uses affine linear sections of determinantal varieties and compares rank strata.Source: §1 · p. 2
  • Prior LoRA theory addressed expressive power and training convergence, but its generalization behavior from the singular-learning perspective remained uninvestigated.The choice of layers to adapt also remains an open question.Source: §1 · p. 3
  • Under nondegeneracy of the forward-backward covariance, LoRA’s RLCT matches reduced-rank regression; otherwise, the two can differ.The paper develops the differing geometry theoretically and studies it empirically.Source: §1 · p. 4
  • LoRA’s learning complexity cannot be determined from adapted layers alone because it depends on the frozen pretrained network’s forward-backward covariance.The paper connects this dependence to the unresolved question of which layers to adapt.Source: §1 · pp. 3–4
  • A curvature-corrected multi-temperature estimator uses a local quadratic fit in inverse temperature to mitigate sensitivity to temperature choice.The motivation is that finite-sample tempered posterior loss need not vary linearly with inverse temperature.Source: §1 · p. 4

2 Preliminaries on Singular Learning Theory

These preliminaries explain why singular learning complexity is governed by the geometry of the KL zero set and how local RLCT calculations can be simplified. They introduce RLCTs for analytic ideals and a local splitting that separates regular directions from a residual singularity.

2 Preliminaries on Singular Learning Theory

  • The local splitting principle separates regular directions from the remaining singular part, whose lower-dimensional model determines the residual contribution to the RLCT.The regular directions contribute by dimension counting.Source: §2 · p. 4

2.1 Real Log Canonical Thresholds

  • The RLCT is the smallest pole of the zeta function, and in singular models it replaces parameter dimension in the leading asymptotics of Bayesian learning.For regular models, the RLCT is half the parameter count.Source: §2.1 · pp. 5, 7
  • An ideal’s RLCT is independent of its chosen generators, so one collection of analytic equations can be replaced by another that generates the same ideal.RLCTs of independent variable groups combine according to a product rule.Source: §2.1 · p. 6
  • The global RLCT is determined by the minimum local threshold over the zero set, with multiplicity deciding dominance when the threshold values tie.At equal threshold values, the larger multiplicity is dominant.Source: §2.1 · p. 6

2.2 Contact Equivalence and Local Splitting

  • Contact equivalence preserves the local RLCT while allowing analytic reparameterization and invertible analytic changes to the defining equations.The local splitting isolates first-order regular directions and leaves nonlinear behavior in a residual map.Source: §2.2 · pp. 7–8
  • At a point where the differential has rank s, the local model can be written as s regular coordinates plus a residual map with vanishing differential.The corresponding RLCT separates the regular contribution from the residual threshold.Source: §2.2 · pp. 7–9

3 LoRA and Forward-Backward Geometry

LoRA’s learning geometry depends both on its low-rank factorization and on which update directions the pretrained network makes statistically observable. The forward–backward covariance identifies those directions and determines when reduced-rank regression’s classical RLCT applies.

3.1 Local Linearization and Low-Rank Trace Regression

  • Under local linearization, a single LoRA-adapted layer becomes low-rank trace regression, with KL geometry governed by the forward–backward covariance of activations and backpropagated sensitivities.This covariance determines which update perturbations are statistically distinguishable.Source: §3 · p. 9, §3.1 · p. 10
  • When the covariance preserves all relevant low-rank directions, the KL loss is equivalent to reduced-rank regression and the classical RLCT applies.For a positive-definite covariance, the stated RLCT is that of reduced-rank regression, with its expression depending on the true update rank and an integer minimization over s.Source: §3 · p. 9, §3.1 · p. 11

3.2 The Forward–Backward Covariance and the Geometry of Statistical Observability

  • Only the projection of the update difference onto the covariance’s active subspace contributes to KL; directions in its kernel are statistically invisible.The active subspace is the image of the covariance, and the KL divergence is equivalent to the squared norm of the projected update difference up to positive constants.Source: §3.2 · p. 12
  • The indistinguishable LoRA parameters are preimages of an affine linear section of the rank-at-most-r determinantal variety, with each product-matrix fiber comprising gauge orbits.The section is formed by translating the covariance kernel by the true update and intersecting it with the determinantal variety.Source: §3.2 · p. 13

3.3 The Nondegenerate Regime

  • Ambient covariance degeneracy is harmless when its kernel contains no nonzero low-rank difference accessible to LoRA; then the reduced-rank regression geometry and RLCT are recovered.A sufficient condition is that the kernel contain no nonzero matrix of rank at most r + ρ; independent forward and backward signals with positive-definite second moments also ensure a positive-definite covariance.Source: §3.3 · pp. 14–15
  • When the covariance kernel intersects directions accessible to the low-rank model, the statistically indistinguishable set can exceed the gauge fiber, and the reduced-rank regression comparison no longer suffices.The subsequent analysis addresses the resulting singular geometry.Source: §3.3 · p. 15

4 Local Singular Geometry

This section characterizes local RLCT through observable first-order directions and a residual singularity when the active covariance hides low-rank perturbations. At equal-rank points, it reduces the residual problem to a slice with a determinantal image and gauge symmetry, giving an explicit evaluation under a kernel-inclusion condition and bounds otherwise.

4 Local Singular Geometry

  • Observable tangent directions contribute a regular dimension-counting term to the local RLCT, while the remaining singular behavior is captured by a residual function.The decomposition uses the active coordinate map and the local splitting result.Source: §4 · p. 15, §4.2 · p. 17

4.1 Active Subspaces and the Local Product Geometry

  • At an equal-rank point of rank ρ′, the product-map linearization has kernel dimension ρ′^2 + (r − ρ′)(m + d) and rank ρ′(m + d − ρ′).The active covariance makes directions in im MX′ ∩ ker Π+ unobservable; their dimension κ(X′) reduces the observable rank.Source: §4.1 · pp. 16–17

4.2 Local RLCT and the Residual Singularity

  • At equal-rank points, the observable dimension is s = ρ′(m + d − ρ′) − κ(X′), and the local threshold also depends on the residual function χ.The splitting writes the active coordinate map as s regular coordinates plus χ; the theorem leaves χ’s threshold as a separate term.Source: §4.2 · p. 17
  • The residual vanishes exactly when WΠ has local dimension r(m + d) − s; then the local RLCT is (s/2, 1) and the model is Morse–Bott nearby.For full-rank points with κ(X′) = 0, this condition holds and recovers the regular dimension-counting threshold.Source: §4.2 · p. 18
  • For coordinate projections at full-rank points, a nonzero residual can occur only over the exceptional set where the projected determinantal rank drops.In the rank-one case, the generic observable dimension is m + d − c, where c is the number of connected components of the associated bipartite graph.Source: §4.2 · pp. 19–20
  • On a slice transverse to the gauge orbit, residual variables form a rank-at-most-(r − ρ′) determinantal model in a linear subspace of dimension (m − ρ′)(d − ρ′).For ρ′ < r, the product-map kernel includes directions that are not tangent to the gauge orbit.Source: §4.2 · p. 20
  • The slice splits into observable directions SM and residual directions Sres; M is an isomorphism from SM onto im M and vanishes on Sres.Moreover, ker M = T ⊕ Sres, so the residual directions represent ker M/T.Source: §4.2 · pp. 21–22
  • The residual map has image exactly the matrices of rank at most r −ρ′ and is invariant under GLr−ρ′ gauge transformations.Its inputs are the lower-right perturbation blocks (a22, b22), and the map sends them to b22a22.Source: §4.2 · p. 22
  • When ker Π+ ⊆ im M, Proposition 4.2.9 gives an explicit local-threshold characterization, with κ = dim ker Π+.If ker Π+ is not contained in im M, that decomposition is unavailable and the local threshold is bounded instead.Source: §4.2 · p. 23
  • For the general case, Proposition 4.2.10 bounds the local RLCT: directions in im M ∩ ker Π+ contribute 1/4 each rather than 1/2.The section attributes the upper bound’s adjustment to these κ directions entering α only at second order.Source: §4.2 · p. 23
  • In Example 4.2.11, κ = 1 gives 1 ≤ λ ≤ 5/4 with λ = 1, while κ = 2 gives 1/2 ≤ λ ≤ 1 with λ = 1.Both examples satisfy ker Π+ ⊆ im M, so the values also follow from Proposition 4.2.9.Source: §4.2 · p. 24

5 Global Geometry and Determinantal Sections

The global RLCT is determined by the affine section of possible products, not necessarily by the true update; it suffices to compare local thresholds at equal-rank points over those products.

5.2 The Global Threshold

  • The global RLCT is the minimum of local RLCTs on the domain, and the theorem identifies equal-rank points that realize this minimum among the candidate values.The result assumes a compact domain with WΠ intersecting it; if a minimizing point lies in the interior, its local threshold equals the global threshold.Source: §5 · p. 24, §5.2 · p. 26
  • The global threshold can be controlled by a singularity away from the true update, so the minimum must range over all products in Z.The products in Z need not have the rank of B0A0, and Z may be singular elsewhere.Source: §5 · p. 24, §5.2 · p. 27

5.1 Reduction to Equal-Rank Points

  • Every point can degenerate to an equal-rank point with the same product and no larger local threshold; equal-rank points over one product share a threshold.Thus the global RLCT reduces to local thresholds at equal-rank points over products in Z.Source: §5.1 · pp. 25–26

5.2 The Global Threshold

  • For the example, the determinantal section is a union of three planes, and its triple intersection has a three-fold normal crossing with RLCT (3, 3).That local threshold is the RLCT for this specific model.Source: §5.2 · p. 28

6 RLCT Estimation and Experiments

This section introduces a curvature-corrected estimator for RLCT and uses posterior computations across synthetic regression, rotated-digit transfer, and prompt-based fine-tuning to examine learning complexity and forward-backward geometry.

6.1 Estimating the RLCT

  • The curvature-corrected multi-temperature estimator uses the slope of a local quadratic fit in β^-1 to estimate λ(β⋆), with the quadratic term absorbing variation around β⋆.Finite-sample nonlinearity can make the linear multi-temperature estimator depend on the selected temperature window.Source: §6.1 · p. 30

6.2 Additional Setup

  • Posterior estimates use localized or global Gaussian priors and MALA or SGLD; multi-temperature fits use 12 inverse temperatures and average results across four ladders.The fitting window is β⋆/4 to 4β⋆.Source: §6.2 · p. 30

6.3 Synthetic Trace Regression with Frozen FB Features

  • Equal-rank estimates match Laplace with a negligible residual term, whereas excess-rank estimates exceed Laplace as excess directions add residual singularity.The estimates lie below the Aoyagi and Watanabe theoretical value in both settings.Source: Figure 3 · p. 31, §6.3 · p. 32

6.4 Transfer to Rotated Digits

  • MLP modules have exact covariance kernels while transformer eigenvalues decay over orders of magnitude; estimates mostly match Laplace except for fc1 and the first transformer block.Within the MLP, estimates decrease from fc1 to fc4.Source: Figure 4 · p. 32, Figure 5 · p. 32, §6.4 · p. 33
  • Relative linearization error is small early in fine-tuning but rises toward order 1, so the theory characterizes local geometry around pretrained weights rather than the full nonlinear trajectory.The rate of increase differs by layer.Source: §6.4 · pp. 33–34, Figure 6 · p. 33

6.5 Real NLP task: Prompt-Based Surgical Fine-Tuning

  • All prompt-based fine-tuning RLCT estimates are below 256 and far below the Aoyagi and Watanabe value 6112; both spectra and RLCT vary strongly by layer.The estimates are not determined by the Laplace approximation alone.Source: Figure 8 · p. 34, §6.5 · p. 35

7 Discussion

LoRA’s learning complexity depends on how its low-rank geometry interacts with pretrained-network statistics, and experiments show that this interaction varies across layers. The framework treats one layer at a time, leaving multi-layer interactions for future study.

7 Discussion

  • LoRA modules with the same rank and parameter count can have different learning complexities depending on their network location and forward-backward covariance.The covariance determines which adapted-weight perturbations are statistically observable.Source: §7 · p. 35
  • Both forward-backward covariance spectra and RLCTs vary significantly across layers and modules; ambient covariance degeneracy alone does not determine LoRA complexity.The relevant issue is how covariance degeneracy intersects the low-rank directions seen by the adapter.Source: §7 · p. 35
  • The authors suggest studying whether forward-backward geometry can help identify effective adaptation locations, while layer selection remains an open problem.The geometry may depend on architecture, pretrained representation, data distribution, task, and module location.Source: §7 · p. 35
  • The theoretical study considers one layer at a time, so extending it to simultaneous multi-layer adaptation requires analyzing observable directions and their interactions across the network.The authors present the framework as a starting point for studying how learning complexity distributes across pretrained-model components.Source: §7 · p. 36

AI Usage Disclosure

The disclosure describes AI assistance with language, code, proofreading, and literature-review support, with author review and verification; mathematical statements and proofs were produced without AI.

AI Usage Disclosure

  • AI supported language and stylistic edits, code work, proofreading, and literature-review checks; authors reviewed and verified its outputs, while mathematical statements and proofs used no AI.Source: AI Usage Disclosure · p. 36

Related papers

Loading 2610.32854v1…