PaperlayerView exact revision on arXiv

Source-linked AI summary

What Does the Rank Buy? A Spectral and Distributional Analysis of Low-Rank Adaptation

Babak Barazandeh

arXiv:2609.32002v2cs.AIcs.LGstat.ML

TL;DR

LoRA rank is often treated as a capacity control, but this paper examines that premise under hard per-factor norm budgets. It characterizes the reachable updates and analyzes their complexity, spectral cancellation, and distributional displacement, finding both rank-independent and rank-sensitive regimes.

  • Problem

    LoRA rank is widely treated as a capacity control, with smaller rank assumed to yield a simpler model that generalizes better.

    Source: §I · p. 1

  • Method

    The paper characterizes updates under hard per-factor budgets and analyzes the resulting complexity, spectral cancellation, and source-distribution displacement.

    Source: A. What This Paper Does · p. 2, §VII · pp. 11–12

  • Results

    The studied free-readout class and pushforward radius are rank-independent, while spectral cancellation has matching-order threshold-rank bounds under two-sided decay and specified alignment conditions.

    Source: §VII · pp. 11–12

  • Takeaways & Limitations

    Rank governs which updates are reachable and the budget required for spectral cancellation, rather than the capacity measured by the studied functionals.

    Source: A. What This Paper Does · p. 2, §VII · p. 11

    Whether the pushforward-radius upper bound is attained for the single-update cone in general remains open.

    Source: Appendix B · p. 17

Abstract

from arXiv · show

The rank $r$ in LoRA is widely treated as a capacity control: a smaller rank is assumed to yield a simpler model that generalizes better. We show that, under hard per-factor norm budgets---the idealization of the weight decay and norm control used in practice---this intuition breaks down. The reason is structural: under such budgets, the updates LoRA can reach are exactly the matrices of rank at most $r$ inside a nuclear-norm ball, and every complexity and displacement functional we analyze is maximized over this set by a rank-one update---so the rank cap never binds. The consequences follow directly. The linear-readout model class we study is identical for every $r \ge 1$, its Rademacher complexity carries no dependence on $r$, and the distance the adaptation can move the source distribution obeys a rank-independent upper bound that we show is sharp. If rank does not control capacity, where does it act? We identify two places. Statistically, replacing the per-factor budgets with a joint budget on the product restores a data-dependent, rank-sensitive complexity bound---though the gain appears only for well-spread feature distributions, and the worst case remains rank-free. Spectrally, rank sets the price of adaptation: canceling the leading singular directions of the pretrained weight requires both sufficient rank and sufficient budget. We bound the smallest rank achieving a desired source--target alignment, with upper and lower bounds that match under two-sided spectral decay. Together, these results recast rank as governing which updates are reachable and what cancellation costs---not how much capacity the model has.

I. INTRODUCTION

LoRA rank is commonly treated as a capacity control, but the introduction asks whether that premise holds under the norm constraints used to train adapters. The paper argues that rank instead governs reachable updates and the cost of spectral cancellation.

I. INTRODUCTION

  • LoRA offers parameter-efficient adaptation, while the usual capacity argument assumes a tighter rank bottleneck shrinks the hypothesis class and generalization gap.The paper notes that this premise is rarely examined against LoRA’s actual constraint set.Source: §I · p. 1

A. What This Paper Does

  • Under hard per-factor budgets, reachable updates form a rank-truncated nuclear-norm ball, but rank-one extremality makes the analyzed complexity and displacement functionals rank-independent.The paper treats hard budgets as an idealization of practical norm control; weight decay itself is a soft penalty.Source: A. What This Paper Does · pp. 1–2

B. Three Questions

  • The paper organizes its analysis around whether rank reduces statistical complexity, how rank and spectral decay set cancellation rank, and how far adaptation moves the source distribution.Source: B. Three Questions · p. 2

C. Contributions

  • Contributions include rank-free scalar complexity, conditional rank-sensitive matrix complexity, matching-order spectral threshold bounds, a rank-independent pushforward radius, and synthetic confirmation of theoretical predictions.Source: C. Contributions · pp. 2–3

1) Theoretical analyses of LoRA:

  • The paper frames its reachability result as a budgeted, single-layer refinement of prior work on the ranks needed for LoRA networks to represent target models.It distinguishes reachable-update analysis from prior analyses of optimization trajectories [34],.Source: 1) Theoretical analyses of LoRA · p. 3

2) Spectrally aware adaptation:

  • The spectral threshold rank provides a theoretical counterpart to spectral-domain adaptation methods, identifying top singular directions as where rank and budget bind.Source: 2) Spectrally aware adaptation · p. 3

3) Structural and compositional LoRA variants:

4) Domain adaptation and technical tools:

II. SETUP AND THE RANK-ONE EXTREMALITY PRINCIPLE

This section formalizes LoRA updates under per-factor norm budgets and identifies the structural reason the rank cap does not affect the paper’s central extrema. It also sets up the discrepancy and transferability concepts used later.

A. Background: The LoRA Parameterization

  • LoRA freezes the pretrained weight and trains two factors whose product has rank at most r, reducing trainable parameters while per-factor norm control motivates the hard-budget analysis.The factors can be merged into the weight after training, adding no inference latency.Source: A. Background: The LoRA Parameterization · p. 3

B. Notation and Standing Conventions

  • The setup defines source and target feature second moments, the budget product ρ, and bounded-feature conditions that apply only where explicitly invoked.Gaussian sharpness witnesses are not subject to the bounded-feature assumption.Source: B. Notation and Standing Conventions · p. 4

C. The Rank-One Extremality Principle

  • Per-factor Frobenius budgets yield exactly the rank-at-most-r matrices in a nuclear-norm ball, and the studied support-functional and Frobenius-norm suprema are attained at rank one independently of r.The two extrema govern the statistical-complexity and Wasserstein-displacement analyses, respectively.Source: C. The Rank-One Extremality Principle · p. 4

D. Discrepancy, Cone, and Risk

  • The section defines discrepancy and transferability using induced readout classes; the divergence-defined cone can have ambient-scale radius when an admissible adapted weight is singular, motivating a pushforward restriction.Source: D. Discrepancy, Cone, and Risk · pp. 4–5

III. Q1: STATISTICAL COMPLEXITY—COLLAPSE, AND WHAT IT WOULD TAKE TO AVOID IT

Under the per-factor budgets, rank does not change the scalar linear-readout class or the matrix-valued complexity studied here. Rank-sensitive complexity appears for the matrix-valued setting under a joint budget, but the worst-case bound remains rank-free.

A. The Baseline Adaptation Bound

  • The adaptation bound separates target risk into source performance, distributional discrepancy, and ideal joint risk, locating complexity and spectral alignment as distinct terms.Rademacher complexity controls the first; spectral alignment and transport address the discrepancy term.Source: A. The Baseline Adaptation Bound · p. 5

B. Collapse at the Scalar Level

C. Collapse Persists at the Matrix Level

  • The matrix-valued adaptation-layer Rademacher complexity is also rank-free, although the underlying update sets can differ across ranks.The result concerns the computed vector-valued complexity, not identity of the matrix classes or every possible nonlinear-readout composition.Source: C. Collapse Persists at the Matrix Level · p. 5

D. Rank-Sensitivity Under the Joint Budget

  • Under a joint product budget, matrix-valued complexity has a data-dependent rank-sensitive bound, while its worst-case simplification remains rank-free.The operator-norm branch can improve on the Frobenius branch in an effective-rank regime; under worst-case features, the Frobenius branch applies.Source: D. Rank-Sensitivity Under the Joint… · p. 6

IV. Q2: SPECTRAL STRUCTURE AND THE THRESHOLD RANK OF CANCELLATION

The spectral analysis studies cancellation of the leading singular directions of the pretrained weight and the rank needed for a cancellation certificate. The required budget grows with the canceled block, and threshold bounds depend on spectral decay and alignment conditions.

A. Cancellation and the Tail Profile

  • Canceling the top-r singular block is feasible under a nuclear-norm budget condition, and the required budget grows with r.The remaining weight is the residual SVD tail; the tail profile is nonincreasing, making the spectral threshold searchable by increasing rank.Source: A. Cancellation and the Tail Profile · pp. 6–7

B. From Alignment Error to Discrepancy

  • Under alignment control along the cancellation family, the smallest rank meeting the discrepancy tolerance has upper bounds shaped by the singular-value decay of W0.The alignment assumption is invoked for cancellation weights; the text gives a concrete Gaussian model where it holds uniformly along that family.Source: B. From Alignment Error to Discrepancy · p. 7
  • In the non-saturated regime, lower bounds for the cancellation certificate match the upper bounds’ dependence on tolerance and decay when directional alignment is comparable to ∥∆Σ∥2.These lower bounds apply only to the top-singular-block cancellation family; they do not rule out other rank-r updates certifying the tolerance at smaller rank.Source: B. From Alignment Error to Discrepancy · p. 7

C. Generalization Bounds: Transport and Spectral Routes

The paper gives generalization bounds through transport and through spectral cancellation. The transport route uses a rank-free worst-case complexity term, while the spectral route connects cancellation to a discrepancy certificate and its own complexity.

C. Generalization Bounds: Transport and Spectral Routes

  • The transport-based generalization bound uses a rank-free worst-case complexity term, with a data-dependent rank-sensitive refinement noted separately.The stated setting includes absolute loss, a finite first moment for the target distribution, and a sample-size condition.Source: C. Generalization Bounds: Transport… · pp. 7–8
  • At the spectral threshold rank, feasible top-block cancellation certifies discrepancy at most ϵ and can also shrink the certificate class’s complexity.This route yields a sample-based generalization statement and is complementary to, not a consequence of, the transport route.Source: C. Generalization Bounds: Transport… · p. 8

V. Q3: GEOMETRY OF THE TRANSFERABILITY CONE

The geometry results distinguish a rank-independent bound on adaptation-induced displacement from a divergence-defined cone whose radius can reflect ambient support instead. They also show how cancellation membership depends on the budget for the canceled spectral block.

V. Q3: GEOMETRY OF THE TRANSFERABILITY CONE

  • When an admissible adapted weight has a nontrivial kernel, the divergence-defined cone can have infinite radius on Rd or radius at least Xmax/2 on a bounded ball.That radius reflects ambient support and is insensitive to ϵ, r, and ρ within the stated premise, so it does not measure adaptation.Source: §V · p. 8
  • The two-witness pushforward cone has a rank-independent reachable-radius upper bound that is sharp under a mild budget condition.When the budget covers the weakest input direction of W0, an isotropic Gaussian source attains equality through a rank-one update.Source: §V · pp. 8–9
  • Under alignment control, a target belongs to the pushforward cone if its tail alignment is at most ϵ/κ and the budget covers the canceled top block.The required nuclear budget grows with the cancellation rank, so rank affects affordability rather than the fixed-budget radius.Source: §V · p. 9

VI. NUMERICAL ILLUSTRATIONS

  • The numerical section uses synthetic, linear, single-layer experiments to illustrate predicted regimes and probe how low-nuclear-norm solutions are selected.The authors explicitly make no claim about LoRA fine-tuning of deep networks; the common setup is teacher–student regression with Gaussian inputs and a rank-4 teacher update.Source: §VI · p. 9

A. Rank-One Extremality of the Worst Case

Across the tested regimes, rank does not increase worst-case capacity once the nuclear budget can be spent; it instead controls affordability and the spectral cost of adaptation. Rank-sensitive complexity appears only under a joint budget for well-spread features, while spectral threshold rates depend on directional alignment.

A. Rank-One Extremality of the Worst Case

  • The worst-case gap stays flat across nominal ranks, and the maximizing update has effective rank 1 at every rank and initialization.For ρ = 1 and ρ = 4, rank 1 reaches 91%–96% of the best value found at any rank; the maximizer saturates the nuclear budget.Source: A. Rank-One Extremality of the Worst… · p. 9, Figure 1 · p. 10

B. The ERM Gap: Affordability, then Rank-Freeness

C. Spectral Threshold Scaling and the Role of Alignment

D. The Complexity Branches Across Feature Spectra

  • The joint-budget complexity is rank-sensitive only for well-spread features; under per-factor budgets it is flat in rank, and near-collinear features gain nothing.For well-spread features, the joint-budget value grows until saturation tracks effective rank.Source: D. The Complexity Branches Across… · pp. 10–11

E. Implicit Nuclear-Norm Bias of Training

  • In rank-4 target recovery, learned updates remain near the target’s nuclear norm and effective rank even at nominal rank 64, with or without weight decay.The nuclear-norm drift from rank 4 to 64 was 12.7% at zero weight decay versus 11.9–13.3% across the weight-decay grid; whether weight decay matters at longer horizons or in deep nonlinear models remains open.Source: E. Implicit Nuclear-Norm Bias of… · p. 11

VII. DISCUSSION AND CONCLUSION

  • The scalar linear-readout class and its studied complexity are rank-free; rank-sensitive statistical gains arise at the matrix level under a joint budget when feature effective rank is large.The worst-case simplification of the joint-budget bound remains rank-free.Source: §VII · pp. 11–12
  • Rank sets the budget needed for top-block cancellation, with matching threshold-rank bounds under two-sided spectral decay and alignment; the fixed-budget transport radius is rank-independent and can be sharp.The threshold bounds match in order under comparable directional constants and away from rank saturation; the transport bound is sharp under the stated mild budget condition.Source: §VII · pp. 11–12
  • The paper’s practical conclusion is to select rank by pretrained-weight spectral decay and budget, not as a capacity control.The effective rank of the feature distribution determines whether a statistical benefit is available.Source: §VII · p. 12

APPENDIX A AUXILIARY RESULTS

The auxiliary results connect rank-one extremality to both statistical complexity and distributional displacement. They also show how leading singular directions can be canceled when the budget covers their nuclear norm, while clarifying that the induced Wasserstein bound need not be tight.

Auxiliary results

  • The frozen weight contributes nothing to vector-valued adaptation complexity, while the Rademacher functional and Frobenius displacement are both maximized by rank-one updates.The distinction is that the frozen matrix is an additive constant when only the update is optimized.Source: Appendix A · p. 14
  • If the budget covers the top-r singular values of W0, canceling those directions realizes the spectral tail and gives alignment value τr.The cancellation update is −(W0)≤r, feasible when ρ ≥ ∑i≤r si(W0).Source: Appendix A · p. 14
  • For the cancellation pushforward, the linear map gives a Wasserstein upper bound, but it need not be the optimal transport coupling.Equality requires additional conditions, such as Gaussian marginals with commuting covariances.Source: Appendix A · p. 14
  • A rank-one Gaussian construction attains the rank-independent displacement bound when the budget covers the weakest input direction of W0.The construction uses separate rank-one updates to generate the pushforward and certify transferability.Source: Appendix A · p. 15

APPENDIX B SECONDARY RESULTS AND EXTENDED REMARKS

The appendix clarifies where rank sensitivity disappears and where it can re-enter. Scalar linear-readout complexity remains rank-free, while a joint matrix-valued budget can yield data-dependent gains under well-spread features; spectral cancellation also depends on both rank and budget.

Secondary results

  • The discrepancy used here is a continuous-valued supremum over expected absolute score differences, not the binary HΔH-divergence.The results use this discrepancy directly and do not require a margin condition linking soft and hard disagreement.Source: Appendix B · p. 15
  • Scalar linear-readout classes remain rank-free across fixed or free readouts and per-factor or joint budgets, and adding W0 does not change these conclusions.Rank dependence is possible at the matrix-valued level under a joint rank-and-Frobenius constraint, but not for the scalar linear formalizations considered.Source: Appendix B · pp. 16, 18
  • Under a joint budget, the rank-sensitive complexity bound can improve for well-spread features, but not for collinear features or in the worst case.The comparison is sufficient rather than an exact characterization; the worst-case bound remains rank-free.Source: Appendix B · p. 16
  • A spectral threshold rank identifies when cancellation meets an alignment tolerance, but it is affordable only if the budget covers the canceled singular values.When any budget-feasible certifying rank exists, it equals the minimally certifying spectral rank.Source: Appendix B · p. 17
  • The frozen weight is absent from vector-valued layer complexity but enters the transport term through the input-space Lipschitz constant.The distinction follows because W0 is not optimized against the sample, whereas transport acts through the layer output.Source: Appendix B · p. 17

A. Proof of Lemma 1

These proofs establish the budgeted reachable set, the rank-one extremality principle, the risk decomposition, and the rank-invariant scalar linear-readout class.

A. Proof of Lemma 1

  • Per-factor Frobenius budgets yield exactly the updates of rank at most r whose nuclear norm is at most ρ.The proof factors a compact SVD and rescales its two factors while preserving their product.Source: A. Proof of Lemma 1 · p. 18

B. Proof of Lemma 2

  • The linear support functional and Frobenius norm attain their maxima over the reachable set at rank-one updates, for every r ≥ 1.For the support functional, the maximizing update aligns with the top singular vectors of the signal matrix.Source: B. Proof of Lemma 2 · p. 18

C. Proof of Theorem 1

  • The target-risk bound follows by applying the triangle inequality through a source-optimal predictor and then taking an infimum.The proof chains the target predictor to the source predictor under the target and source distributions.Source: C. Proof of Theorem 1 · p. 18

D. Proof of Proposition 1

  • Every scalar linear-readout function attainable at rank r has a rank-one surrogate with the same readout and frozen-weight term, so the function class is identical across ranks.Consequently, its covering and chaining functionals cannot distinguish ranks.Source: D. Proof of Proposition 1 · p. 18

E. Proof of Proposition 2

The proofs give an exact rank-free complexity for per-factor budgets, establish rank-free scalar results under joint-budget variants, and derive an exact matrix-valued joint-budget complexity with rank-sensitive bounds.

E. Proof of Proposition 2

  • The vector-valued Rademacher complexity under per-factor budgets is exactly ρ E∥S∥2/n, independent of r.The proof reduces the sample sum to a matrix inner product and applies the rank-one extremality result.Source: E. Proof of Proposition 2 · p. 18

F. Proof of Proposition 7

  • Scalar linear-readout complexity remains rank-free when the readout is fixed or free, even under the joint Frobenius budget.The relevant signal matrix is rank one, so truncation at any r ≥ 1 leaves it unchanged.Source: F. Proof of Proposition 7 · p. 19

G. Proof of Lemma 3

  • Under the joint rank-and-Frobenius constraint, the exact matrix-valued complexity is ρ E∥Sr∥F/n; its rank-sensitive branch has a rank-free Frobenius baseline.The operator-norm bound can improve under suitable effective rank, while the worst-case simplification is rank-free.Source: G. Proof of Lemma 3 · p. 19

H. Proof of Corollary 2

These proofs establish complexity bounds under two constraint sets, characterize how the spectral threshold rank varies with rank, and derive a source-to-target generalization bound using transport and concentration. The threshold-rank certificate is paired with decay-dependent upper and lower bounds under directional alignment.

H. Proof of Corollary 2

  • The two constraint sets yield worst-case complexity bounds, with the joint-budget set admitting a data-dependent minimum; the worst-case bounds hold uniformly over samples satisfying Assumption 1.The proof uses E∥S∥F ≤ Xmax√mn to obtain the stated bounds.Source: H. Proof of Corollary 2 · p. 19

I. Proof of Lemma 7

  • τr is nonincreasing in r, so the set of ranks satisfying τr ≤ ϵ/κ is upward closed and nonempty.For r at least rank(W0), τr vanishes.Source: §I · p. 19

J. Proof of Theorem 2

  • The top-block cancellation at rspec(ϵ) certifies discrepancy at most ϵ, with upper bounds on rspec(ϵ) under polynomial and geometric spectral decay.The proof derives the decay-dependent bounds from τr ≤ sr+1(W0)^2∥∆Σ∥2.Source: J. Proof of Theorem 2 · pp. 19–20

K. Proof of Proposition 3

  • Under uniform directional alignment and in the non-saturated regime, Proposition 3 gives lower bounds on rspec(ϵ) corresponding to polynomial or geometric spectral decay.The proof applies the lower bound at r = rspec(ϵ).Source: K. Proof of Proposition 3 · p. 20

L. Proof of Theorem 3

  • For LψM > 0, the transport function φ=|h−h′| is 2LψM-Lipschitz, and uniform sample-wise source-complexity bounds also bound the expected complexity used in the source-loss argument.For contraction, the loss map is centered as ˜ϕi(t)=|t−fS(xi)|−|fS(xi)|, making it 1-Lipschitz and zero at 0.Source: L. Proof of Theorem 3 · p. 20

M. Proof of Corollary 1

The proofs connect fixed-weight discrepancy and generalization bounds to complexity estimates, then characterize distribution displacement under kernel translations and feasible updates. They also show how top-block cancellation certifies membership in the transferability cone.

M. Proof of Corollary 1

  • The fixed-weight proof combines a discrepancy bound with a source generalization bound, and specializes the complexity estimate to linear readouts.The bound uses symmetrization and McDiarmid; its linear-readout step uses the singular-value tail of W0.Source: M. Proof of Corollary 1 · p. 20

N. Proof of Proposition 4

  • Translations along a unit vector in ker(W) preserve every readout’s output law, giving zero hypothesis discrepancy; within a radius-Xmax ball, the construction reaches W2-distance Xmax/2.The stated distance is independent of ϵ, r, and ρ within the kernel premise.Source: N. Proof of Proposition 4 · p. 21

O. Proof of Theorem 4

  • For feasible updates, the pushforward displacement bound is sharp when sd(W0) ≤ ρ, for both constraint sets.The proof uses the update Frobenius-norm supremum ρ, with no factor of √r.Source: O. Proof of Theorem 4 · p. 21
  • Top-block cancellation under the stated nuclear budget makes the update admissible and certifies transferability-cone membership when κτtail ≤ ϵ.The discrepancy bound is evaluated at W = (W0)>r.Source: O. Proof of Theorem 4 · p. 21
Loading 2609.32002v2…