How Much Rank Does LoRA Need? Rank-Error Bounds for Transformer Attention

Gerard Conangla Planes

Source-linked AI summary

How Much Rank Does LoRA Need? Rank-Error Bounds for Transformer Attention

Gerard Conangla Planes

arXiv:2608.26052v1Aug 26, 2026cs.LGcs.AIcs.CL

TL;DR

LoRA rank is usually selected empirically, but the paper asks how much task-dependent attention approximation error each rank can achieve. It develops bounds linking centered score error to KL and downstream-weighted spectral tails, then extends the theory to alternative regimes, softmax saturation, fused multi-head updates, and joint query/key LoRA. The resulting rank-indexed curves characterize representational error at the attention layer, while leaving optimization and downstream output guarantees outside scope.

Problem
Empirical rank sweeps show which LoRA rank worked but do not determine whether smaller adapters lacked capacity or were harder to train.
Method
The paper bounds the smallest expected attention KL over rank-r candidates by combining global softmax comparisons with downstream-weighted spectral approximation, under explicit assumptions where required.
Results
The theory produces two-sided, rank-indexed error curves based on task-activated tail energy, with additional bounds for weak probability floors and extensions to saturation, fused heads, and joint query/key updates.
Takeaways & Limitations
LoRA rank can be evaluated through the target update component activated by downstream queries and keys, rather than through raw update singular values alone.
Takeaways & Limitations
The guarantees concern best candidates’ attention KL, not optimizer behavior, head outputs, or final task loss, and rely on theorem assumptions that finite samples do not certify.

Abstract

from arXiv · show

Choosing the rank of a low-rank adaptation (LoRA) update is usually an empirical task. In this paper, we provide a task-dependent theory of the approximation error achievable at each LoRA rank for Transformer attention. We fix a pretrained attention head, a target attention function, and a distribution over inputs from the downstream task, and bound the smallest expected Kullback--Leibler (KL) error achievable by a rank-$r$ query LoRA update. When target attention probabilities are bounded away from zero, we prove a lower bound of the error proportional to $ψ(\|d\|_2)$, where $d$ is the difference between candidate and target attention scores and $ψ(t)=\min\{t^2,t\}$. We also prove an unconditional upper bound $\min\{\|d\|_2^2/4,\sqrt2\|d\|_2\}$. Under explicit realizability, geometry, and moment conditions, we then bound the best rank-$r$ error between an explicit multiple of $ψ(\sqrt{T_r})$ and $\min\{T_r/4,\sqrt{2T_r}\}$, where $T_r$ is the downstream-weighted tail energy of the target update. We also provide target-Fisher bounds when candidate scores remain within a fixed range of the target scores, and an unrestricted lower bound when a subset of tokens carries most of the probability mass. These spectral bounds describe finite-score approximation. We then construct explicit families in which softmax saturation makes the rank required to match the attention function strictly smaller than the rank required to match the finite logits. Finally, we extend the analysis to fused multi-head LoRA and joint query/key updates, exposing the effects of rank sharing and query/key factorization constraints.

1 Introduction

The paper develops a task-dependent theory for how well rank-r LoRA updates can reproduce target Transformer attention, replacing empirical rank selection with rank-indexed approximation bounds. It connects centered score error to attention KL, then connects that score error to downstream-weighted spectral tails, while extending the analysis to alternative assumptions, saturation, multi-head updates, and joint query/key adaptation.

Motivation: LoRA rank reduces trainable parameters and computation but limits adapter representational power, so empirical rank sweeps cannot distinguish capacity limits from optimization difficulty.The paper studies this representation question for a fixed pretrained head, target attention function, and downstream input distribution.
Problem and scope: Attention KL measures how much a query update changes the attention distribution before later Transformer layers, rather than directly measuring head output or final task loss.The analysis begins with query-only adaptation in one head and compares candidate attention probabilities with those of a dense or high-rank target adapter.
Score-to-KL connection: Theorem 3.1 makes pointwise KL comparable to ψ(∥d∥2) under a probability floor and gives an unconditional upper bound ψup(∥d∥2), yielding quadratic small-error and linear large-error behavior.Here d is centered candidate-minus-target score error; Corollary 3.2 lifts the result to the rank-constrained expected objective Er.
Rank-dependent bounds: Under explicit realizability, geometry, probability-floor, and moment assumptions, Theorem 4.1 brackets best rank-r error using the downstream-weighted spectral tail energy T_r.The upper bound constructs a candidate, while the lower bound applies to every candidate, producing a rank–error curve that can be compared with a desired tolerance.
Alternative bounds: When the main lower-bound constant is weak, the paper provides target-Fisher bounds for score-restricted candidates and a high-mass lower bound for the unrestricted candidate class.The alternatives use different assumptions and do not both provide unrestricted full-KL guarantees.
Extensions and saturation: Softmax saturation can make the rank needed to approximate an attention function strictly smaller than the rank needed to match finite logits, while fused multi-head and joint query/key LoRA introduce sharing and factorization constraints.An explicit Walsh family gives a constant-factor rank separation, and later theorems extend the rank–KL analysis to the two broader adaptation settings.

3 From score error to attention KL

The paper connects centered attention-score error to KL error through robust softmax bounds, then uses task-weighted spectral structure to bound rank-constrained approximation. The resulting bounds depend on explicit probability, dependence, and moment assumptions, with alternatives when the global lower bound is weak.

Global softmax bounds: Theorem 3.1 relates centered score error to attention KL, with a quadratic regime for small errors and a linear regime for large errors.The upper bound is unconditional, while the lower bound requires target probabilities bounded away from zero; Corollary 3.2 lifts the result to expected rank-constrained error.
Task-weighted spectral approximation: The task-weighted residual T_r measures score error after the best rank-r update, removing directions annihilated by keys or never activated by downstream queries.This weighting can favor a smaller raw singular direction that changes attention frequently over a larger direction the task never uses.
Task-weighted spectral rank–KL bounds: Under realizability, probability-floor, geometry, independence, and moment conditions, Theorem 4.1 brackets the best rank-r KL error between explicit lower and upper functions of T_r.The upper candidate comes from truncated SVD in task-weighted coordinates, while the lower bound applies to every rank-r candidate.
Task-weighted spectral rank–KL bounds: The rank bounds can be inverted to obtain sufficient and necessary rank conditions for a desired error tolerance.The resulting rank–error curve can be estimated from downstream data and compared with a target tolerance.
Assumptions and limitations: The theorem’s lower-bound constants may be far weaker than its upper-bound constants when the target probability floor is small.For a = 10^-2, Λ = 5, and κ_h = 3, the worst-case ratio is about 2.0 × 10^4; dependence-allowing bounds replace independence with direct comparison and moment conditions.

5 Which rank–KL bound should be used?

The paper offers three rank–KL routes selected by the assumptions that can be justified: a main theorem, target-Fisher bounds for score-restricted candidates, and an unrestricted high-mass lower bound.

Choosing a route: Use the decision diagram by stopping at the first theorem whose assumptions are justified; upper bounds concern some candidate, whereas lower bounds concern every candidate.The Fisher lower bound specifically excludes the unrestricted optimum Er.
Choosing a route: Table 2 summarizes the three rank–KL routes and their logical scope.
Target-Fisher bounds: The target-Fisher route applies when candidate score differences have range at most R0 and bounds optimal KL using the target-Fisher quadratic optimum.Its factors are c+(R0) and c−(R0).
Target-Fisher bounds: The target-Fisher lower bound cannot be transferred to the unrestricted optimum Er because it concerns only a restricted candidate class.The restriction is a span condition, and small target probabilities enter through the Fisher matrix H*.
High-mass lower bound: The high-mass route selects token sets carrying at least 1 − δ of target probability mass and assumes their conditional target probabilities are bounded below.This route focuses the lower bound on tokens that carry most of the target mass.
High-mass lower bound: The high-mass theorem lower-bounds the unrestricted Er but does not provide a full-KL upper bound.Matching conditional scores on the selected tokens need not match the total mass assigned to those tokens.

6 Saturation changes the relevant notion of rank

Softmax saturation separates finite-logit rank from attention-function rank, while extensions to fused multi-head and joint query/key LoRA add structural rank constraints.

Saturation changes rank: softmax(T, 0) and softmax(2T, 0) converge to the same attention distribution even as their logit difference grows with T.This is the basic saturation mechanism behind the rank separation.
Saturation changes rank: The closure rank measures the smallest rank whose score matrices converge, within the allowed score space, to a boundary attention target.For query adaptation, that score space is restricted by the fixed key matrix.
Saturation changes rank: For the isolated-triple Walsh family, finite-logit realization requires rank k, whereas saturated attention can be recovered with a lower closure rank.The equality is specific to the constructed family; other target families may differ.
Saturation changes rank: The Walsh construction achieves a constant-factor reduction through rank-two paths per isolated triple, but its attention-function rank does not collapse to o(k).Restricted Fourier identities and columnwise normalization supply the lower bound.
Additional saturation family: A separate linear-token family reaches ratio 4/7 on complete seven-context blocks, but it does not establish that ratio as its minimum closure rank.The two constructions concern different target families.
Multi-head and joint updates: Fused multi-head LoRA shares one rank budget across vertically stacked head updates, so a fused rank direction can serve several heads.The resulting bounds depend on the shared constraint rank(M) ≤ r and on the aggregation of KL across heads.
Multi-head and joint updates: The fused multi-head bracket can widen by a factor of order H, reflecting the summed headwise KL objective.Independent per-head adapters instead separate after ranks are fixed, leaving a discrete total-budget allocation problem.

8 Joint query/key LoRA

Joint query/key LoRA is analyzed through an effective score update constrained by separate query and key ranks. The resulting bounds include factorization error, while RoPE can prevent the single-matrix spectral specialization from applying automatically.

The effective query/key score update has rank at most rQ + rK, providing the main containment used for joint-update bounds.
The joint analysis compares weighted rank-s approximators with those realizable by separate query and key updates.A weighted optimizer need not admit the required query/key factorization.
Theorem 8.1 assumes a positive target-probability floor and independence between G_X(u) and h(u), alongside feasibility conditions for the candidate factors.
The factorization gap can be zero under exact one-sided support or sequential two-sided factorization, but may otherwise be large.
The bounds are useful only when the factorization-gap term is comparable to or smaller than the weighted spectral scale.
With RoPE, each relative-position slice retains rank at most rQ + rK, but the coupled effective matrices invalidate automatic use of the single-matrix spectral tail.

9 Estimating LoRA rank from downstream data

Given a target adapter and downstream calibration data, the procedure estimates task-weighted rank–KL curves and compares them with a desired tolerance. These curves can certify or exclude ranks, but population assumptions, optimization, and architecture scope limit the conclusions.

The theory outputs upper and lower rank–error curves that can certify a rank, exclude it, or leave the choice unresolved.
The applicable bound depends on target realizability, separability or dependence geometry, and whether the target probability floor is sufficiently informative.
Calibration uses complete downstream inputs to collect query activations, centered key Gram matrices, and target attention probabilities.Complete prompts, rather than isolated token positions, are the sampling units.
The main spectral route estimates G and Σ, forms the weighted target matrix, and uses its truncated SVD to construct a rank-r candidate.
If Ur ≤ ϵ, the constructed candidate meets tolerance; if Lr > ϵ, no rank-r candidate in the relevant class can meet it.
The curves lack finite-sample confidence guarantees, and the method does not ensure an optimizer finds the candidate attaining the upper bound.
The analysis concerns attention probabilities rather than head outputs or task loss, and RoPE limits the joint query/key spectral specialization.

Guide to the appendix

The appendices provide proofs for the paper’s softmax, approximation, moment, alternative lower-bound, saturation, and multi-head or joint query/key results. They also include computational checks that support reproducibility but are not part of the proofs.

Appendices A–D prove the global softmax, moment, spectral, and dependence-allowing rank–KL bounds.
Appendices E and F develop the target-Fisher and high-mass alternative lower-bound routes.
Appendix G contains saturation constructions and the closure-rank lower bound.
Appendices H and I prove the fused multi-head and joint query/key extensions.
The accompanying code verifies finite constructions, rank calculations, and selected inequalities for reproducibility and debugging, not for proving the results.
The global softmax proof treats KL divergence as the log-sum-exp Bregman divergence.
A fixed-floor construction shows that best rank-one quadratic error can approach one while expected attention KL approaches zero.

C Moment and spectral lemmas

The moment and spectral lemmas connect downstream-weighted score error to a truncated weighted SVD. Under independence, geometry, and moment assumptions, this yields robust lower and constructive upper bounds for rank-constrained approximation.

The fourth-moment bridge converts a second-moment quantity into bounds involving ψ, under a fourth-moment condition.
Independence between G(u) and h(u), together with G(u) ⪯ ΛG, supplies the uniform moment control used by the theory.
The weighted target matrix is D* = G^1/2 Δ* Σ^1/2, whose truncated SVD determines the rank-r approximation.
The weighted spectral optimizer remains valid when G or Σ is singular.
The optimizer Mr attains the weighted tail because its transformed form equals the rank-r truncated SVD and rank(Mr) ≤ r.
Every feasible candidate has score-error objective at least Tr, while the weighted-SVD candidate achieves Tr and produces the corresponding upper bound.

E Target-Fisher bounds

Target-Fisher bounds refine rank–KL guarantees when candidate score differences stay within a fixed range of the target. They relate constrained approximation to Fisher-weighted spectral tails and offer alternatives when global probability-floor constants are weak.

Fisher geometry: If αp_i ≤ q_i ≤ γp_i, then the Fisher matrices satisfy αH(p) ⪯ H(q) ⪯ γH(p).The comparison lets target-based Fisher geometry control candidate Fisher geometry when probabilities remain multiplicatively comparable.
Fisher-weighted bounds: Theorem 5.1 supplies pointwise lower and upper KL bounds through monotone constants c− and c+ as functions of the score-error range.The lower constant is nonincreasing in the range, while the upper constant is nondecreasing.
Fisher-weighted bounds: Corollary E.1 transfers these pointwise inequalities to the best rank-r candidates whose score differences have range at most R0.The constrained class must be nonempty for the result to apply.
Fisher-weighted bounds: The unrestricted rank-r Fisher objective is governed by a weighted tail energy obtained from the Fisher-weighted SVD.If the Fisher-weighted SVD optimizer is feasible under the range constraint, it attains the constrained quadratic infimum and gives a matching upper bound.
High-mass alternatives: When a token subset carries most target mass, restricting the KL analysis to that subset yields an unrestricted lower bound using restricted logits and geometry.The proof uses target mass at least 1−δ and core probability and moment conditions.

G Saturation constructions and lower bound

Softmax saturation can reduce the rank needed to approach a limiting attention function below the rank needed to realize its finite logits. Explicit Walsh constructions achieve and match this reduction through isolated triples and diverging score gaps.

Closure-rank formulation: Softmax closure rank permits rank-r score sequences whose columnwise softmax converges to a boundary target, even when finite-logit matching requires higher rank.For query adaptation, the allowed score space is restricted by the fixed key matrix.
Finite-logit rank: At every finite T, exact realization of the Walsh target requires update rank k.Centered-softmax injectivity and orthonormality force the exact update to equal TIk.
Achievable construction: A triple-block construction uses rank-two paths to achieve closure rank at most 2q+s = k−⌊k/3⌋.The construction makes within-support variation vanish while the support gap diverges.
Achievable construction: A seven-character packing gives rank at most k−3⌊k/7⌋ with fewer than 8(k+1) tokens.The packed construction uses rank-four factors for each seven-character block.
KL convergence: The constructed candidate distributions converge in KL because off-support mass vanishes, within-support KL vanishes, and score ranges grow only polynomially in T.The remaining rare-mass contribution is bounded by εT poly(T) and tends to zero.
Token-efficient family: The triple construction uses fewer than 4k^2+8 tokens while retaining exact finite-logit rank k and closure rank at most 2q+s.The token-set construction isolates additive triples and controls the number of leftovers.
Lower bound: Every limiting real triple block has rank at least two, so saturation yields a constant-factor reduction but not o(k) closure rank in this family.The lower bound follows from restricted Fourier identities, normalization, and closedness of the rank-constrained matrix set.

H Fused multi-head proofs

The fused multi-head extension aggregates headwise rank–KL bounds under a shared rank allowance. Its constants are tight, while granting every head the full allowance can make the resulting bound loose.

Error aggregation: The aggregate error obeys the same quadratic-to-linear transition as ψ, with constants attained when head errors are respectively small or equal.The quadratic branch is attained when all Xh are small; the linear branch is attained when errors are equal across heads.
Fused-head bound: For fused multi-head adaptation, each head block has rank at most the shared allowance s, enabling headwise weighted low-rank approximation.The proof combines weighted Eckart–Young headwise with an aggregate moment bridge.
Independent adapters: Independent head adapters yield a Cartesian-product feasible set once integer head ranks are fixed, producing an allocation identity.This separates rank allocation across independently parameterized heads.
Output bridge: Head-output differences are controlled through value-vector diameters and output-projection blocks before total variation and Pinsker convert them to attention-output error.The multi-head proof first bounds each head’s value-weighted output difference.

I Joint query/key proofs

Joint query/key LoRA is analyzed through the effective score update, whose rank is constrained by separate query and key factors. The resulting bounds separate effective spectral approximation from the additional realizability price of factorization.

Rank structure: The effective score update decomposes into two summands with ranks at most rank(A) and rank(B), and the rank-2r example shows the sum bound is sharp.For K0=A=[Ir 0] and Q0=B=[0 Ir], the resulting effective update has rank 2r.
Rank structure: The joint candidate class also satisfies rank((K0+B)^T(Q0+A)) ≤ p, giving a width-aware rank constraint.This constraint comes from the ambient feature width p.
Effective-rank containment: Every joint query/key candidate lies in the effective rank-s class with s = rQ+rK, plus a width-aware subclass.This containment permits effective-score rank bounds even when factorization restricts which updates are realizable.
Approximation bounds: Weighted Eckart–Young–Mirsky supplies the effective spectral lower bound, while fourth-moment assumptions bridge score approximation to KL error.The proof takes the infimum over feasible factor pairs after applying the moment-based error bound.
Realizability: The factorization realizability price vanishes when column-space and row-space conditions permit the effective update to be realized within the separate rank budgets.Sequential and one-sided realizability follow from projector identities under the stated conditions.
Constructive upper bound: An always-valid factor-dependent upper bound controls score error through the two factor errors and their pretrained counterparts.The factor perturbation identity yields a constructive upper bound after submultiplicativity and the global softmax bound.
Multi-head extension: The fused multi-head joint bound applies the shared allowance s independently to every head and is therefore safe but potentially loose.It combines headwise weighted approximation with the aggregate moment bridge.