PaperlayerView exact revision on arXiv

Source-linked AI summary

How Finite-Rank Bottlenecks Shape the Low-Rank Adaptation Landscape

Long Nguyen-Chi, Quynh Nguyen, Thanh Nguyen-Cung, Binh T. Nguyen

arXiv:2610.23343v1cs.LG

TL;DR

The paper addresses why low-rank updates suffice when existing explanations often rely on NTK linearization. It analyzes frozen-network bottlenecks and trace-norm regularization, finding rank-controlled global minimizers and near-optimality for perturbed factorized minimizers.

  • Problem

    The objective-level conditions that make low-rank optima available remain unclear, while much existing theory relies on linearization around pretrained weights.

    Source: §1 · pp. 1–2

  • Method

    The paper models each frozen-network update through a finite-rank linear map and studies trace-norm and perturbed factorized LoRA objectives.

    Source: §1 · p. 2, §3 · p. 4

  • Results

    Bottleneck rank controls the rank of global minimizers, and global minimizers of the perturbed factorized objective are near-optimal for the original trace-norm problem.

    Source: §1 · p. 2, §6 · p. 10

  • Takeaways & Limitations

    Frozen features provide an exact, non-linearized basis for understanding low-rank optima in the stated setting.

    Source: §6 · p. 10

    The rank theorems establish existence rather than gradient-based LoRA convergence, and the attention bounds are a posteriori.

    Source: §6 · p. 10

Abstract

from arXiv · show

Low-rank adaptation (LoRA) has become the standard method for parameter-efficient fine-tuning of large pretrained models, yet theoretical explanations for why low-rank updates suffice remain incomplete. Most existing analyses rely on the Neural Tangent Kernel (NTK) regime, which linearizes the network around the pretrained weights. In this work, we take a different approach that requires no linearization: our key observation is that, once the backbone is frozen, the loss sees each adapted weight matrix only through a finite-rank linear map induced by the frozen activations. Trace-norm regularization then forces the optimizer to pick a low-rank matrix among all matrices that produce the same observed output. This yields global minimizers whose ranks are controlled by the rank of this map, which we call the \emph{bottleneck rank}, rather than the sample size. For multi-head attention, we obtain tighter rank upper bounds by exploiting two additional invariances: row-wise softmax is insensitive to additive row constants in the score matrix, and value updates are observable only after attention weighting and frozen output projection. For the factorized LoRA objective with weight decay, we prove a complementary result on the nonconvex side: after an arbitrarily small generic positive semidefinite perturbation, every first-order stationary point is rank-deficient once the LoRA rank exceeds a threshold determined by the bottleneck rank. Finally, we show that global minimizers of the perturbed objective are near-optimal for the original trace-norm problem, and we provide uniform generalization bounds governed by the spectral structure of the frozen features.

1 Introduction

The paper asks what in the frozen fine-tuning objective makes low-rank optima available, then answers with finite-rank bottlenecks and trace-norm regularization. It extends the account to attention and factorized LoRA, including perturbation-based results.

1 Introduction

  • LoRA adapts selected linear maps while freezing the pretrained backbone, but its rank is usually chosen as a hyperparameter rather than derived from the objective.The paper asks when the regularized objective admits a low-rank global minimizer and what controls its rank.Source: §1 · pp. 1–2
  • Earlier analyses studied linearized objectives or training dynamics, leaving open what structural property of the objective makes low-rank optima available.The introduction identifies this as the paper’s central question.Source: §1 · p. 2
  • For a frozen network, the loss observes an adapted block through a finite-rank linear map, and trace-norm regularization yields a global minimizer whose rank is controlled by that map’s bottleneck rank.The result extends to multiple adapted stages and does not require linearizing the network around pretrained weights.Source: §1 · p. 2
  • Attention yields tighter subspace rank bounds by using row-wise softmax invariance to additive row constants and the fact that value updates are observed only after attention weighting and frozen output projection.Projector compression combines these invariances in the optimizer-dependent bounds.Source: §1 · p. 2
  • After a generic positive semidefinite perturbation, full-column-rank first-order stationary points are excluded when the Pataki–Barvinok rank threshold exceeds the symmetric KKT-matrix-family dimension.This is a first-order exclusion result, not a guarantee against spurious local minima.Source: §1 · p. 2
  • Global minimizers of the perturbed factorized objective are near-optimal for the original trace-norm-regularized problem when the perturbation is small.The paper also provides uniform generalization bounds governed by frozen-feature spectral structure.Source: §1 · p. 2

2 Related works

The related-work discussion distinguishes structural questions about frozen objectives from analyses of linearized objectives, LoRA training dynamics, and transformer expressivity.

Linearized and kernel views of fine-tuning

  • The comparison with Jang et al. (2024) contrasts an NTK low-rank existence result with an exact reduction for frozen upstream computation that does not linearize around pretrained weights.The paper states an attention rank scale of r ≤ min{Kout, N}, better than Jang et al. (2024)’s bound.Source: §2 · p. 3
  • Afzal et al. (2025) analyze fine-tuning through linearization with an explicit ℓ2 bias, whereas this paper’s reduction is exact once upstream computation is frozen.Afzal et al. also quantify when nonlinear fine-tuning stays close to the linearized problem.Source: §2 · p. 3

Optimization theory beyond linearization

  • Kim et al. (2025), Xu et al. (2025), Liu et al. (2025), and Zhang and Pilanci (2024) study LoRA dynamics, gradient flow, landscape comparisons, or preconditioning; this paper instead asks when frozen objectives admit low-rank optima and when factored stationary points must be rank-deficient.These works address how training behaves, while the paper frames its question before analyzing dynamics.Source: §2 · p. 3

Attention and Transformer expressivity

3 Problem setup

The setup freezes all but selected linear maps and studies trace-norm regularization through the information the frozen network retains about each update. It also formalizes stagewise dependence for adapted attention modules.

Single adapted block

  • For a single adapted block, the frozen input activation is independent of the update when other parameters are fixed, so the sample output depends on the update only through its product with that activation.Stacking the frozen activations across samples gives Φ(T) = TH.Source: §3 · p. 3
  • The bottleneck rank is the rank of the linear map through which the loss observes an update; for Φ(T) = TH, it is m rank(H).Updates with the same TH produce identical training predictions.Source: §3 · p. 3

Trace-norm regularization and the LoRA factorization

  • The setup uses a nuclear-norm-regularized empirical objective and a rank-r LoRA factorization with weight decay.Balanced weight decay makes the factorized problem share the trace-norm problem’s global minimum when restricted to rank at most r.Source: §3 · p. 4
  • When the unrestricted trace-norm problem has a global minimizer of rank at most r, the rank-r LoRA factorization incurs no loss of optimality.This follows from the stated equivalence between nuclear-norm regularization and balanced weight decay.Source: §3 · p. 4

Multiple adapted layers

  • For multiple adapted attention stages, each block enters through a stage-specific linear map, while transition maps may couple blocks arbitrarily, including nonconvex coupling.The frozen token matrix for an attention projection can depend on updates at more-upstream stages.Source: §3 · p. 4

Standing assumptions

  • The standing assumptions require a bounded-below, lower-semicontinuous objective with continuous maps and a deterministic per-sample forward pass without Dropout or BatchNorm.The paper identifies these as the standard evaluation-mode condition.Source: §3 · p. 5

4 Main results

Freezing the backbone restricts each adapted block to a finite-rank observation map, and trace-norm regularization yields global minimizers whose ranks are controlled by that bottleneck. The results extend to multiple stages and attention, characterize rank-deficient stationary points under generic perturbations, and relate perturbed minimizers to the original objective.

4.1 Low-rank minimizers for a single frozen block

  • For a frozen block, the trace-norm-regularized objective has a global minimizer whose rank is controlled by the observation map’s bottleneck rank; the result extends to multiple adapted stages.For a single block with Φ(T) = TH, the bottleneck rank is at most m rank(H) ≤ mN.Source: §4.1 · p. 5

4.2 Rank bounds for multi-head attention

  • Attention-specific score-centering and value-projection compressions yield tighter per-head, per-projection rank certificates for query, key, and value updates.Row-wise softmax removes a key-column direction; value updates are observed only after attention-weighted pooling and frozen output projection.Source: §4.2 · p. 6

4.3 Generic rank deficiency of first-order stationary points

  • For the perturbed factorized LoRA objective, generic PSD perturbations exclude full-rank first-order stationary points when the rank threshold exceeds the dimension of the associated symmetric-matrix family.The result is a first-order exclusion theorem, not a guarantee that stationary points are globally optimal.Source: §4.3 · p. 8
  • Spurious local minima can nevertheless persist: for any rank r ≥ 2, some smooth nonconvex losses retain a strict nonglobal local minimum under arbitrarily small PSD perturbations.Rank deficiency at stationary points does not imply global optimality.Source: §4.3 · p. 8

4.4 Perturbation transfer and generalization

  • Global minimizers of the perturbed factorized objective are near-optimal for the original trace-norm problem when the perturbation is small.Uniform generalization bounds for trace-norm-bounded updates in frozen networks are governed by the spectral norm of random-sign-weighted frozen features.Source: §4.4 · pp. 8–9

5 Experiment

Across the tested GLUE tasks, ranks at or above the estimated threshold converge to nearly indistinguishable losses close to full fine-tuning, while lower ranks follow worse trajectories. The experiment tests this threshold with selected query-projection blocks in Qwen2.5-1.5B.

5 Experiment

  • Ranks at or above the estimate nearly overlap and approach full fine-tuning loss, while lower ranks follow distinct, worse trajectories.High-rank LoRA runs reach almost the same point as full fine-tuning, though their training dynamics differ.Source: Figure 1 · p. 9, §5 · p. 10
  • The threshold test uses SST-2, MNLI, and RTE with one query-projection block in a head of layers 7 and 8 adapted.After classification-head warm-up, both the backbone and classifier are frozen.Source: §5 · p. 9
  • The estimated threshold is the smallest rank at which both upstream and downstream rank proxies saturate.The study evaluates many-example and few-example regimes; for Qwen2.5-1.5B, the head cap is 128.Source: §5 · pp. 9–10

6 Conclusion

The paper attributes low-rank global minimizers to finite-rank observations of adapted weights, with bottleneck dimensions controlling rank without NTK linearization. It also derives tighter attention rank bounds, studies perturbed factorized LoRA, and gives generalization results for frozen networks.

6 Conclusion

  • Finite-rank observations of adapted weights let trace-norm regularization select global low-rank minimizers, with rank controlled by bottleneck dimensions rather than ambient parameter count.The conclusion emphasizes that this result does not require NTK linearization.Source: §6 · p. 10
  • Attention invariances shrink observable subspaces and yield per-head rank certificates for query, key, and value updates at rank-minimal global minimizers.Row-wise softmax removes one key-column direction per sample; value updates are observed only after attention-weighted pooling and frozen output projection.Source: §6 · p. 10
  • Small perturbations exclude full-rank stationary points in nonconvex LoRA while preserving near-optimality for the original trace-norm problem.The result concerns first-order stationary points, not the absence of spurious local minima.Source: §6 · p. 10
  • The rank theorems establish existence, not convergence of gradient-based LoRA; attention bounds are a posteriori and the stationary-point result permits spurious local minima.The authors also state that they cannot explicitly calculate the Theorem 4.4 bounds, which depend on global solutions.Source: §6 · p. 10

A Full list of notations and auxiliary definitions

B Generalization bounds

  • For one adapted block, empirical loss depends on each update only through its bottleneck outputs.The appendix presents uniform generalization bounds for trace-norm-bounded updates in frozen networks.Source: B Generalization bounds · p. 16

Define the (conditional) empirical Rademacher complexity

The generalization analysis reduces bounds for trace-norm-bounded updates in frozen networks to the spectral structure of signed frozen features. For consecutive attention modules, it extends this analysis stage by stage, accounting for upstream effects on downstream inputs.

Define the (conditional) empirical Rademacher complexity

  • Uniform generalization over trace-norm-bounded updates is reduced to the spectral norm of a random signed feature matrix EH^T.Theorem B.5 states this bound under its listed assumptions.Source: Define the (conditional) empirical… · p. 18
  • The effective capacity is governed by the spectral geometry of frozen features, not the ambient parameter dimension mn.The trace norm and operator norm are dual in the complexity argument.Source: Define the (conditional) empirical… · p. 19
  • The multi-stage attention bound uses random signed matrices stage by stage and requires controlling how upstream output changes affect downstream adapted blocks.The result assumes the stated regularity conditions for consecutive attention modules.Source: Define the (conditional) empirical… · pp. 20, 22
  • The multi-stage statement also applies to multi-head attention, including head-indexed Q, K, and V blocks or shared blocks in grouped-query and multi-query attention.The theorem’s stagewise bound is the attention analogue of the single-block result.Source: Define the (conditional) empirical… · p. 22

C Proofs for Main results

The proof strategy combines projection invariance with a positive-semidefinite lift to apply a Barvinok–Pataki rank bound.

C Proofs for Main results

  • If projecting an update onto fixed left and right subspaces leaves the loss unchanged, nuclear-norm regularization does not prefer the unprojected update.Source: C Proofs for Main results · p. 23
  • The nuclear norm is rewritten through a positive-semidefinite lift as a second reduction in the proof strategy.Source: C Proofs for Main results · p. 23
  • The lift makes it possible to apply the Barvinok–Pataki rank bound.The passage cites Pataki (1998) and Barvinok (2001).Source: C Proofs for Main results · p. 23

C.1 Proofs for Section 4.1

The proofs establish existence of low-rank global minimizers by compressing updates without changing their observed outputs or increasing their nuclear norms. The argument extends from one adapted block to multiple stages by processing blocks conditionally.

C.1 Proofs for Section 4.1

  • Projector compression preserves the observed map while not increasing the nuclear norm, so a compressed update remains globally optimal.The proof uses equality of the loss term and global optimality of the original minimizer to establish optimality after compression.Source: §C.1 · p. 23
  • A lifted positive-semidefinite formulation reduces rank control to an extreme-point bound with dΦ independent affine constraints.The proof applies the Barvinok–Pataki rank bound to the spectrahedron of matrices producing the same observed output.Source: §C.1 · pp. 23–24
  • Coercivity of the nuclear-norm-regularized objective ensures that a global minimum is attained under the stated lower-semicontinuity and bounded-below conditions.The same existence argument is applied to the multi-stage objective when its regularization coefficients are positive.Source: §C.1 · pp. 24–25
  • The multi-stage proof replaces each block by a conditional minimizer in upstream-to-downstream order, preserving stage states and the global objective value.Already processed blocks remain valid because the replacement leaves the relevant stage and downstream states unchanged.Source: §C.1 · p. 25

C.2 Proofs for Section 4.2

The attention proofs tighten blockwise rank bounds by identifying which parts of query, key, and value updates remain observable. They extend the compression argument to adapted output projections, while treating that extension separately from a joint Q/K/V/O observation map.

C.2 Proofs for Section 4.2

  • Row-wise softmax is unchanged by adding row constants to scores, so query and key updates can be compressed while preserving attention matrices.For values, observability is determined after attention weighting and the frozen output projection.Source: §C.2 · pp. 25–26
  • For the query, key, and value blocks, conditional loss depends only on their corresponding observed products after the other blocks are fixed.The proof identifies these products blockwise and uses them to verify the projector-compression conditions.Source: §C.2 · pp. 27–28
  • A rank-minimal global minimizer satisfies blockwise rank bounds from the ranks of the corresponding compression projectors.The proof first establishes a rank-minimal minimizer and then bounds the query, key, and value projector ranks.Source: §C.2 · pp. 28–29
  • The adapted output projection can be compressed using the inputs formed after attention weighting, preserving the loss and not increasing its nuclear norm.This extension uses blockwise compression, not a simultaneous linear observation map for the full Q/K/V/O tuple.Source: §C.2 · pp. 29–30

C.3 Proofs for Section 4.3

The generic rank-deficiency proof combines a symmetric KKT condition with a dimension bound on the gradient matrices reachable through the observation maps. A measure-zero argument then excludes full-column-rank stationary factors for almost every PSD perturbation under the stated conditions.

C.3 Proofs for Section 4.3

  • Stationarity gives a symmetric block KKT equation, while each gradient block lies in the adjoint range of its stage’s observation map.These facts connect the stationary-point equations to the low-dimensional family used in the genericity argument.Source: §C.3 · pp. 31–32
  • The measure-zero argument excludes full-rank factors when the dimensions of the nullspace strata and gradient family fall below the ambient symmetric-matrix dimension.It yields a Lebesgue-null exceptional set, and hence the result for almost every PSD perturbation.Source: §C.3 · pp. 31–32
  • Reachable states and moving adjoint ranges admit finite C1 parameterizations with dimensions bounded recursively by β_t and β_(t+1) + d_max.The parameterization bounds the family of gradient matrices over all reachable upstream states.Source: §C.3 · pp. 32–34
  • For almost every PSD perturbation, every first-order stationary point has rank-deficient factors when the theorem’s dimension threshold is satisfied.The stagewise result applies under differentiability, blockwise linear-map dependence, and Assumption 4.Source: §C.3 · p. 34
  • The theorem excludes full-column-rank first-order stationary points, but does not establish the absence of spurious local minima.A smooth nonconvex loss can have a strict nonglobal local minimum at R = 0 even under small PSD perturbations.Source: §C.3 · p. 35

C.4 Proofs for Section 4.4

The proof transfers global solutions of the perturbed factorized objective to near-optimal solutions of the original trace-norm problem. It also relates the factorized problem to the trace-norm objective restricted to rank at most r.

C.4 Proofs for Section 4.4

  • Global optimality of the perturbed factorized solution yields the stated inequality comparing its trace-norm objective value with that of an original-problem minimizer.The proof combines the comparison with a bound on the summed perturbation penalties.Source: §C.4 · p. 36

D Experiment details

The experiment details describe proxy-based estimation of activation rank alongside LoRA and full-fine-tuning hyperparameters, with training-sample counts specified for two bound cases.

D Experiment details

  • The proxy rank is 128 for Qwen2.5-1.5B, matching its query-head dimension d_k.The proxy is chosen large enough to exactly represent the selected head block.Source: D Experiment details · p. 36
  • The proxy uses λ = 10^-4, AdamW at learning rate 5 × 10^-4, and lowest-training-loss checkpoint selection.The proxy objective matches the regularized objective used for the LoRA sweep.Source: D Experiment details · p. 36
  • The downstream activation rank is estimated from HH⊤ by counting eigenvalues above 10^-4.The LoRA update is merged into the base weights before evaluation on the training split.Source: D Experiment details · p. 36
  • Table 4 lists hyperparameters for LoRA and full fine-tuning on NLP tasks.The supplied caption does not specify the individual settings.Source: Table 4 · p. 37
  • The classification head is warmed up for 10 epochs and frozen; the high-rank and low-rank bound cases use 100 and 4 training samples, respectively.A proxy-based estimate approximates the optimizer-dependent quantity in Theorem 4.4; the final predicted threshold is based on estimated upstream and downstream observable ranks.Source: D Experiment details · pp. 36–37

NeurIPS Paper Checklist

The checklist records the paper’s theoretical and experimental disclosures, including limitations in reproducibility, statistical reporting, and compute-resource information.

NeurIPS Paper Checklist

  • The authors report complete theoretical assumptions and proofs in Sections 3 and 4 and the appendix.The checklist also records that the main claims are supported by technical proof and experimental evidence.Source: NeurIPS Paper Checklist · pp. 38–39
  • The experiments use GLUE tasks and an open-source Qwen2.5 checkpoint, but the draft does not include released code.The authors direct readers to experiment descriptions in the paper and appendix.Source: NeurIPS Paper Checklist · pp. 39–40
  • The current figures show single optimization trajectories without error bars, and the checklist answers no to statistical-significance reporting.The checklist guidance identifies error bars, confidence intervals, or significance tests as suitable reporting options.Source: NeurIPS Paper Checklist · p. 41
  • The draft does not report GPU type, memory, runtime per run, or total compute.The checklist marks the compute-resource disclosure question no.Source: NeurIPS Paper Checklist · p. 41
  • The paper reports following the NeurIPS Code of Ethics and says it discusses positive and negative societal impacts.It identifies possible positive and negative impacts from advances in parameter-efficient fine-tuning methods.Source: NeurIPS Paper Checklist · p. 42
  • The checklist says the paper uses no crowdsourcing or human subjects, releases no new assets, and treats safeguards for high-risk releases as not applicable.The authors say existing experimental assets are properly credited and the work poses no such release risks.Source: NeurIPS Paper Checklist · pp. 42–44
Loading 2610.23343v1…