The Expressive Power of Low-Rank Adaptation

1 INTRODUCTION

• LoRA uses lightweight adapters on pretrained weights and can match or exceed full fine-tuning while reducing the costs of updating all parameters.It has been used for downstream tasks with language and image-generation models.Source: §1 · p. 1
• Despite LoRA's empirical success, its minimum required rank, architectural effects, and approximation error below the rank threshold remain open theoretical questions.Prior analysis established approximate equivalence to full fine-tuning only in the lazy regime (Malladi et al., 2023).Source: §1 · p. 1
• The paper characterizes LoRA's expressive power in fully connected and Transformer networks, deriving rank requirements for exact matching and approximation bounds for fully connected networks.Its scope is whether effective adapters exist, not optimization or generalization.Source: §1 · pp. 1–2

1.1 RELATED WORKS

• Prior work on network expressivity generally does not account for pretrained parameters and adaptation methods; earlier adaptation analyses considered normalization tuning and neural reprogramming, but not LoRA.The cited examples include normalization fine-tuning (Giannou et al., 2023a) and neural reprogramming (Englert & Lazic, 2022).Source: §1.1 · p. 2

1.2 NOTATIONS

• The notation defines [N] as the integers from 1 to N and uses ∧ and ∨ for minimum and maximum, respectively.The identity matrix is denoted I; the text also defines singular values and the best rank-r approximation LRr(W).Source: §1.2 · pp. 2–3

2 WARM UP: EXPRESSIVE POWER OF LINEAR MODELS WITH LORA

• The warm-up models the frozen network as a product of linear layers and represents each adapted layer by adding an update of rank at most R.The target is a single linear layer; the frozen model has L layers, with zero biases and linear activations.Source: §2 · p. 3
• With non-singularity conditions, the minimum achievable discrepancy identifies when LoRA updates can exactly represent the target linear model.The setup uses a one-layer linear target and an L-layer frozen model with equal-width layers and zero biases; the paper says it extends to wider frozen models.Source: §2 · p. 3
• For a product of L matrices, each with a rank-R update, the effective rank is LR, and configured updates can make the product the target matrix's best rank-LR approximation.This extends matrix approximation to a product whose individual factors are each subject to low-rank constraints.Source: §2 · p. 4

3.1 PROBLEM SETTING

• The analysis considers width-D ReLU FNNs and focuses on a deeper frozen model with bounded inputs; the results extend to a wider frozen model.The section assumes the frozen model has at least as many layers as the target.Source: §3.1 · p. 4

3.2 ONE-LAYER RELU FNN APPROXIMATION

• For a one-layer ReLU target, the construction linearizes the first L − 1 adapted layers with large biases, then aligns the final bias and approximates the target weights.This reduces the task to a linear-model approximation result.Source: §3.2 · p. 5

3.3 MULTI-LAYER RELU FNN APPROXIMATION

• Under Assumption 1, a frozen FNN can be adapted to exactly match the target FNN on the input domain using sufficiently ranked LoRA adapters.The assumption is a non-singularity condition on weight matrices.Source: §3.3 · p. 6
• With randomly drawn weights, rank R ≥ D/M suffices with probability 1 for exact target representation; the stated parameter count is at most twice the target model’s.Here M is the number of frozen-model layers per target layer under uniform partition.Source: §3.3 · p. 6
• When rank is below the threshold, the approximation-error bound depends on model and input magnitude, adapter rank and weight discrepancy, and frozen-model depth.The rank and discrepancy contribute through the layerwise error terms E_i.Source: §3.3 · pp. 6–7
• A general layer partition allows different adapter ranks and provides conditions for exact target representation and an approximation-error bound.The authors note that partitions tailored to a scenario can use a lower rank than uniform partitioning.Source: §3.3 · p. 7
• For randomly generated frozen networks with D ≥ 2, tuning only the final L − 1 layers cannot exactly match even a one-layer target, whereas LoRA can do so with the same or fewer parameters when L ≥ 3.The comparison is stated for the paper’s theoretical setting.Source: §3.3 · pp. 7–8

4.1 PROBLEM SETTING

• The analysis considers frozen and target transformers with the same number of blocks and excludes skip connections for analytical feasibility.It assumes D × D weight matrices and D-dimensional bias vectors.Source: §4.1 · p. 8

4.2 MAIN RESULTS ON TRANSFORMER NETWORKS

• The exact-matching result relies on a non-singularity assumption that the authors describe as mild and satisfiable by randomly generated weights.The full theorem also conditions its rank requirement on each block’s or output layer’s rank-based functionality gap.Source: §4.2 · p. 8
• Attention-layer adapters can exactly match the target under the theorem’s rank and architecture conditions.The theorem allows other low-rank adapters to be zero and updates bias vectors; its construction aligns attention scores before softmax and first feedforward outputs before ReLU.Source: §4.2 · p. 8
• For single-head transformers, the adapter construction differs because the multi-head output projection W^h_O is absent, so those results are deferred to Sec. F.1.The main results here concern multi-head attention transformers.Source: §4.2 · p. 8

• MSE is close to zero for both linear-model and FNN approximation when R ≥ D L/L = 8.The paper depicts these comparisons in Figure 1a and 1b.Source: §5 · p. 9, Figure 1 · p. 9
• Pretraining further reduces the rank needed for near-zero approximation error in the linear-model comparison.In linear models, the constructed adapter’s error curves align with those from gradient updates, supporting the optimality claimed in Lemma 1.Source: §5 · p. 9
• In the small-rank FNN region, gradient updates outperform the construction, which the authors conjecture may reflect intermediate-output matching or the uniform partition.The evaluation uses MSE and considers dimension D = 16, with linear-model depths 1 and 2 and FNN depths 2 and 4; frozen models are randomly initialized or pretrained on the target distribution.Source: §5 · p. 9
• The appendix reports additional simulations, classification and GLUE evaluations, comparisons with final-layer tuning, and studies of bias updates, generalization, and optimization.Table 1 summarizes the empirical findings alongside theoretical results.Source: §5 · p. 9, Table 1 · p. 9

6 CONCLUSIONS

• The paper’s theory and empirical evidence examine how LoRA’s effectiveness relates to rank, model depth, and proximity to the target model.Future work includes quantifying transformer approximation error below the required ranks and refining adapter-update algorithms.Source: §6 · p. 9

Appendix

• The appendix includes additional proofs, experiments, and results for single-head transformers and randomly generated multi-head transformers.It also discusses extending results to varying model dimensions, while the main work focuses on frozen and target models with the same width D.Source: Appendix · p. 15

B EXPANDED RELATED WORKS

• The related-work discussion distinguishes this study from prior adaptation analyses by focusing on LoRA’s expressive power, which it says had not yet been explored.It also contrasts its setting with a prior LoRA analysis based on linear models.Source: B EXPANDED RELATED WORKS · pp. 18–19

C.1 COMMON MATRIX INEQUALITIES

• The appendix gathers matrix-multiplication, norm, trace, and Frobenius-norm properties for reference.One listed identity is tr(AB) = tr(BA).Source: §C.1 · p. 19

C.2 NON-SINGULARITY OF RANDOMLY GENERATED MATRICES

• A nonzero polynomial’s zero set has Lebesgue measure zero, providing a key step in the non-singularity argument.The proof views a matrix determinant as a polynomial of its vectorized entries.Source: §C.2 · p. 20
• A random matrix with an arbitrary continuous distribution whose support has nonzero Lebesgue measure is non-singular with probability 1.The appendix also concludes that a randomly generated matrix X is non-singular with probability 1.Source: §C.2 · p. 20

D PROOFS FOR LINEAR MODEL APPROXIMATION

• The deep linear model is used as a preliminary setting for extending the analysis to nonlinear FNNs and Transformers.The section presents results and proofs for the linear approximation problem introduced earlier.Source: D PROOFS FOR LINEAR MODEL APPROXIMATION · p. 20, E PROOFS FOR FNN APPROXIMATION · p. 25
• The proof turns the adapted-product discrepancy into a low-rank matrix approximation problem, bounding its rank by the sum of the adapter ranks.For a common rank R across L layers, the bound is LR.Source: D PROOFS FOR LINEAR MODEL APPROXIMATION · pp. 21, 24
• The proof extends to adapters with different layerwise ranks, with the low-rank approximation constrained by their total rank.The construction follows the equal-rank proof with a modified rank constraint.Source: D PROOFS FOR LINEAR MODEL APPROXIMATION · p. 24
• In the linear case, distributing rank R across layers or concentrating total rank RL in the final layer can achieve exact approximation with the same parameter efficiency.The passage also compares this with assigning rank D to the last layer and rank D/L per adapter.Source: D PROOFS FOR LINEAR MODEL APPROXIMATION · p. 25

E.1 APPROXIMATING ONE-LAYER RELU FNN VIA LORA

• For inputs drawn from a distribution supported on X, the section bounds the expected squared approximation error using the input covariance Σ.The supplied result states the bound’s setting but not its full expression.Source: §E.1 · pp. 25, 27
• The proof activates all ReLUs in the first L−1 adapted layers using sufficiently large biases, reducing the network to a single-layer FNN.This relies on the input space X being bounded.Source: §E.1 · pp. 25–26
• When the total adapter rank satisfies sum_l R_l ≥ RE, the adapted one-layer ReLU model exactly matches the target on X.With a common rank R, exact matching holds when R ≥ ceil(RE/L).Source: §E.1 · pp. 25, 27

E.2 APPROXIMATING MULTI-LAYER RELU FNN VIA LORA WITH UNIFORM MODEL PARITION

• The non-singularity assumption holds with probability 1 when frozen weights are independently drawn from arbitrary continuous distributions.This supports the uniform-partition results’ stated conditions.Source: §E.2 · p. 27
• Under Assumption 1, Theorem 3 gives exact approximation when the adapter rank covers the largest layerwise discrepancy rank divided by the uniform group size.The proof assigns M consecutive frozen layers to approximate each target layer.Source: §E.2 · p. 28
• When frozen weights are independently drawn from arbitrary continuous distributions, rank R ≥ D/M suffices for functional coverage of the target on X with probability 1.Here M is the number of frozen layers grouped to approximate each target layer.Source: §E.2 · p. 28
• Theorem 5 defines each layer’s residual error through the singular value σ_(RM+1) of its weight discrepancy and bounds the overall expected approximation error.The bound is stated for inputs with second-moment matrix Σ.Source: §E.2 · p. 29

E.3 APPROXIMATING MULTI-LAYER RELU FNN VIA LORA WITH GENERAL MODEL PARITION

• For independently drawn frozen weights from arbitrary continuous distributions, Assumption 2 holds with probability 1.Assumption 2 is specified for the given adapter-rank sequence and partition.Source: §E.3 · p. 31
• Theorem 6 gives exact target approximation under Assumption 2 for a general partition of the frozen model.The proof replaces the uniform partition with the general partition and uses the corresponding approximation lemma.Source: §E.3 · p. 32
• For the general partition, the section also states an expected approximation-error bound for inputs with second-moment matrix Σ.Its layerwise error measure is defined using singular values of the discrepancy between target and grouped frozen weights.Source: §E.3 · p. 32

E.4 APPROXIMATING MULTI-LAYER RELU FNN VIA FINAL LAYERS TUNING

• Final-layer tuning cannot exactly match a target when frozen initial layers map two inputs identically but the target maps them differently.Tuning later layers cannot undo that equality in the frozen model’s initial-layer outputs.Source: §E.4 · p. 32
• Non-activation regions collapse frozen-model inputs to identical outputs despite later tuning, while target-model outputs can differ for those inputs.The target activates all ReLUs for the selected inputs, whereas the frozen model’s initial layers produce equal outputs.Source: §E.4 · pp. 32–33
• When the stated activation-region condition holds and target weight matrices are nonsingular, tuning the last L − l layers cannot make the models equal.The result applies to any tuning of those final layers.Source: §E.4 · p. 32
• For D ≥ 2, a one-layer target FNN with independently drawn continuous weight entries differs with probability 1 after any tuning of the last L − 1 layers.The probability-one claim holds for arbitrary continuous distributions of the weight-matrix entries.Source: §E.4 · p. 33

F PROOFS FOR TFN APPROXIMATION

• Section F introduces the problem setting for TFNs with single-head attention layers and presents the corresponding results.Source: F PROOFS FOR TFN APPROXIMATION · p. 33

F.1 APPROXIMATING TRANSFORMER NETWORK WITH SINGLE-HEAD ATTENTION LAYERS

• For single-head Transformers with continuously drawn target and frozen weights, rank R ≥ ⌈D/2⌉ adapters and tuned biases yield exact representation for every input.The adapters are added to specified attention and feedforward weights, the final-block second feedforward weight, and the output weight.Source: §F.1 · p. 36

F.2 APPROXIMATING TRANSFORMER NETWORK WITH MULTI-HEAD ATTENTION LAYERS

• Under Assumption 4, for a given LoRA rank R ∈ [D] satisfying R ≥ max_i∈[L+1]⌈G_i/2⌉, there exist low-rank adapters such that the adapted model exactly approximates the target for every input X ∈ R^{D×N}.Source: §F.2 · pp. 37–38
• The multi-head construction first matches each block’s first feedforward-layer output, then chooses the final-block and output-layer weights to match the target output.The proof inductively aligns intermediate outputs across blocks before completing the output-layer match.Source: §F.2 · p. 38

G EXPERIMENTS

• Experiments on synthetic and real datasets are intended to corroborate the theoretical results, including adapter construction, final-layer tuning, updatable bias, training curves, generalization, and classification.The passage describes experimental aims, not outcomes.Source: G EXPERIMENTS · pp. 39–40

G.1 ADDITIONAL DETAILS OF EXPERIMENT SETUP

• The experiments parameterize each LoRA adapter as ΔW = AB^⊤, with A and B in R^(D×R), using the initialization scheme of Hu et al. (2022a).This setup is stated for experiments in Sec. 5, G.3.1, G.3.2, G.4, and G.5.Source: §G.1 · p. 40

G.3.1 FNN APPROXIMATION

• For FNNs, pretrained frozen models yield less approximation error; construction aligns with gradient updates at target depth L = 1 but diverges at L = 2 for R ≤ 4.The construction assumes the frozen and target models’ intermediate outputs align, which limits its effectiveness in the low-rank region.Source: Figure 3 · p. 40, §G.3.1 · p. 41
• At sufficiently large rank, FNN LoRA construction reaches approximately 10^-8 MSE, versus approximately 10^-4 for gradient updates.The comparison is presented on a logarithmic scale for the higher-rank region, R ≥ 6.Source: §G.3.1 · p. 41, Figure 4 · p. 41

G.3.2 TFN APPROXIMATION

• For TFNs, gradient updates outperform the construction at small ranks but lag as rank increases, while their performance improves little with rank.The authors attribute the low-rank advantage to TFNs’ greater flexibility for constructing adapters, and the limited improvement to TFN optimization complexity.Source: §G.3.2 · p. 41, Figure 5 · p. 41

G.4 COMPARISON TO TUNING FINAL LAYERS

• Final-layer tuning has substantially higher MSE than LoRA for randomly generated models at equal tunable-parameter counts, and remains worse for pretrained models despite improving.The reported pretrained comparison is attributed to the dependence of final-layer tuning on shared-representation quality (Tripuraneni et al. (2020)).Source: §G.4 · p. 42, Figure 6 · p. 42

G.5 BENEFITS OF TUNING BIASES

• Tuning biases alongside LoRA improves performance over fixed biases at relatively small parameter counts, but the gap diminishes as parameter counts increase.This pattern holds for both random and pretrained models; without bias tuning, LoRA remains competitive.Source: §G.5 · p. 42, Figure 6 · p. 42

G.6 TRAINING CURVES

• Small-rank LoRA models converge quickly but settle at higher training loss, medium-rank models converge more slowly, and highly overparameterized models appear to converge faster.The reported examples are R = 1, 4 for small rank and R = 8 for medium rank; the text gives R = 13,18 for the highly overparameterized example.Source: §G.6 · p. 43, Figure 7 · p. 43

G.7 GENERALIZATION PERFORMANCES

• The training–test MSE gap shows no clear rank-related pattern in one comparison, but finetuning results show an increasing generalization gap with rank, especially for L = 2 and less so for L = 4.The authors suggest Adam may not precisely find the minimum in the first comparison, potentially avoiding overfitting; they relate the increasing gap to the larger hypothesis class at higher ranks.Source: §G.7 · p. 43

G.8 EVALUATION ON CLASSIFICATION TASKS

• The classification experiments use cross-entropy to optimize LoRA adapters and report accuracy on binary and multi-class tasks.Figure 9 plots accuracy against rank for FNN classification tasks.Source: §G.8 · p. 43, Figure 9 · p. 44

Multi-class Classification

• 100% accuracy is achieved by the construction when R ≥8 in multi-class classification.The experiments optimize LoRA adapters with cross-entropy and report accuracy.Source: §G.8 · p. 43, Figure 9 · p. 44

Binary Classification

• At R = 1, binary accuracy improves significantly and is comparable to higher ranks; the construction beats gradient updates at higher ranks.The setup adds a fixed block-diagonal 2×D output layer and optimizes cross-entropy on LoRA adapters.Source: §G.8 · pp. 43–44, Figure 9 · p. 44

G.9 EVALUATION ON REAL DATASETS

• RoBERTa-large (340M) with rank 2 outperforms RoBERTa-base (110M) with rank 6 on 7 of 8 GLUE tasks.The reported pattern supports the theoretical finding that larger models require lower LoRA ranks to achieve the desired performance.Source: §G.9 · p. 44, Table 2 · p. 45
• Pretrained RoBERTa-base significantly surpasses randomly initialized RoBERTa-base at the same LoRA rank, consistent with lower-rank sufficiency when the frozen model is closer to the target.The comparison is reported on GLUE.Source: §G.9 · p. 45, Table 3 · p. 45

H EXTENSION TO CASES WITH DIFFERENT MODEL DIMENSIONS

• For different model dimensions, the analysis expands the target matrix and bounds the resulting error-matrix rank by D.The discussion concerns linear-model and FNN approximation when the frozen model is wider than the target.Source: H EXTENSION TO CASES WITH DIFFERENT… · pp. 45–46

I EXTENDED FUTURE WORKS

• Open questions include more parameter-efficient adapter constructions, subthreshold Transformer approximation error, broader Transformer architectures, gradient-based optimization, and generalization.The Transformer analysis constrains frozen and target models to identical embedding size and depth and omits skip connections and layer norms.Source: I EXTENDED FUTURE WORKS · p. 46