Source-linked AI summary
Automatic Rank Allocation for Low-Rank Adaptation in Large Language Models via lp Regularization
Zebang Xie, Chuanyang Zheng, Yik-Chung Wu, Yihang Gao
TL;DR
LoRA reduces fine-tuning parameters, but allocating ranks across adaptation matrices remains a resource-allocation problem because components can need different capacities. ℓp-LoRA penalizes the energy of rank-one components and uses an implicit thresholding rule to identify redundant ones. On natural language understanding and question-answering tasks, it is competitive with fixed-rank and adaptive-rank methods under the same rank budget.
Problem
LoRA rank determines adaptation capacity and efficiency, yet different matrices may need different ranks and existing adaptive methods typically rely on designed importance scores.
Source: §1 · p. 1Method
ℓp-LoRA penalizes each rank-one component's energy and reduces its optimization to a two-dimensional problem whose implicit thresholding decision identifies components for pruning.
Source: §1 · p. 1, §3.2 · p. 3Results
ℓp-LoRA achieves the best performance on two of four natural language understanding tasks and two of four question-answering tasks, while remaining competitive on the others.
Source: §4.1 · p. 4, §4.2 · p. 4Takeaways & Limitations
Under the same rank budget, ℓp-based rank allocation provides a competitive alternative to fixed-rank and adaptive-rank LoRA methods.
Source: §5 · p. 4The training-loss form depends on the model, downstream task, and training data.
Source: §3.1 · p. 2
Abstract
from arXiv · showhide
Low-rank adaptation (LoRA) has become a popular parameter-efficient fine-tuning method for large language models. A key challenge in LoRA is how to determine the rank of each adaptation matrix, as rank directly controls its capacity and efficiency. Existing adaptive-rank methods typically allocate ranks according to manually designed importance scores, which are not directly derived from an optimization objective. In this work, we propose $\ell_p$-LoRA, a principled rank-allocation method based on $\ell_p$ regularization with $0<p<1$, which is a classical sparsity-inducing technique in signal processing and statistics. Specifically, we regularize the energy of each rank-one LoRA component, encouraging redundant components to vanish while preserving important ones. We derive the corresponding proximal subproblem and reduce the matrix optimization to a two-dimensional problem, leading to an implicit thresholding criterion for identifying redundant components. Experiments on natural language understanding and question-answering tasks demonstrate that the proposed method achieves competitive performance with existing LoRA baselines.
1. INTRODUCTION
LoRA makes fine-tuning more parameter-efficient, but assigning ranks across adaptation matrices remains an important resource-allocation question. This section motivates an ℓp-based approach that treats rank-one components as candidates for retention or removal.
1. INTRODUCTION
- LoRA freezes pretrained weights and represents each trainable update with a low-rank factorization, reducing trainable parameters.It is presented as a widely adopted PEFT method for adapting pretrained models to downstream tasks.Source: §1 · p. 1
- Uniformly assigning ranks may waste a limited rank budget because different model components can require different adaptation capacities.The budget is determined by computational or memory constraints.Source: §1 · p. 1
- Prior adaptive-rank methods mainly allocate ranks using designed importance scores, while Bayesian approaches add modeling and inference mechanisms beyond standard LoRA training.AdaLoRA uses scores built from training information, and later methods incorporate gradient or layer-wise information; probabilistic approaches offer another perspective [8].Source: §1 · p. 1
- ℓp-LoRA penalizes the energy of each rank-one component, encouraging redundant components to vanish while retaining informative ones.The approach frames rank selection as structured sparsity using ℓp regularization with 0 < p < 1.Source: §1 · p. 1
2. BACKGROUND
The background describes LoRA as a factorized update to pretrained parameter matrices and motivates allocating rank according to component relevance. It also introduces ℓp regularization as a sparsity-promoting penalty with a thresholding behavior that maps sufficiently small values to zero.
2.1. Low-Rank Adaptation
- LoRA represents each fine-tuned parameter matrix as a pretrained matrix plus a low-rank adaptation, with trainable factors A and B.For a matrix W, A ∈ R^(r×n) and B ∈ R^(m×r), where r ≪ min{m,n}; the formulation extends across LoRA-equipped matrices.Source: §2.1 · pp. 1–2
- Different layers may need different effective ranks, since uniform allocation can waste budget and unnecessarily large ranks add trainable parameters and redundant degrees of freedom.Some matrices need only a few rank-one components, while others may require greater adaptation capacity.Source: §2.1 · p. 2
2.2. ℓp Regularization
- For 0 < p < 1, the nonconvex ℓp penalty more closely approximates ℓ0 and generally induces stronger sparsity than the convex ℓ1 surrogate.The objective combines a data-fitting function f with a regularization parameter λ > 0.Source: §2.2 · p. 2
- The ℓp proximal subproblem reduces coordinate-wise to scalar minimizations that set sufficiently small entries to zero and retain sufficiently large entries.This zero–nonzero transition provides a thresholding mechanism for variable selection and sparse recovery.Source: §2.2 · p. 2
3. METHOD
Rank allocation is formulated as structured sparsity over rank-one LoRA components, with an ℓp penalty on component energy. A tractable per-component optimization yields an implicit prune-or-retain rule that is periodically applied during standard LoRA training.
3.1. Motivation and Problem Formulation
- The ℓp penalty acts on rank-one component energies, whose nonzero count provides an effective-rank measure for rank-budget allocation.The penalty replaces a difficult cardinality constraint with a sparsity-promoting surrogate.Source: §3.1 · p. 2
3.2. Algorithm Design: Implicit Thresholding
- A local quadratic model of the training loss around current LoRA factors makes the regularized update tractable.The model uses gradients with respect to both factors evaluated at the current iterate.Source: §3.2 · p. 3
- Separability across rank-one components and alignment with nominal vectors reduce each vector subproblem to a two-dimensional optimization.The resulting two-variable problem supplies the component-level thresholding decision.Source: §3.2 · p. 3
- A component is pruned when the two-dimensional solution has α_iβ_i = 0 and retained otherwise.The solution is selected from nonzero stationary points and boundary candidates.Source: §3.2 · p. 3
3.3. LoRA Rank Allocation Based on Thresholding
- ℓp-LoRA keeps the standard fine-tuning optimizer and periodically invokes the thresholding criterion to identify redundant components.At pruning steps, the algorithm solves the component problem and prunes components whose solution has α_i = 0.Source: §3.3 · p. 4
4. EXPERIMENT
Across NLU and question-answering benchmarks, ℓp-LoRA performs best on two of four tasks in each setting and remains competitive on the others. The experiments use half of the initial rank as the final budget.
4.1. Natural Language Understanding
- On GLUE with DeBERTaV3-base, the four tasks use Matthews correlation, accuracy, and Pearson correlation, with results averaged over five seeds.The comparison uses LoRA, AdaLoRA, IGU-LoRA, and ℓp-LoRA under the same schedule and a final rank budget half the initial rank.Source: §4.1 · p. 4, Table 1 · p. 4
4. EXPERIMENT
- ℓp-LoRA is best on two of four GLUE tasks and competitive on the rest, and is best on two of four QA tasks while competitive on the others.For BoolQ, the improvement is more pronounced; on CommonsenseQA, performance is comparable to the strongest baseline.Source: §4.1 · p. 4, Table 1 · p. 4, §4.2 · p. 4
4.2. Question Answering
- On BoolQ, ARC-Easy, OpenBookQA, and CommonsenseQA, Qwen2.5-7B accuracy is averaged over five seeds under a final rank budget half the initial rank.ℓp-LoRA's BoolQ improvement is more pronounced, while its CommonsenseQA result is comparable to the strongest baseline.Source: §4.2 · p. 4, Table 2 · p. 4
5. CONCLUSION
The conclusion frames LoRA rank allocation as structured sparsity, using an ℓp penalty on rank-one component energy. A component-wise subproblem and two-dimensional scalar reduction yield an implicit pruning threshold, with competitive performance under the same rank budget.
5. CONCLUSION
- An ℓp penalty on rank-one LoRA component energy formulates rank allocation as structured sparsity and yields a component-wise subproblem reduced to a two-dimensional scalar problem.The resulting implicit threshold prunes redundant components while preserving standard LoRA training for the remaining components; experiments show competitive performance against fixed- and adaptive-rank methods under the same budget.Source: §5 · p. 4