Adaptive Utilization of Low-Rank Adaptation via Conditioned Gating

Guang Yang; Changhao Guan; Chao Huang; Yufeng Chen; Kaiyu Huang

Source-linked AI summary

Adaptive Utilization of Low-Rank Adaptation via Conditioned Gating

Guang Yang, Changhao Guan, Chao Huang, Yufeng Chen, Kaiyu Huang

arXiv:2610.05800v1Oct 5, 2026cs.CL

TL;DR

Shared low-rank updates in LoRA limit input-specific use of the adaptation subspace, while independently selected token updates may lack sequence-level consistency. U-LoRA coordinates token-wise utilization with sequence context and stabilizes it across training steps. Across diverse backbones and benchmarks, it reports consistent improvements over representative LoRA baselines and variants under comparable parameter budgets.

Problem
LoRA shares low-rank directions across inputs, which can cause interference across semantic contexts and limit context-sensitive adaptation.
Source: §1 · pp. 1–2
Method
U-LoRA learns token-wise utilization coefficients over low-rank directions, coordinates them with sequence-level aggregation, and calibrates signals with a bias-corrected historical prior.
Source: §1 · p. 2
Results
Across diverse backbones and benchmarks, U-LoRA shows consistent improvements over representative LoRA baselines and variants under comparable parameter budgets.
Source: §6 · p. 9
Takeaways & Limitations
The results identify utilization-aware modeling as an effective direction for parameter-efficient fine-tuning.
Source: §6 · p. 9
Token-wise adaptive weights cannot be statically merged into pretrained parameters, requiring extra inference computation and moderately increasing latency over standard LoRA.
Source: D. Limitations · p. 15

Abstract

from arXiv · show

Low-Rank Adaptation (LoRA) achieves parameter-efficient fine-tuning by constraining model updates to a low-rank subspace and has been widely used in practice. However, LoRA typically employs a shared low-rank update across tokens, which limits its ability to fully exploit the adaptation subspace for tokens from different sequences. To address this issue, we propose an adaptive utilization of Low-Rank Adaptation (U-LoRA), which employs conditioned gating to explicitly learn effective token-level utilization of the limited low-rank adaptation subspace. Specifically, U-LoRA generates utilization coefficients along low-rank directions for each token and jointly coordinates and constrains them using sequence-level contextual information, thereby inducing more consistent adaptive patterns within a sentence. To further enhance training stability, we introduce a bias-corrected exponential moving average (EMA) historical prior that calibrates utilization signals across optimization steps, suppressing noise caused by batch-to-batch fluctuations. The effectiveness of our method arises from a better utilization of the existing low-rank subspace via input-conditioned strategies, rather than from expanding the subspace. Experiments on mathematical reasoning and natural language understanding benchmarks demonstrate that U-LoRA achieves competitive performance under comparable parameter budgets when with strong LoRA baselines and recent variants.

1. Introduction

LoRA offers parameter-efficient adaptation, but its shared low-rank directions can interfere across semantic contexts, while independently chosen token-level updates may lack sequence-level alignment. U-LoRA coordinates token-wise utilization within sentences, stabilizes it across training steps, and reports gains over LoRA baselines and variants under comparable parameter budgets.

1. Introduction

LoRA freezes pretrained parameters and uses a small low-rank update, but shared directions can cause interference across semantic contexts.This limits fine-grained, context-sensitive behavior in a fixed subspace.Source: §1 · p. 1
U-LoRA assigns input-conditioned utilization coefficients to low-rank directions, selectively allocating limited adaptation capacity rather than uniformly sharing it.The coefficients reweight shared directions, including for different semantic instantiations of the same token.Source: §1 · p. 2
Under comparable parameter budgets, U-LoRA consistently outperforms representative LoRA baselines and recent variants across the evaluated benchmarks.The evaluation covers mathematical reasoning and GLUE natural language understanding benchmarks.Source: §1 · p. 2
Sequence-level aggregation coordinates token utilization to encourage consistent patterns within a sentence and preserve LoRA’s parameter efficiency.This addresses the dispersion of independently selected updates within a task or sentence.Source: §1 · p. 2
A lightweight historical prior calibrates utilization signals across optimization steps to improve fine-tuning stability.The paper motivates this prior by noting that frequent input-conditioned learning may cause utilization decisions to fluctuate.Source: §1 · p. 2

2. Related Work

Prior PEFT work has developed low-rank structure and optimization variants, as well as input-conditioned mechanisms for finer-grained adaptation. The cited work identifies stable, task-consistent utilization within a fixed low-rank subspace as an open question.

2. Related Work

Prior PEFT work improves LoRA through capacity-oriented changes to low-rank structures and optimization-oriented changes to learnability or stability.Examples include AdaLoRA and structured low-rank variants, alongside DoRA, PiSSA, and LoRA-GA.Source: §2 · p. 2
Figure 2 depicts U-LoRA adding token-wise input-conditioned gating and sequence context to modulate low-rank updates while a historical prior stabilizes signals and the backbone stays frozen.Token features are projected into a rank-aligned space before sequence-level coordination.Source: Figure 2 · p. 3
Input-conditioned LoRA methods select or combine components, and TopLoRA adds token-wise diagonal modulation, but stable task-consistent utilization within a fixed subspace remains open.The related work also describes projection-based methods that learn key input–output components under tighter parameter budgets.Source: §2 · pp. 2–3

3. Methodology

U-LoRA makes LoRA’s low-rank adaptation input-conditioned, coordinating token-level gates with sequence-level information and using a historical prior to stabilize utilization.

3.1. The Overview

U-LoRA integrates token-wise conditional gating with sequence-level aggregation to modulate low-rank components for each token.A bias-corrected EMA prior calibrates utilization signals across optimization steps.Source: §3.1 · p. 3

3.2. Parameterization with Input-conditioned Gating

U-LoRA replaces LoRA’s constant down-projection with an input-conditioned one, element-wise modulated to retain the same rank-r subspace.A and B remain shared low-rank bases, while G(x) determines their utilization; setting G(x) to 1 recovers vanilla LoRA.Source: §3.2 · p. 3

3.3. Token-wise Utilization and Sequence-level Aggregation

Token representations are projected into rank-aligned features, summarized across the sequence, and combined with token signals to produce token-specific gates.Decoder-only sequence aggregation uses a causal mask so each token depends only on its current prefix.Source: §3.3 · p. 4

3.4. Stabilizing Utilization with a Historical Prior

A bias-corrected EMA historical prior calibrates gates to smooth batch-to-batch noise, then remains frozen at inference without extra computation.The prior is added to each token’s intermediate gating representation.Source: §3.4 · p. 4

4. Experiments

Experiments evaluate U-LoRA on mathematical reasoning and GLUE natural-language-understanding tasks across decoder-only and encoder-only models. U-LoRA achieves the best average results in both settings against LoRA and recent variants under comparable parameter budgets.

4.1. Experimental Setup

Evaluation spans six mathematical-reasoning datasets across four decoder-only model families and GLUE's eight tasks on RoBERTa-Base and RoBERTa-Large, against LoRA and representative variants.The baselines include DoRA, MELoRA, HydraLoRA, and TopLoRA.Source: §4.1 · pp. 4–5

4.2. Results on Mathematical Reasoning Tasks

U-LoRA has the best average accuracy across four decoder-only LLMs, outperforming LoRA and recent variants at comparable parameter budgets.The comparison is reported for the mathematical-reasoning benchmarks.Source: Table 1 · p. 5, §4.2 · p. 5

4.3. Results on Natural Language Understanding Tasks

U-LoRA has the best average performance on both RoBERTa-Base and RoBERTa-Large, outperforming LoRA and recent variants at comparable parameter budgets.The comparison is reported on GLUE.Source: Table 2 · p. 6, §4.3 · p. 6

4.2. Results on Mathematical Reasoning Tasks

On Qwen3-8B, U-LoRA scores 43.31 on AQuA and 85.29 on GSM8K, versus 36.61 and 75.21 for LoRA with r=32.The passage describes these as gains on more challenging reasoning tasks.Source: §4.2 · p. 6

4.3. Results on Natural Language Understanding Tasks

60.61% vs 58.58% on RoBERTa-Base CoLA and 66.51% vs 60.90% on RoBERTa-Large CoLA, U-LoRA exceeds LoRA with r=32.The passage identifies CoLA as linguistic acceptability and notes larger gains on this task.Source: §4.3 · p. 6
90.62% vs 90.05% on RoBERTa-Base STS-B and 91.68% vs 91.54% on RoBERTa-Large STS-B (vs TopLoRA r=8); U-LoRA remains competitive on RTE.The passage presents STS-B as stable representation alignment and describes the RTE result in terms of smaller datasets.Source: §4.3 · p. 6

5. Discussion

The discussion examines U-LoRA’s component contributions, behavior across adaptation settings, update geometry, depth-wise utilization patterns, and efficiency–performance trade-off. Across these analyses, U-LoRA remains competitive across ranks, tuning granularities, and model scales, while its update patterns show task structure and context-sensitive behavior.

5.2. Scalability Analysis

U-LoRA outperforms LoRA across tested ranks and tuning granularities, remains effective at 72B, and gains +3.24 over TopLoRA on 14B-Instruct.The rank sweep spans 2, 4, 8, 16, and 32; tuning-granularity comparisons use rank r=8.Source: Figure 3 · p. 7, §5.2 · p. 7

5.1. Ablation Study

Removing TOK lowers average performance from 83.97 to 78.95, while removing all components yields 78.52, matching the LoRA baseline.The reported drops are 5.02 points for TOK and 5.45 points when all components are removed; SEQ removal drops performance from 83.97 to 79.16.Source: §5.1 · pp. 6–7, Table 3 · p. 6

5.3. Geometry of Low-Rank Parameter Updates

U-LoRA forms the most compact, structured deeper-layer clusters, aligning same-task updates and placing semantically related tasks closer together.TopLoRA is more scattered, while LoRA shows partial deeper-layer clustering with weaker compactness for some tasks.Source: §5.3 · pp. 7–8

5.4. Case Study

For the same token in different contexts, U-LoRA shows lower similarity in middle and upper layers than LoRA, with a smoother trajectory than TopLoRA.U-LoRA’s layer-wise pattern gradually stabilizes later, whereas TopLoRA shows no consistent trend across layers.Source: §5.4 · p. 8
For different tokens in one sentence, U-LoRA preserves intermediate-layer distinctions before similarity rises in deeper layers.The passage describes this as a “separate-then-integrate” pattern, unlike TopLoRA’s middle- and upper-layer fluctuations.Source: §5.4 · pp. 8–9

5.5. Latency discussion

For every 0.1× increase in latency, U-LoRA gains +1.40 versus TopLoRA’s +0.80; longer generated sequences account for part of the added latency.Accounting for output length decreases observed latency for both methods.Source: §5.5 · p. 9, Table 4 · p. 9

6. Conclusion

The conclusion frames LoRA’s key limitation as largely uniform use of low-rank directions across inputs and training stages. U-LoRA preserves LoRA’s low-rank parameterization while learning sequence-conditioned token utilization and stabilizing it with a bias-corrected historical prior; experiments report consistent improvements under comparable parameter budgets.

LoRA’s key limitation is implicit, largely uniform use of low-rank directions across inputs and training stages.Source: §6 · p. 9
U-LoRA retains LoRA’s low-rank parameterization and combines sequence-conditioned token-wise utilization coefficients with a bias-corrected historical prior.Source: §6 · p. 9
Across diverse backbones and benchmarks, U-LoRA consistently improves on representative LoRA baselines and variants under comparable parameter budgets.Source: §6 · p. 9
The conclusion presents utilization-aware modeling as an effective direction for parameter-efficient fine-tuning.Source: §6 · p. 9

A. Dataset Details

The appendix describes the mathematical-reasoning and GLUE evaluation datasets. The math suite spans six arithmetic reasoning benchmarks, while GLUE covers several language-understanding task types.

Math10K, mainly sourced from GSM8K and AQuA, is used to fine-tune models evaluated on six arithmetic reasoning benchmarks.The benchmarks are MultiArith, AddSub, GSM8K, AQuA, SingleEq, and SVAMP.Source: A. Dataset Details · p. 13, Table 5 · p. 13
GLUE evaluation covers textual entailment, paraphrase identification, semantic textual similarity, linguistic acceptability, and sentiment analysis.The listed datasets include RTE, QNLI, MNLI, MRPC, QQP, STS-B, CoLA, and SST-2.Source: A. Dataset Details · p. 13, Table 6 · p. 13

B. Experimental Settings

The experimental settings specify training configurations for mathematical reasoning and GLUE tasks, shared LoRA settings, and a PCA visualization procedure for update representations. Experiments use PyTorch and Hugging Face models and datasets on RTX 3090 GPUs.

Experiments use PyTorch and Hugging Face models and datasets on NVIDIA GeForce RTX 3090 GPUs.Source: B. Experimental Settings · p. 13
Mathematical-reasoning baselines use specified ranks and AdamW settings, with learning rate 10^-4; experiments include Gemma-7B, LLaMA-3-8B, Qwen3-8B-Base, and Qwen2.5-14B.DoRA uses rank 16, HydraLoRA rank 8 with three B matrices, and MELoRA Mini-LoRA rank 16 with four groups.Source: B. Experimental Settings · pp. 13–14
LoRA-based methods use α=2r, query/key/value projections, dropout 0.05, BF16, batch size 16, sequence length 256, and one training epoch.The momentum parameter β is 0.99 across experiments.Source: B. Experimental Settings · p. 13
Table 7 reports benchmark-wise accuracies and an unweighted average for ablations of token-wise utilization, sequence-level aggregation, and the EMA prior.Source: Table 7 · p. 14
GLUE training runs for task-specific epoch counts, with learning rates of 3 × 10^-4 for RoBERTa-Base and 10^-4 for RoBERTa-Large.Training uses batch size 32 and sequence length 512; methods target query and value weights.Source: B. Experimental Settings · p. 14
The PCA procedure samples 500 tokens per GLUE task and projects update representations from Layers 2 and 22 into two dimensions.It examines the Query-projection update trajectories for five tasks over one epoch of RoBERTa-Large multi-task training.Source: B. Experimental Settings · p. 14

C. Additional Experimental Results

Additional experiments examine U-LoRA’s components, scalability across rank and tuning granularity, convergence, and sensitivity to the EMA momentum parameter. The results report transfer to larger instruction-tuned models, with stable performance variability across repeated runs.

C. Additional Experimental Results

The ablation study evaluates token-wise utilization, sequence-level aggregation, and the EMA-based historical prior individually and in combination on Qwen3-8B mathematical reasoning tasks.Averaged results appear in Table 3, with benchmark-wise results in Table 7.Source: C. Additional Experimental Results · p. 14
U-LoRA exhibits lower training loss from early stages and faster convergence than LoRA across four model architectures, and transfers effectively to 72B instruction-tuned models.The reported scalability experiments also examine different ranks and tuning granularities on Qwen3-8B mathematical reasoning tasks.Source: C. Additional Experimental Results · pp. 14–15
The scalability analysis varies LoRA rank and tuning granularity on Qwen3-8B mathematical reasoning tasks.Accuracy comparisons for target modules and ranks are reported in Tables 8 and 9.Source: C. Additional Experimental Results · p. 14, Table 8 · p. 15, Table 9 · p. 15
U-LoRA is moderately sensitive to β; β = 0.995 gives the best average result, while the experiments use β = 0.99 as a stable trade-off.Further increasing β to at least 0.998 slightly degrades average score; the default is not strictly optimal.Source: C. Additional Experimental Results · p. 15
Across three repeated runs, standard deviations remain stable and are much smaller than U-LoRA’s accuracy improvement on GLUE and mathematical reasoning tasks.The GLUE results use a separate model for each dataset, while one model covers the mathematical reasoning subtasks.Source: C. Additional Experimental Results · p. 15

D. Limitations

U-LoRA’s primary limitation is higher inference latency than standard LoRA. Its dynamic token-wise adaptation cannot be statically merged, leaving runtime overhead that engineering optimizations can reduce but not eliminate.

D. Limitations

U-LoRA has moderately higher inference latency than standard LoRA because its token-wise adaptive weights cannot be statically merged and require additional inference computation.The remaining overhead is characterized as an inherent trade-off of dynamic, input-dependent adaptation.Source: D. Limitations · pp. 15–16
Compared with TopLoRA, U-LoRA has similar parameter counts and peak memory usage but slightly higher inference latency.The overhead includes additional memory access from token-level dynamics and sequence-level aggregation computations.Source: D. Limitations · p. 16
Dynamic non-mergeable methods such as U-LoRA and TopLoRA remain an open efficiency research direction because they add inference overhead over standard LoRA.Engineering optimizations can reduce constant factors but cannot readily eliminate the structural overhead.Source: D. Limitations · p. 16