PaperlayerView exact revision on arXiv

Source-linked AI summary

Normalized Low-Rank Adaptation

Jiale Kang, Ziyin Yue, Zheng Zhan, Yangyi Huang, Weiyang Liu

arXiv:2608.31036v1cs.LG

TL;DR

LoRA’s training dynamics remain underexplored, particularly because its zero-initialized up-projection makes early optimization depend on the down-projection. The paper introduces NoRA, which normalizes that projection during training or only at initialization, and reports consistent gains across pretraining, supervised finetuning, and reinforcement learning. These improvements preserve exact mergeability and require neither extra trainable parameters nor inference-time computation.

  • Problem

    LoRA’s optimization dynamics remain poorly understood, motivating study of whether regularizing its down-projection can improve optimization.

  • Method

    NoRA normalizes the down-projection matrix A along the rank dimension during training, while NoRA-init applies the normalization only at initialization before standard LoRA optimization.

  • Results

    Across pretraining, supervised finetuning, and reinforcement learning, NoRA and NoRA-init consistently accelerate convergence, improve stability and downstream performance, and mitigate catastrophic forgetting.

  • Takeaways & Limitations

    Rank-dimension normalization is an important LoRA design principle that improves optimization while preserving exact weight mergeability without additional trainable parameters or inference-time computation.

Abstract

from arXiv · show

While low-rank adaptation (LoRA) is widely used for parameter-efficient model adaptation, how to regularize its training dynamics for stable and effective optimization remains underexplored. Because LoRA initializes the up-projection to zero, its early optimization dynamics are largely governed by the down-projection. Building on this observation, we introduce Normalized Low-Rank Adaptation (NoRA), a simple yet effective method that normalizes the down-projection matrices during training. We further show that the same normalization can be applied only at initialization, improving standard LoRA without requiring repeated normalization throughout training. Across pretraining, supervised finetuning, and reinforcement learning, NoRA consistently accelerates convergence, improves performance and training stability, and mitigates catastrophic forgetting. These benefits require neither additional trainable parameters nor inference-time computation, making NoRA a simple and broadly applicable enhancement to LoRA.

1 INTRODUCTION

LoRA’s down-projection is an underexplored determinant of optimization dynamics. NoRA normalizes it along the rank dimension, improving convergence, stability, performance, and resistance to catastrophic forgetting across training settings.

  • The method is motivated by normalized latent representations that stabilize training while preserving LoRA’s linearity and exact weight mergeability.Direct latent normalization would make the transformation input-dependent and nonlinear, so NoRA transfers scale control to the projection matrix.
  • Experiments across pretraining, supervised finetuning, and reinforcement learning show improved convergence, training stability, downstream performance, and resistance to catastrophic forgetting.The reported benefits span diverse base models, tasks, and training configurations.
  • LoRA’s down-projection matrix A is identified as an important design dimension affecting optimization dynamics, training efficiency, and downstream performance.
  • NoRA normalizes the down-projection A along the rank dimension, while NoRA-init applies the same normalization only once at initialization.NoRA maintains the constraint during training; NoRA-init subsequently uses standard LoRA optimization.

2 PRELIMINARIES

LoRA represents weight updates with low-rank factors and initializes its up-projection at zero, making early optimization depend on the initial down-projection. A latent-feature perspective unifies related low-rank methods by how they construct the feature presented to the up-projection.

  • LoRA parameterizes a pretrained weight update as ΔW = αBA, where A and B are low-rank factors and r ≪ min(d, k).
  • Standard LoRA initializes B(0) = 0 to preserve the pretrained model, so its earliest optimization is determined by the latent representation induced by A(0).
  • A unified formulation treats low-rank methods as constructing a latent feature ϕ(x) that is presented to the up-projection.
  • Within this perspective, LoRA uses Ax, structured methods use A_initx, and normalized approaches use Norm(Ax).
  • The latent-projection perspective motivates regularizing the latent projection while retaining LoRA’s exact mergeability.

3 NORA: NORMALIZED LOW-RANK ADAPTATION

NoRA normalizes LoRA’s down-projection along the rank dimension, preserving a linear, exactly mergeable update while regulating latent projection magnitudes. NoRA-init applies the same normalization only at initialization, and the paper connects both approaches to stronger early optimization and improved stability.

  • 3.1 FROM NORMALIZED LATENT FEATURES TO NORMALIZED LOW-RANK PROJECTION: NoRA normalizes each input-to-latent projection vector in the down-projection A along the rank dimension, producing B · Norm(A)x while preserving exact weight mergeability.The normalization depends only on LoRA parameters, so the transformation remains linear in x and can be absorbed into the low-rank update after training.
  • 3.1 FROM NORMALIZED LATENT FEATURES TO NORMALIZED LOW-RANK PROJECTION: NoRA can be applied continuously during training or once at initialization through NoRA-init, which then uses the standard LoRA parameterization without persistent normalization.NoRA-init is motivated by analyses suggesting that much of NoRA’s benefit is established at the beginning of optimization.
  • 3.2 CONNECTION TO MISS AND BLOCK IDENTITY MATRIX INITIALIZATION: MiSS is a special case of NoRA because its fixed block-identity down-projection has unit-norm columns, while BIMI uses that structure only to initialize a fully trainable projection.BIMI retains the standard LoRA update form, ∆y = αBAx, during training.
  • 3.3 WHY DOES NORA WORK? A PRECONDITIONING PERSPECTIVE: Figure 1 shows substantially stronger early NoRA gradients than standard LoRA, with gradient norms approaching full finetuning; MiSS exhibits a highly similar pattern.The figure compares gradient-norm dynamics across initialization methods and LoRA ranks on LLaMA-3.2-3B trained on Math.
  • 3.3 WHY DOES NORA WORK? A PRECONDITIONING PERSPECTIVE: At initialization, LoRA behaves like full finetuning with the gradient right-multiplied by P = α^2A^⊤A, a rank-r input-side preconditioner determined by A.The first-order update includes coordinate-specific scaling by the squared latent-vector lengths and crosstalk between input coordinates.
  • 3.3 WHY DOES NORA WORK? A PRECONDITIONING PERSPECTIVE: Random A initialization makes coordinate learning rates too small and unbalanced, whereas rank-dimension normalization sets ∥a_j∥2 = 1 and yields Diag(P) = I deterministically.The paper states that the resulting adapter gradient norm can match full finetuning’s gradient norm in expectation independently of rank, unlike row normalization.

4 EXPERIMENTS AND RESULTS

The experiments evaluate NoRA and its initialization-only variant across pretraining, supervised finetuning, and reinforcement learning. Across these settings, normalized low-rank projections improve convergence, stability, and downstream performance.

  • Evaluation scope: NoRA is evaluated across pretraining, supervised finetuning, and reinforcement learning, with ablations on normalization dimension and initialization distribution.The evaluation includes LLM pretraining, diverse SFT tasks, RL with verifiable rewards, and targeted ablations.
  • Normalization ablations: Normr improves performance across random, Gaussian, and Kaiming Uniform initializations, while Normk provides little improvement.After Normr normalization, the three initialization schemes achieve similar performance, indicating limited dependence on initialization distribution.
  • Supervised finetuning: NoRA raises the SFT average from 37.93 with standard LoRA to 43.37, a gain of 5.44 points.It also reaches 61.63 on GSM8K and 42.10 on HumanEval, outperforming several representative PEFT methods in average performance.
  • Supervised finetuning: NoRA-init raises the SFT average from 37.93 to 42.38, while persistent normalization further increases it to 43.37.The comparison indicates that initialization accounts for much of the gain, with continued normalization providing additional improvement.
  • Knowledge retention: NoRA improves retained-benchmark change to +0.02, compared with −0.56 for LoRA and -0.70 for MiSS.The result indicates improved adaptation performance alongside retention of pretrained knowledge.
  • Reinforcement learning: NoRA achieves an average RLVR score of 44.4, compared with 42.8 for standard LoRA and 41.0 for the base model.The method improves broadly across the evaluated mathematical reasoning benchmarks.

5 RELATED WORK AND CONCLUDING REMARKS

The paper situates NoRA among parameter-efficient adaptation methods and concludes that down-projection geometry and scale critically shape LoRA optimization. It presents rank-dimension normalization as a broadly effective improvement across training regimes.

  • Related work: Parameter-efficient finetuning methods address the growing cost of adapting increasingly large language models.LoRA is highlighted for its simplicity, efficiency, and empirical performance.
  • Related work: Recent LoRA studies show sensitivity to initialization, scaling, and early-stage optimization dynamics.The related methods include spectral, orthogonal, gradient-guided, activation-based, and rank-dependent approaches.
  • Concluding remarks: NoRA normalizes the down-projection along the rank dimension, controlling input-to-latent scale while preserving linearity and exact mergeability.The paper interprets this as correcting scale imbalance in LoRA’s implicit input-side preconditioner.
  • Concluding remarks: NoRA consistently improves convergence, training stability, and downstream performance across pretraining, supervised finetuning, and RLVR.The conclusion presents rank-dimension normalization as a general optimization principle for low-rank adaptation.

A PRETRAINING SETTINGS

The pretraining experiments use a fixed training setup on FineWeb-10BT. The configuration specifies the optimization schedule, sequence length, batch size, and gradient clipping.

  • Training setup: Pretraining uses FineWeb-10BT for 20,480 optimization steps with sequence length 2,048 and global batch size 256.These settings define the main pretraining workload and batching configuration.
  • Optimization: AdamW uses a peak learning rate of 3 × 10^-4, epsilon 10^-15, 1,024 warmup steps, cosine decay to 10% of peak, and gradient clipping at 1.0.The schedule and clipping settings are held within the stated common training configuration.

B SUPERVISED FINETUNING SETTINGS

The SFT experiments tune each method individually and select NoRA’s default scaling by balancing adaptation with retained pretrained knowledge.

  • Hyperparameter tuning: SFT hyperparameters are tuned separately for each method to ensure competitive performance.The comparison therefore uses individually tuned method configurations rather than one shared hyperparameter setting.
  • Scaling choice: NoRA uses α = r by default because its early gradient norm is close to full finetuning’s gradient norm.Although α = 2r can improve loss fitting in some cases, it may increase forgetting of pretrained knowledge.

C REINFORCEMENT LEARNING SETTINGS

The reinforcement-learning experiments use a fixed DAPO-based setup on DeepSeek-R1-Distill-Qwen-1.5B, with controlled parameter-efficient training configurations across methods.

  • Experiments use DeepSeek-R1-Distill-Qwen-1.5B with the DAPO objective and DAPO-Math-17K dataset.Training uses 8 GPUs and bfloat16 precision.
  • Training runs for 1,024 optimization steps with global batch size 128 and learning rate 1 × 10−5 under cosine decay without warmup.Each prompt produces 8 responses, with maximum completion and prompt lengths of 16,384 and 512 tokens, respectively.
  • Parameter-efficient training uses rank 32 and α = 64, applying low-rank modules to all attention and MLP projections.The covered projections are q, k, v, o, up, down, and gate.
  • The same training configuration is used across methods to ensure a controlled comparison.These settings are listed as the hyperparameters for the reinforcement-learning experiments in Table 9.
Loading 2608.31036v1…