Source-linked AI summary
Rotated Manifold Optimization for Low-Rank Adaptation
Yuhui Ding, Javier Zazo, James Hensman
TL;DR
LoRA makes finetuning less costly by updating low-rank factors, but their gauge symmetry motivates geometry-aware optimization. RoM extends rotated normalization to the fixed-rank manifold, achieving lower held-out loss and better or comparable downstream performance across supervised finetuning and reinforcement learning tasks.
Problem
Full-parameter finetuning is costly in computation and memory, while LoRA's factorization has gauge symmetry that motivates optimizing with its geometry in view.
Source: §1 · p. 1Method
RoM directly optimizes LoRA's weight adaptation on the fixed-rank manifold, applying normalization in a rotated tangent-momentum basis and projecting the result back to the tangent space.
Source: §1 · p. 1, §3 · p. 3, §3.2 · p. 5Results
Takeaways & Limitations
Both rotation and normalization contribute to RoM's optimization performance, while its memory requirement remains comparable to conventional LoRA optimization.
Source: §5 · p. 9The BaLoRA results use a matched training protocol with independent learning-rate tuning, so they are not directly comparable to those reported by Castin et al. (2026).
Source: §4.2 · p. 7
Abstract
from arXiv · showhide
We propose a novel optimizer for low-rank adaptation (LoRA) that explicitly incorporates the gauge symmetry of low-rank factorization. Our optimizer extends recent matrix optimizers for full-parameter training to the manifold of fixed-rank matrices by interpreting them as normalization under a rotated basis. We show how rotation and normalization can be integrated with the fixed-rank manifold efficiently. Our optimizer converges faster to lower held-out loss and achieves better or comparable downstream performance on both supervised finetuning and reinforcement learning tasks.
1 INTRODUCTION
LoRA reduces the compute and memory costs of finetuning, but its factorization has gauge symmetry that motivates geometry-aware optimization. The paper introduces RoM, which applies rotated optimization on the fixed-rank manifold, and reports lower held-out loss with better or comparable downstream performance.
1 INTRODUCTION
- LoRA freezes pretrained weights and represents weight updates with low-rank matrix products, reducing trainable parameters and memory while maintaining competitive performance.These savings address the computation and memory expense of full-parameter finetuning as models grow.Source: §1 · p. 1
- LoRA's factors admit infinitely many reparameterizations with the same weight update, motivating an optimizer that respects this gauge symmetry.The model output depends on the low-rank factors through their product.Source: §1 · p. 1
- RoM optimizes the product of low-rank factors on the fixed-rank manifold, applying row and column normalization in a tangent-momentum basis before projecting and retracting.Its low-rank tangent-vector operations avoid materializing full weight-sized matrices.Source: §1 · p. 1
- RoM achieves substantially lower held-out loss and better or comparable downstream performance than other LoRA optimizers on supervised finetuning and reinforcement learning tasks.The optimizer is reported to add only modest computational overhead.Source: §1 · p. 2
2 BACKGROUND
This section introduces rotated optimization, LoRA's low-rank factorization and symmetry, and the manifold concepts used to describe optimization over constrained spaces.
2 BACKGROUND
- The background reviews rotated optimization, LoRA, and Riemannian optimization as foundations for the paper's method.It notes that the section's notation is used throughout the paper.Source: §2 · p. 2
2.1 ROTATED OPTIMIZATION
- Rotated optimization expresses matrix momentum in rotated coordinates, applies normalization there, and rotates the result back.The rotation matrix defines the coordinate system for normalization.Source: §2.1 · p. 2
2.2 LOW-RANK ADAPTATION (LORA)
- LoRA freezes the pretrained weights and represents the adaptation as AB⊤; because the output depends only on that product, many factor reparameterizations preserve the loss.The paper's optimizer explicitly incorporates this gauge symmetry.Source: §2.2 · p. 2
2.3 RIEMANNIAN OPTIMIZATION
- Riemannian optimization equips tangent spaces with a smoothly varying metric and obtains a descent direction by projecting the ambient gradient onto the tangent space.A basic update then takes a step using that Riemannian gradient.Source: §2.3 · pp. 2–3
3 METHOD
RoM directly optimizes the LoRA adaptation on the fixed-rank matrix manifold. It combines factorized Riemannian operations with rotated normalization in a low-dimensional frame.
3 METHOD
- RoM directly optimizes the LoRA adaptation on the fixed-rank manifold, representing rank-r matrices through orthonormal singular-vector factors and positive singular values.The paper describes this as optimizing the product of the low-rank factors rather than each factor separately.Source: §3 · p. 3, §3.1 · p. 3
3.1 FIXED-RANK MANIFOLD
- Tangent vectors use compact factorized components, and gradient computation avoids materializing the full m×n Euclidean gradient.The method computes the required gradient products directly and reports memory use of the same order as storing standard LoRA factor gradients.Source: §3.1 · pp. 3–4
- Retraction applies rank-r truncated SVD to W+Z for its best rank-r approximation, while momentum is projected into the next tangent space before blending with the fresh gradient.The momentum transport operation remains factorized.Source: §3.1 · p. 4
3.2 ROTATED OPTIMIZATION ON MANIFOLD
- Because tangent momentum has rank at most 2r, RoM rotates within a 2r-dimensional frame formed from its left singular vectors, computed using thin QR factorizations and a small SVD.This avoids the m×m rotation matrix required by a naive ambient-space rotation.Source: §3.2 · pp. 4–5
- RoM applies alternating row and column normalization (K=5) to reduced rotated momentum, then projects the mapped-back direction onto the tangent space without materializing an m×n matrix.The normalization operates on FᵀM, and the projection is combined with preceding operations for an efficient update.Source: §3.2 · pp. 5–6
4 EXPERIMENTS
Experiments compare RoM with existing LoRA optimizers on supervised finetuning and reinforcement learning, then assess RoM’s components and learning-rate sensitivity.
4.1 EXPERIMENTAL SETUP
- LoRA adapters cover all attention- and MLP-module linear layers, while one-dimensional parameters, input embeddings, and the output head remain frozen.SFT uses warm-up followed by linear decay; RL uses a constant learning rate. The BaLoRA results use a protocol not directly comparable to Castin et al. (2026).Source: §4.1 · p. 6, §4.2 · p. 7
4.2 SUPERVISED FINETUNING
- Across three model sizes, RoM consistently achieves the lowest final math-SFT validation loss, while downstream accuracy differences are smaller and seed-sensitive.For the Llama models, RoM has the highest mean accuracy on both benchmarks, but differences are small relative to seed variation.Source: Table 1 · p. 7, Table 2 · p. 7, §4.2 · p. 7
- On code SFT, RoM achieves the lowest validation loss, but optimizer differences in HumanEval and MBPP Pass@1 are minor relative to seed variation.The reported validation-loss gaps exceed seed variation, unlike the Pass@1 differences.Source: §4.2 · pp. 7–8, Table 3 · p. 8
4.4 ADDITIONAL ANALYSIS
- On math SFT, full RoM has the lowest validation loss; normalization without rotation narrows the gap, while Riemannian momentum performs substantially worse.The ablation uses Llama-3.2-3B at rank 32.Source: Figure 4 · p. 9, §4.4 · p. 9
- RoM maintains the lowest validation loss at 0.5×, 1×, and 2× its best learning rate, while symmetry-aware optimizers tend to degrade less away from their best rates.The comparison uses the same math-SFT setting.Source: Figure 4 · p. 9, §4.4 · p. 9
4.3 REINFORCEMENT LEARNING
- RoM improves accuracy faster than Adam and finishes higher on all three held-out RL benchmarks, at 1.08× Adam’s per-step training time.Table 4 reports final accuracy across three runs; Figure 2 shows accuracy throughout finetuning.Source: Figure 2 · p. 8, Table 4 · p. 9, §4.3 · p. 9
5 CONCLUSION
The conclusion presents RoM as a fixed-rank manifold optimizer that avoids factorization redundancy and reports lower validation loss across SFT and RL experiments. The AI-use statement describes assistance with coding, documentation, and manuscript polishing.
5 CONCLUSION
- Across SFT and RL experiments, RoM consistently achieves lower validation loss than existing LoRA optimizers and improves downstream performance in RL.The conclusion also reports memory requirements comparable to conventional LoRA optimization and says ablations indicate both rotation and normalization contribute to optimization performance.Source: §5 · p. 9
AI USE STATEMENT
- The authors report using LLMs for coding and debugging, organizing experimental progress, and polishing manuscript text and figures; they say they reviewed and verified all outputs.They take responsibility for the final content.Source: AI USE STATEMENT · p. 10
A ALGORITHMS
The algorithms describe factorized momentum transport on the fixed-rank manifold and an iterative row-and-column normalization function for a reduced representation.
A ALGORITHMS
- Algorithm 2 takes old and new orthonormal frames, a momentum coefficient, old momentum components, and Riemannian gradient components, and outputs updated momentum components.The displayed fragments also identify projection onto the new tangent space and accumulation with the fresh Riemannian gradient.Source: A ALGORITHMS · p. 13
- Algorithm 3 takes a reduced representation C and an iteration count K, and outputs its normalized representation.The listed operations normalize rows and columns by their respective Euclidean norms.Source: A ALGORITHMS · p. 13
- The normalization function applies row-wise and column-wise L2 normalization to the reduced representation.The algorithm listing gives separate row and column normalization steps.Source: A ALGORITHMS · p. 13
B RELATED WORK
The related-work discussion distinguishes prior LoRA optimizers by how they address factor learning, reparameterization, and fixed-rank geometry, then identifies RoM's rotated optimization framework as its distinction.
B RELATED WORK
- Prior approaches include separate learning rates for low-rank factors, matrix preconditioners for feature learning, reparameterization-invariant adapter updates, and balanced projection after Adam updates.The discussion notes that Castin et al. (2026) theoretically connect balanced low-rank factors with fast convergence, but do not account for momentum.Source: B RELATED WORK · p. 13
- A concurrent approach bounds per-sample loss change, while the reviewed optimizers compute individual updates for each low-rank factor.Source: B RELATED WORK · p. 13
- Riemannion projects Muon updates onto the fixed-rank manifold's tangent space, making it the closest related work discussed.Source: B RELATED WORK · p. 13
- RoM also uses the fixed-rank manifold perspective, but develops its optimizer through rotated optimization, with rotation and normalization as core components.Source: B RELATED WORK · p. 13
C EFFECT OF LORA RANK
The supplied captions describe two math-SFT validation-loss views: a training-step trace at rank 32 and final losses across LoRA ranks for RoM and Adam.
C EFFECT OF LORA RANK
- Figure 5 reports mean cross-entropy across response tokens over the last 100 training steps on math SFT at rank 32.It is described as the counterpart of Figure 1.Source: Figure 5 · p. 14
- Figure 6 compares final math-SFT validation loss for RoM and Adam across LoRA ranks using Llama-3.2-3B.Learning rates are independently tuned for each optimizer and rank.Source: Figure 6 · p. 14