GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection

Motivation

• 58 GB is required to pre-train LLaMA 7B from scratch with a single batch, exceeding the 24 GB memory of consumer GPUs.The estimate includes 14 GB for parameters, 42 GB for Adam optimizer states and weight gradients, and 2 GB for activations.Source: §1 · p. 1

Limitations of Existing Methods

• LoRA reduces trainable parameters and optimizer states by learning low-rank adaptors while freezing the full-rank pre-trained weight.It reparameterizes W as W0 + BA, with rank r much smaller than the matrix dimensions.Source: §1 · p. 2
• LoRA may underperform full-rank fine-tuning and require full-rank warmup before pre-training in a low-rank subspace.The cited reasons are that optimal weights may not be low-rank and reparameterization changes gradient training dynamics.Source: §1 · p. 2

GaLore

• GaLore projects the gradient matrix into a low-rank form, reducing optimizer-state memory while allowing full-parameter learning.Projection matrices are updated occasionally, such as every 200 iterations, to limit amortized computational overhead.Source: §1 · p. 2

Results

• Comparable performance to the full-rank counterpart is achieved when 8-bit GaLore uses less than 10% of optimizer-state memory during LLaMA 7B pre-training.The result combines 8-bit optimizers and layer-wise weight updates on the C4 dataset.Source: §1 · p. 2
• 85.89 average GLUE score is achieved by rank-4 GaLore on RoBERTa-Base, exceeding LoRA’s 85.61 score.The comparison concerns fine-tuning pre-trained language models on GLUE tasks.Source: §1 · p. 2

Low-rank adaptation

• LoRA reduces memory by maintaining a low-rank weight adaptor for each layer during fine-tuning.Variants extend performance, multitask learning, and memory-footprint improvements.Source: §2 · p. 2

Subspace learning

• Subspace learning optimizes model weights within a low-rank parameter subspace and has been applied in meta-learning and continual learning.The cited studies report that learning primarily occurs within a significantly low-dimensional parameter subspace.Source: §2 · pp. 2–3

Projected gradient descent

• GaLore differs from traditional projected gradient descent by analyzing matrix-structured gradients that arise naturally in multilayer neural networks.Traditional PGD generally treats the objective as a black-box nonlinear function and studies gradients in vector space.Source: §2 · p. 3

Low-rank gradient

• Neural-network gradients are naturally low-rank, a property previously used to reduce communication costs and training memory.The cited applications span gradient communication and memory-footprint reduction.Source: §2 · p. 3

Memory-efficient optimization

• Memory-efficient optimization reduces gradient-statistics memory through adaptive-optimizer designs, quantization, and fused backward-update operations.These approaches target optimizer states or weight-gradient storage through different mechanisms.Source: §2 · p. 3

3.1. Background

• Full-rank training stores parameter weights, gradients, and stateful optimizer statistics, making optimizer memory a major cost.For Adam, the parameter matrix and two optimizer-state matrices together require 3mn memory.Source: §3.1 · p. 3
• Low-rank parameter updates reduce trainable parameters, but constraining the weight matrix can prevent reaching a high-rank optimum.The paper also identifies altered gradient dynamics as a limitation of low-rank reparameterization.Source: §3.1 · p. 3, §3.2 · p. 3

3.2. Low-Rank Property of Weight Gradient

• Gradient matrices can become low-rank during training for certain gradient forms and associated network architectures, even when weight matrices are not low-rank.Under the stated conditions, the stable rank decreases as one term decays exponentially; in particular, the paper derives sr(G_t) ≤ n/2 for large t.Source: §3.2 · pp. 3–4

3.3. Gradient Low-rank Projection (GaLore)

• GaLore projects G into the compact form P^⊤GQ, applies optimizer statistics there, and projects the resulting update back to the original parameter space.The projection matrices can be obtained from the leading singular vectors of the gradient via SVD.Source: §3.3 · pp. 4–5
• Changing projection subspaces lets GaLore learn full-rank weights when gradient principal subspaces evolve during complex optimization.With fixed projections, weights grow only along those subspaces; GaLore switches subspaces without increasing the memory footprint.Source: §3.3 · p. 5
• When the projection rank reaches min(m, n) and ρ_t ≡ 1, GaLore follows the exact training trajectory of the original model, unlike simultaneous full-rank LoRA-factor optimization.This distinction arises because GaLore projects updates rather than introducing additive low-rank adaptors.Source: §3.3 · pp. 4–5

Composition of Low-Rank Subspaces

• GaLore switches across low-rank subspaces during training, allowing full-rank weights without increasing the memory footprint.Projection matrices are recomputed from the current gradient when switching subspaces.Source: §4 · p. 5, §4.1 · p. 5
• A single fixed low-rank subspace may miss changing gradient trajectories, whereas switching subspaces supports successful LLM pre-training.The authors identify multiple low-rank subspaces as key to successful pre-training.Source: §4 · p. 5, §4.1 · p. 5
• The switching frequency T trades computational overhead and optimization fidelity against the risk of remaining in an unimportant subspace.The authors report that T between 50 and 1000 makes little difference in practice, with SVD overhead below 10%.Source: §4.1 · p. 5, §4.4 · p. 6

Memory-Efficient Optimization

• GaLore projects gradients into low-rank form so optimizers such as Adam track statistics only in the compact space.The method uses projection matrices and updates the weight in the original space after compact-space optimization.Source: §4.2 · p. 6
• GaLore requires (mn + mr + 2nr) memory, compared with (mn + 3mr + 3nr) for LoRA.GaLore can merge updates into the base weight and therefore does not store a separate low-rank factorization.Source: §4.2 · p. 6, Table 1 · p. 6

Combining with Existing Techniques

• GaLore is compatible with 8-bit optimizers and per-layer weight updates to further reduce training memory.Per-layer updates apply weight updates during backpropagation rather than storing all layer gradients until the end.Source: §4.3 · p. 6

Evaluation setup

• GaLore is evaluated on LLaMA-based pre-training over C4 and RoBERTa fine-tuning on GLUE tasks.Experiments use model sizes up to 7B for pre-training and compare GaLore with full fine-tuning and LoRA for fine-tuning.Source: §5 · pp. 6–7

Optimizer compatibility

• GaLore applies to AdamW, 8-bit Adam, and Adafactor without significantly affecting convergence.On LLaMA 1B for 10K steps, rank 512 reduces memory by up to 62.5% on top of savings from 8-bit Adam or Adafactor.Source: §5.2 · p. 8

Memory measurement

• 22.0G memory enables pre-training LLaMA 7B with a small per-GPU token batch on a single 24GB consumer GPU.The setup excludes activation checkpointing and memory offloading; checkpointing can raise the per-GPU batch size to 4096.Source: §5.5 · pp. 8–9
• 63.3% total-memory reduction versus the BF16 Adam baseline comes from 65.5% optimizer-state savings and reduced weight-gradient storage.Compared with 8-bit Adam, GaLore reduces 9.6G in optimizer states and 13.5G in weight gradients.Source: §5.5 · p. 9

Throughput

• 1019.63 tokens/second for 8-bit GaLore incurs 17% throughput overhead compared with 8-bit Adam.Disabling per-layer weight updates reaches 1109.38 tokens/second, improving throughput by 8.8%.Source: §5.5 · p. 9

Subspace updates

• Both overly frequent and overly slow subspace changes hurt convergence.For small ranks, more frequent switching avoids optimization in an unsuitable subspace, whereas larger ranks provide more coverage.Source: How many subspaces are needed during… · p. 9, Figure 5 · p. 9

Rank and training duration

• Within a certain rank range, decreasing rank only slightly slows convergence in an approximately linear trend.Rank 128 trained for 80K steps achieves lower loss than rank 512 trained for 20K steps.Source: How does the rank of subspace affect… · p. 9
• Reducing rank trades memory for computation by allowing longer training within a fixed memory budget.The paper presents this as a way to preserve performance in memory-constrained settings.Source: How does the rank of subspace affect… · p. 9

Contribution

• GaLore reduces optimizer-state memory by up to 65.5% while maintaining efficiency and performance in large-scale LLM pre-training and fine-tuning.The conclusion frames GaLore as a strategy for memory-efficient training rather than a low-rank parameterization of the weights.Source: §7 · p. 9

Open problems

• Open problems include applying GaLore to vision transformers and diffusion models, lowering projection-matrix memory, and testing elastic distributed training on low-bandwidth hardware.These are identified as future research directions.Source: §7 · p. 9

Implications

• The authors position GaLore as a potential tool for training large-scale models on consumer-grade hardware with limited resources.The stated motivation is to inspire further research on memory-efficient training through gradient low-rank projection.Source: §7 · p. 9

Relation to prior methods

• Adafactor factorizes second-order statistics, whereas GaLore exploits low-rank gradient structure and can reduce memory for both first- and second-order statistics.GaLore can also be combined with Adafactor for further memory reduction.Source: A. Additional Related Works · p. 13
• LOMO and AdaLOMO reduce optimizer memory through fused backward operations, but their direct applicability to larger-scale pre-training from scratch is constrained by Adafactor instability.The paper specifically notes increased training instabilities for vanilla Adafactor at larger scales.Source: A. Additional Related Works · p. 13

Theoretical analysis

• The theoretical analysis derives gradient structure for chained reversible networks and notes that self-attention analysis via JoMA is left for future work.The result is stated for reversible architectures and includes linear layers and reversible activations as examples.Source: §B.1 · p. 13

• The gradient follows structured dynamics whose vectorized update is governed by a positive semidefinite operator S.The analysis assumes a parametric gradient form and studies vanilla SGD weight updates.Source: §B.2 · p. 15
• The initial gradient decomposes into components in the minimal eigenspace V1 and its orthogonal complement.The component in V1 evolves according to the smallest eigenvalue, while the residual evolves separately.Source: §B.2 · pp. 15–16
• The stable rank of the gradient component aligned with V1 is bounded by n − N′ under the stated rank condition.Here N′ is the rank of the relevant feature set, with N′ < n.Source: §3.2 · p. 4, §B.2 · p. 16
• When the residual term becomes negligible, the gradient can be represented as a sum of N′ rank-1 matrices and therefore has rank at most N′.This establishes low-rank behavior after sufficiently long training under the theorem’s assumptions.Source: §B.2 · p. 16
• If the minimal eigenspace is one-dimensional with a decomposable eigenvector, the projected gradient has stable rank 1 and becomes rank-1.The projection onto V1 is proportional to the decomposable eigenvector.Source: §B.2 · pp. 15–16

Gradient structure in Transformers

• The Transformer analysis studies the project-up FFN weight W and shows its gradient becomes low-rank over time under stationary-gradient assumptions.The project-up weight has dimensions m × n, where m is the embedding dimension and n is the number of hidden nodes.Source: §B.3 · p. 17
• The transformed matrix V := U ⊤W is decomposed row-wise, with row growth rates differing across token indices.Different growth rates produce different row norms and contribute to low-rank structure.Source: §B.3 · pp. 17–18
• Eventually, V becomes rank-1 because one row’s growth diverges while the others remain finite.The result follows from the relative critical times of the row-growth functions.Source: §B.3 · p. 18
• The gradient of V is even more low-rank than V itself because the derivative growth contains α_l in its exponents.The analysis characterizes this as the gradient being “exponentially more low-rank” than the matrix.Source: §B.3 · pp. 17–18

Convergence of fixed projections

• With fixed projection matrices P and Q, GaLore converges when the minimum κ_t exceeds LA + LBLCD^2.Under the theorem’s continuity and boundedness assumptions, the projected residual R_t tends to zero.Source: §B.4 · pp. 18–19

LLaMA architecture and training settings

• All LLaMA models use sequence length 256, a batch size of 131K tokens, 10% learning-rate warmup, and cosine annealing to 10% of the initial rate.These settings are shared across the pre-training experiments.Source: §C.1 · p. 20, Table 5 · p. 20
• GaLore uses learning rate 0.01, scale factor α = 0.25, and a subspace change frequency of 200 across model sizes.The authors report that GaLore is insensitive to hyperparameters and stable with the same learning rate across sizes.Source: §C.1 · p. 20

Memory measurement

• Memory estimates include BF16 weight parameters and optimizer states, with low-rank methods additionally accounting for their trained low-rank parameters.The estimates are based on the number of original and low-rank parameters rather than direct component measurement.Source: §C.2 · p. 20, Table 6 · p. 20
• GaLore retains full-rank weight memory while low-rank methods reduce weight memory, according to the reported parameter-memory estimates.The table separates full-rank, GaLore, low-rank, LoRA, and ReLoRA estimates.Source: §C.2 · p. 20

Training progression

• Across 130M, 350M, 1B, and 7B models, GaLore closely matches the full-rank training trajectory and initially converges slightly faster than LoRA.These comparisons are reported for the training progression shown in Figure 6.Source: §C.3 · p. 21, Figure 6 · p. 21

SQuAD evaluation

• On SQuAD, GaLore and LoRA are evaluated with the pre-trained BERT-Base model using rank 16 for both methods.The evaluation reports both Exact Match and F1 scores.Source: §D.2 · p. 22, Table 8 · p. 22
• GaLore outperforms LoRA on both Exact Match and F1 scores in the SQuAD evaluation.Source: §D.2 · p. 22

Other fine-tuning evaluations

• Additional experiments apply rank-128 GaLore and LoRA to Gemma-2b, Phi-2, and LLaMA-7B on OpenAssistant Conversations and Belle-1M.The Belle-1M evaluation reports testing perplexity.Source: §D.3 · p. 22, §D.4 · p. 22, Table 10 · p. 23