PaperlayerView exact revision on arXiv

Source-linked AI summary

Q-GaLore: Quantized GaLore with INT4 Projection and Layer-Adaptive Low-Rank Gradients

Zhenyu Zhang, Ajay Jaiswal, Lu Yin, Shiwei Liu, Jiawei Zhao, Yuandong Tian, Zhangyang Wang

arXiv:2407.08296v1cs.LG

TL;DR

LLM training is memory-intensive, and GaLore's repeated SVD subspace updates add training latency. Q-GaLore combines quantization, low-rank gradients, and adaptive subspace updates; it reports reduced memory use in pre-training and fine-tuning, including 7B training on a 16GB GPU. Its reported 7B pre-training run was limited to 40K steps and had higher perplexity than the 1B model.

  • Problem

    LLM training requires substantial memory, while GaLore's frequent SVD subspace updates add latency and its memory needs can exceed those of 16GB devices.

    Source: §1 · p. 1

  • Method

    Q-GaLore combines low-precision weights and projection matrices with low-rank gradients, stochastic rounding, and adaptive layer-wise SVD updates.

    Source: §1 · p. 2

  • Results

    Fine-tuning uses up to 50% less memory than LoRA and GaLore while outperforming QLoRA at the same memory cost; pre-training reduces memory requirements by 61% versus full-rank training and 30% versus GaLore.

    Source: §1 · pp. 2–3

  • Takeaways & Limitations

    Q-GaLore makes it feasible to pre-train LLaMA-7B from scratch on one RTX 4060 Ti with 16GB, with performance comparable to its full-rank counterpart.

    Source: §1 · p. 3

    The reported from-scratch LLaMA-7B pre-training run covered 40K steps and had higher perplexity than the 1B model; training toward 150K steps was ongoing.

    Source: §4.2.2 · p. 7

Abstract

from arXiv · show

Training Large Language Models (LLMs) is memory-intensive due to the large number of parameters and associated optimization states. GaLore, a recent method, reduces memory usage by projecting weight gradients into a low-rank subspace without compromising performance. However, GaLore relies on time-consuming Singular Value Decomposition (SVD) operations to identify the subspace, and the frequent subspace updates lead to significant training time overhead. Moreover, GaLore offers minimal improvements in accuracy and efficiency compared to LoRA in more accessible fine-tuning scenarios. To address these limitations, we introduce Q-Galore, a novel approach that substantially reduces memory usage by combining quantization and low-rank projection, surpassing the benefits of GaLore. Our method is based on two key observations: (i) the gradient subspace exhibits diverse properties, with some layers converging early in training while others are subject to frequent changes; (ii) the projection matrices are highly resilient to low-bit quantization. Leveraging these insights, Q-GaLore adaptively updates the gradient subspace based on its convergence statistics, achieving comparable performance while significantly reducing the number of SVD operations. We maintain the projection matrices in INT4 format and weights in INT8 format, incorporating stochastic rounding to capture accumulated gradient information. This approach enables a high-precision training trajectory using only low-precision weights. We demonstrate that Q-GaLore achieves highly competitive performance with exceptional memory efficiency. At pre-training, Q-GaLore facilitates training a LLaMA-7B model from scratch on a single NVIDIA RTX 4060 Ti with only 16 GB memory. At fine-tuning, it reduces memory consumption by up to 50% compared to LoRA and GaLore, while consistently outperforming QLoRA at the same memory cost.

1 Introduction

Q-GaLore targets the memory and SVD overheads that limit low-rank LLM training, combining low-precision training with adaptive layer-wise subspace updates. It reports memory-efficient pre-training and fine-tuning, including LLaMA-7B training on a 16GB GPU.

1 Introduction

  • GaLore reduces training memory by up to 63.3%, but its 24GB requirement exceeds the 16GB available on devices such as the RTX 4060 Ti.Regular SVD subspace updates, sometimes every 200 iterations, take about 10 minutes for LLaMA-7B and add training latency.Source: §1 · p. 2
  • Gradient subspaces behave differently across layers: some converge early, some remain stable in a window, and others keep changing.GaLore projection matrices can be quantized to 4 bits without sacrificing training quality.Source: §1 · p. 2
  • Q-GaLore combines low-precision weights and projection matrices with low-rank gradients, using stochastic rounding to preserve gradient information and training stability.The model is quantized to 8 bits and the projection matrix to 4 bits; the approach reduces memory by approximately 28.57% for gradient low-rank training.Source: §1 · p. 2
  • Layer-wise convergence monitoring adaptively reduces SVD frequency for subspaces that change little, saving over 32 hours for 7B training.This targets the training time associated with SVD operations.Source: §1 · p. 2
  • Fine-tuning uses up to 50% less memory than LoRA and GaLore while outperforming QLoRA at the same memory cost.Q-GaLore also matches LoRA, QLoRA, and GaLore in fine-tuning performance.Source: §1 · p. 3
  • Q-GaLore trains LLaMA-7B from scratch on a single RTX 4060 Ti with 16GB, at performance comparable to its full-rank counterpart.The introduction reports this as feasible pre-training on a single GPU.Source: §1 · p. 3

2 Related Work

Related work covers low-rank adaptation and training, alongside low-precision training. It describes memory-efficiency approaches while noting limits in performance, stability, and applicability to large-scale transformers.

2.1 Low-Rank Adaptation and Training

  • LoRA reduces memory by optimizing low-rank adapters that can later be merged into the original model, but its performance can be suboptimal versus full-parameter optimization.Other approaches reduce memory through fused backward operations or by partitioning parameters or selecting layers.Source: §2.1 · p. 3
  • ReLoRA still requires full-rank learning initially, while GaLore uses low-rank gradients for full-parameter learning with reduced memory use.The passage reports GaLore outperforming common low-rank adaptation methods such as LoRA.Source: §2.1 · p. 3

2.2 Low Precision Training

  • Low-precision training stores data in lower-precision formats, but large tensors challenge quantization and some methods retain high-precision latent weights.Training instability is another challenge; proposed responses include stochastic weight averaging, gradient scaling, and second-order optimizer statistics.Source: §2.2 · p. 3

3 Methodology

Q-GaLore combines low-precision quantization with low-rank gradients, adapting subspace updates to layerwise convergence. The methodology examines quantization, adaptive SVD updates, stochastic rounding, and the training pipeline.

3.1 Preliminaries on Quantization

  • Q-GaLore focuses on memory-efficient training without retaining high-precision parameters, unlike quantization-aware training approaches that keep them during training.The method retrains weights in INT8 while computing activations and gradients in BFloat16; block-wise quantization uses per-block scale and zero point.Source: §3.1 · p. 4

3.2 Layerwise Convergence Behaviors of Gradient Subspace

  • Gradient-subspace behavior differs by layer: some layers stabilize early, some within a training window, and others keep changing toward the end.Figure 2 tracks cosine similarity between adjacent projection matrices captured every 250 training iterations.Source: §3.2 · p. 4, Figure 2 · p. 4
  • Over 60% fewer SVD calls can closely mimic original GaLore performance by lengthening each layer’s update interval when recent projection similarities exceed a threshold.The interval doubles when cosine similarity across the previous k intervals remains above a threshold such as 40%.Source: §3.2 · p. 4

3.3 High Quantization Tolerance of Projection Matrix

  • 4-bit projection matrices retain pre-training quality with minimal impact and reduce optimizer-state memory cost by 25%.Figure 3 reports pre-training performance for LLaMA-130M models with projection matrices quantized to different bit widths.Source: Figure 3 · p. 4, §3.3 · p. 5

3.4 Approximating High-Precision Training Trajectories Using Stochastic Rounding

  • Stochastic rounding makes the expected quantized weight equal to the original weight, allowing low-precision parameters to implicitly accumulate small gradient information.The method is reported to achieve comparable performance without maintaining high-precision parameters.Source: §3.4 · p. 5

3.5 The Q-GaLore Algorithm

  • Q-GaLore projects full-rank gradients into a low-rank format for an 8-bit Adam optimizer, then projects optimizer states back to update weights.Gradients are kept in high precision; projection matrices are quantized to 4 bits, and stochastic rounding is used when updating low-precision weights.Source: §3.5 · pp. 5–6, Figure 4 · p. 5

4 Experiments

Across pre-training and fine-tuning experiments, Q-GaLore combines low-precision weights and projection matrices with low-rank gradients to reduce memory use. It reports competitive performance, including 7B pre-training within 16 GB, while adaptive subspace updates and stochastic rounding address SVD costs and low-precision training stability.

4.1 Implementation Details

  • Experiments use C4 for LLaMA pre-training from 60M to 7B parameters and downstream GLUE and MMLU tasks, with Full, low-rank, LoRA, ReLoRA, QLoRA, and GaLore as baselines.Downstream models include RoBERTa-base, LLaMA-3-8B, Gemma-7B, and Mistral-7B.Source: §4.1 · p. 6

4.2.1 Memory-Efficient Pre-training with Q-GaLore

  • Q-GaLore maintains comparable pre-training performance, with less than a 0.84 perplexity increase versus GaLore, while saving 29.68% memory at 1B parameters.The 1B result also reports 60.51% memory savings versus Full.Source: §4.2.1 · p. 7

4.2.2 Q-GaLore Facilitates 7B Model Pre-Training within 16GB Memory

  • Q-GaLore pre-trains LLaMA-7B with around 15GB memory overhead on a 16GB RTX 4060 Ti, with less than a 1-point perplexity difference from the baseline.The Q-GaLore run used a maximum learning rate of 0.004, versus 0.005 for the baseline.Source: §4.2.2 · p. 7, Table 2 · p. 7

4.2.3 Memory-Efficient Fine-Tuning with Q-GaLore

  • Q-GaLore achieves comparable or superior fine-tuning performance to Full and LoRA, and gains up to 5.19 over QLoRA at the same memory cost.The reported Gemma-7B MMLU improvement over LoRA is 1.02.Source: §4.2.3 · p. 8, Table 3 · p. 8, Table 4 · p. 8

4.3 End-to-End Memory Measurement

  • Only Q-GaLore trains LLaMA-7B within 16 GB in the end-to-end measurement, while its quantization and dequantization operations add 14.64% throughput overhead versus GaLore.INT8 weights halve weight memory, and projection quantization further reduces optimizer-state memory.Source: §4.3 · p. 9, Figure 5 · p. 9

4.4 Further Investigation and Ablation Study

  • Without stochastic rounding, perplexity rises by 7.86, 1.98, and 2.27 for 60M, 160M, and 350M models, respectively.During warm-up, vanilla rounding produces a perplexity gap of 18.67 to 47.02 versus stochastic rounding; Q-GaLore stays within 1 point of Full.Source: §4.4 · pp. 9–10, Figure 6 · p. 9
  • Using only 36.20% of GaLore’s SVD operations, Q-GaLore achieves comparable pre-training performance on LLaMA-130M and saves over 32 hours in the stated 7B scenario.The 7B estimate assumes 300 subspace updates across 150,000 iterations, with each SVD taking about 10 minutes on an RTX A6000.Source: Figure 7 · p. 9, §4.4 · p. 10

5 Conclusion

Q-GaLore combines low-rank gradients with INT8 model weights and INT4 projection matrices, using gradient-subspace dynamics and projection-matrix quantization resilience to reduce memory use and SVD operations. It achieves competitive pre-training and fine-tuning performance, including LLaMA-7B training on a single 16GB NVIDIA RTX 4060 Ti.

5 Conclusion

  • Q-GaLore facilitates training LLaMA-7B on a single NVIDIA RTX 4060 Ti with 16GB memory while achieving competitive pre-training and fine-tuning performance.The conclusion describes this as the first such 16GB single-GPU training result.Source: §5 · p. 10
  • Q-GaLore combines low-rank gradients with INT8 model weights and INT4 projection matrices to reduce memory use and SVD operations.Its design draws on layer-varying gradient-subspace behavior and projection matrices that function effectively under 4-bit quantization.Source: §5 · p. 10
  • The method’s experiments demonstrate competitive performance in both pre-training and fine-tuning.Source: §5 · p. 10

A More Implementation Details

The implementation gives PyTorch-style forward and backward operations for an INT8 linear layer. The forward pass reconstructs weights from quantized values, zero points, and scales; the backward pass computes input and weight gradients.

A More Implementation Details

Related papers

Loading 2407.08296v1…