PaperlayerView exact revision on arXiv

Source-linked AI summary

QeRL: Beyond Efficiency -- Quantization-enhanced Reinforcement Learning for LLMs

Wei Huang, Yi Ge, Shuai Yang, Yicheng Xiao, Huizi Mao, Yujun Lin, Hanrong Ye, Sifei Liu, Ka Chun Cheung, Hongxu Yin, Yao Lu, Xiaojuan Qi, Song Han, Yukang Chen

arXiv:2510.11696v1cs.LGcs.CLcs.CV

TL;DR

RL can improve LLM reasoning but requires substantial memory and costly rollouts. QeRL combines NVFP4 quantization and LoRA with adaptive quantization noise; it reports faster training and stronger results than LoRA baselines, while matching full-parameter fine-tuning performance. Its evaluations cover models up to 32B, leaving performance above 70B unestablished.

  • Problem

    RL supports adaptive reasoning in LLMs but is resource-intensive, with substantial memory requirements and costly rollouts.

    Source: §1 · pp. 1–2

  • Method

    QeRL combines NVFP4 quantization with LoRA and uses adaptive quantization noise to adjust exploration during training.

    Source: §1 · p. 2, §3.3 · p. 6

  • Results

    QeRL outperforms 16-bit LoRA and QLoRA while matching full-parameter fine-tuning accuracy, with approximately 1.8× end-to-end training speedup compared to QLoRA.

    Source: §1 · p. 3, §5 · p. 10

  • Takeaways & Limitations

    QeRL enables GRPO training of a 32B model on a single H100 80GB GPU while delivering faster rollout and end-to-end training.

    Source: §1 · p. 3

    Experiments cover models from 3B to 32B and do not establish whether performance is maintained above 70B parameters.

    Source: K LIMITATION ANALYSIS · p. 21

Abstract

from arXiv · show

We propose QeRL, a Quantization-enhanced Reinforcement Learning framework for large language models (LLMs). While RL is essential for LLMs' reasoning capabilities, it is resource-intensive, requiring substantial GPU memory and long rollout durations. QeRL addresses these issues by combining NVFP4 quantization with Low-Rank Adaptation (LoRA), accelerating rollout phase of RL while reducing memory overhead. Beyond efficiency, our findings show that quantization noise increases policy entropy, enhancing exploration, and enabling the discovery of better strategies during RL. To further optimize exploration, QeRL introduces an Adaptive Quantization Noise (AQN) mechanism, which dynamically adjusts noise during training. Experiments demonstrate that QeRL delivers over 1.5 times speedup in the rollout phase. Moreover, this is the first framework to enable RL training of a 32B LLM on a single H100 80GB GPU, while delivering overall speedups for RL training. It also achieves faster reward growth and higher final accuracy than 16-bit LoRA and QLoRA, while matching the performance of full-parameter fine-tuning on mathematical benchmarks such as GSM8K (90.8%) and MATH 500 (77.4%) in the 7B model. These results establish QeRL as an efficient and effective framework for RL training in LLMs.

1 INTRODUCTION

RL can support more adaptive reasoning than imitation-based fine-tuning, but its memory demands and costly rollouts make LLM training resource-intensive. QeRL combines NVFP4 quantization with LoRA and adaptive quantization noise to accelerate training while using quantization noise to encourage exploration.

1 INTRODUCTION

  • RL is resource-intensive because policy and reference models require substantial GPU memory, while repeated sampling of long sequences makes rollouts costly.Multistage processing and RL's sample inefficiency add further costs.Source: §1 · p. 2
  • LoRA reduces trainable parameters but leaves slow rollouts unresolved, while QLoRA's NF4 requires unpacking and lookup-table mapping and slows rollouts by 1.5–2×.Quantized rollout models can also require simultaneous 8-bit and 16-bit models for importance-sampling corrections, increasing memory use.Source: §1 · p. 2
  • Precisely controlled quantization noise increases policy entropy, encouraging exploration and helping models discover better strategies during RL.The paper contrasts this finding with quantization results in supervised fine-tuning.Source: §1 · p. 2, Figure 3 · p. 3
  • QeRL uses NVFP4-quantized weights with LoRA and introduces adaptive quantization noise to address static noise during training.Its Marlin-based approach accelerates rollout and prefilling, with gradient backpropagation through LoRA layers.Source: §1 · pp. 2–3
  • QeRL achieves approximately 1.8× end-to-end training speedup compared to QLoRA and scores 90.8 on GSM8K, while enabling 32B-model GRPO training on a single H100 80GB GPU.The introduction also reports matching full fine-tuning accuracy on MATH 500.Source: §1 · p. 3

2 PRELIMINARY

The preliminary section introduces weight quantization and LoRA as approaches for representing model weights and reducing the cost of adaptation. It distinguishes quantization formats and describes LoRA's low-rank trainable updates.

2 PRELIMINARY

  • Integer quantization maps floating-point weights in [Wmin, Wmax] to an integer range determined by the target bit-width.The paper defines quantization for a weight tensor W ∈ R^d×k.Source: §2 · p. 3
  • NVFP4 uses an FP8 (E4M3) scaling factor for blocks of 16 elements, whereas MXFP4 uses a shared FP8 (E8M0) factor for blocks of 32.The passage also identifies NF4 as designed for normally distributed weights.Source: §2 · p. 3
  • LoRA models weight updates with a low-rank decomposition because updates in large pretrained models often lie in a low-dimensional subspace.The formulation freezes the original weight matrix and optimizes only the low-rank matrices A and B, reducing trainable parameters and memory and computation costs.Source: §2 · p. 3

3 METHOD

QeRL applies quantized, LoRA-based RL to LLMs and finds that quantization can promote exploration through increased policy entropy. It adds adaptive quantization noise to vary exploration during training.

3.1 TRAINING FRAMEWORK OF QERL

  • GRPO samples multiple outputs per query, scores them with rule-based rewards, and uses group-relative advantages; DAPO removes the KL penalty to avoid an upper limit on exploration.The section places QeRL on GRPO and DAPO policy-optimization algorithms.Source: §3.1 · p. 4

3.2 QUANTIZATION ENCOURAGES EXPLORATION

  • On Qwen2.5-7B-Instruct, 4-bit PEFT RL converges faster and achieves higher adjusted evaluation scores than 16-bit counterparts; NVFP4 ultimately reaches better final rewards than MXFP4.NVFP4 and MXFP4 both show better reward growth than NF4.Source: §3.2 · p. 4, Figure 4 · p. 5
  • Quantization raises sampling entropy by perturbing logits, flattening token probabilities and encouraging exploration across plausible next actions.The paper describes this noise as an implicit exploration mechanism that can help models discover better strategies.Source: §3.2 · pp. 4–5
  • Static deterministic quantization noise does not adapt to RL’s changing exploration–exploitation trade-off or adjust exploration at critical training phases.This limitation motivates adding dynamic control to the noise.Source: §3.2 · p. 5

3.3 ADAPTIVE QUANTIZATION NOISE IN PARAMETER SPACE

  • AQN adds a channel-wise Gaussian noise vector to each quantized linear layer, resamples it each forward pass, and exponentially reduces its scale during training.The schedule begins with inherent quantization noise and aims to balance exploration and exploitation.Source: §3.3 · p. 6
  • To avoid separate noise parameters and preserve kernel compatibility, QeRL merges noise into LayerNorm parameters, including those associated with Wq, Wk, Wv, Wgate, and Wup.Wq, Wk, and Wv share one RMSNorm, while Wgate and Wup share another.Source: §3.3 · p. 6

4 EXPERIMENT

The experiments evaluate QeRL on mathematical reasoning tasks across model sizes, then examine its accuracy, reward growth, and efficiency against LoRA and quantized baselines. They also test adaptive noise schedules and LoRA ranks.

4.1 EXPERIMENT SETTINGS

  • Experiments use Qwen2.5 models and evaluate mathematical reasoning with GSM8K, BigMath, MATH500, AIME 2024/2025, and AMC 23.Training uses GRPO or DAPO; speedup tests run on one H100 GPU, while final evaluated models use eight H100 GPUs.Source: §4.1 · p. 7

4.2 EXPERIMENT RESULTS

  • Across GSM8K and BigMath, QeRL matches or exceeds 16-bit LoRA and improves the 14B AMC 23 score to 57.5 versus 55.0 for full-parameter training.The experiments train models from 3B to 32B and report that QeRL trains about 1% of full-training parameters while using 40%–50% of vanilla LoRA GPU memory.Source: §4.2 · p. 8, Table 2 · p. 8
  • QeRL’s reward rises within 200 steps on BigMath, while vanilla LoRA takes over 500 steps to show improvement.The comparison covers 7B and 14B models and reports faster reward growth and higher reward targets for QeRL.Source: §4.2 · p. 8
  • For the 3B model, exponential noise decay yields more stable later improvements than linear, cosine, or logarithmic decay.The schedulers’ early-stage performance differences are negligible.Source: Figure 9 · p. 8, §4.2 · p. 9
  • Periodic additional noise injection with AQN produces steadier reward growth and further reward improvement near convergence.The comparison is against using default quantized noise throughout training, which limits exploration.Source: Figure 8 · p. 7, §4.2 · p. 9
  • LoRA ranks 16, 32, 64, and 128 show similar reward trends and growth rates, with rank 16 converging slightly faster.The authors identify rank 16 as the more economical choice.Source: Figure 10 · p. 8, §4.2 · p. 9

4.3 MEMORY SAVING AND SPEEDUP

  • On 7B and 14B models, QeRL reaches 1.2×–1.5× end-to-end training speedups over vanilla LoRA, while QLoRA runs at 0.7×–0.9×.The reported table shows QeRL model sizes of 5.9 GB and 10.6 GB, compared with 15.2 GB and 29.6 GB for BF16 LoRA.Source: §4.3 · p. 9, §4.2 · p. 9, Table 3
  • QeRL achieves over 2× rollout speedups on 14B and 32B models across LoRA ranks.The figure’s setting uses batch size 1.Source: §4.3 · p. 9, Figure 11 · p. 9

5 CONCLUSION

QeRL combines NVFP4 quantization with LoRA for efficient LLM reinforcement learning, and the paper reports improved accuracy, about 1.5× end-to-end speedup, and reduced memory use. The study uses established open-source research datasets and reports human review of all model-edited sentences.

5 CONCLUSION

  • QeRL integrates NVFP4 quantization with LoRA and uses adaptive quantization noise to improve RL stability.The conclusion reports higher accuracy than 16-bit LoRA and QLoRA, while approaching full-parameter fine-tuning performance.Source: §5 · p. 10

A ETHICS STATEMENT

  • The study uses previously established, publicly available datasets solely for research, with no newly generated text, video, or audio materials.The datasets are not used for commercial applications.Source: A ETHICS STATEMENT · p. 15

C USE OF LARGE LANGUAGE MODELS

  • GPT-5 was used only to refine manuscript language, and every model-edited sentence underwent human review.The authors state that technical content, methods, and interpretations were independently written and verified.Source: C USE OF LARGE LANGUAGE MODELS · p. 15

D RELATED WORK

Related work covers reinforcement learning for LLM reasoning, quantization for model compression, and parameter-efficient fine-tuning. These lines of work provide context for QeRL’s combination of quantization and LoRA-based RL.

Reinforcement Learning for LLMs

  • Recent RL work advances LLM reasoning through algorithms and systems including GRPO, DeepSeek-R1, DAPO, GSPO, and HybridFlow.The cited work addresses mathematical reasoning, training stability, sampling, and RLHF system design.Source: D RELATED WORK · p. 15

Quantization for LLMs

  • Quantization reduces parameter precision to compress LLMs, with PTQ, QAT, and low-bit formats including NF4, FP4, MXFP4, and NVFP4.The passage describes NVFP4 as a 4-bit floating-point format introduced with NVIDIA’s Blackwell architecture.Source: D RELATED WORK · p. 15

Efficient Fine-tuning

  • Efficient fine-tuning methods reduce adaptation costs through approaches such as LoRA, DoRA, QLoRA, and prompt or prefix tuning.Tina is cited as demonstrating reasoning gains from RL with LoRA.Source: D RELATED WORK · p. 16

E EXPERIMENT HYPERPARAMETERS

The experiments use Qwen2.5 models, mathematical reasoning datasets, and GRPO or DAPO, with training settings varying by model and dataset. The deployment procedure divides training into stages and adjusts noise for adaptive exploration.

E EXPERIMENT HYPERPARAMETERS

  • Training used Qwen2.5-3B-Instruct and Qwen2.5-7B-Instruct on GSM8K with GRPO, and Qwen2.5-7B-Instruct, Qwen2.5-14B-Instruct, and Qwen2.5-32B-Instruct on BigMath with DAPO; data for 7B and 14B used grades 3–5, while 32B used grades 4–5.Source: E EXPERIMENT HYPERPARAMETERS · p. 16

F DEPLOYMENT OF QERL

  • QeRL deployment divides training steps into K stages, initializes added noise after stage 0, and adjusts its scale through the training schedule.The algorithm samples prompts, generates outputs, computes rewards and group-relative advantages, then updates the policy with the GRPO objective.Source: F DEPLOYMENT OF QERL · pp. 16–17

G PROOF OF NOISE SHARING

QeRL’s noise-sharing operation makes additive noise equivalent to row-wise multiplicative noise on weight matrices. Sharing noise within attention and feed-forward blocks avoids disrupting NVFP4 and BF16 multiplication kernels or adding matrix multiplications.

G PROOF OF NOISE SHARING

  • QeRL’s noise-sharing mechanism applies equivalent row-wise multiplicative Gaussian noise to weight matrices while sharing noise across Q, K, and V and across feed-forward up and down layers.The noise is incorporated through RMSNorm inputs; the mechanism avoids disrupting NVFP4 and BF16 multiplication kernels or adding matrix multiplications.Source: G PROOF OF NOISE SHARING · p. 17
  • Because multiplicative noise can be more sensitive in deep networks, QeRL initializes its noise standard deviation to 1e-2, below the typical 1e-1 used in traditional noise-based networks.Source: G PROOF OF NOISE SHARING · p. 17

H ADDITIONAL EXPERIMENTS OF TRAINING

QeRL shows faster reward growth than 16-bit LoRA in the reported 7B BigMath training and consistently better reward performance in the 32B comparison. Its 14B model also maintains higher entropy than LoRA during RL, particularly early in training.

Training Rewards of Different Model

More Experiments of Entropy

I ADDITIONAL ABLATION STUDY

The learning-rate ablation reports that QeRL remains stable at a larger learning rate where 16-bit training often collapses. At a smaller rate, QeRL only marginally outperforms LoRA, with reward close to 0.95.

Ablation of Learning Rate

J MORE EFFICIENCY EXPERIMENTS

NVFP4 accelerates rollout and end-to-end RL training in the reported benchmarks, with rollout speedups reaching 2.0× for the 32B model. Across LoRA ranks, NVFP4 remains faster than 16-bit models, though rollout speed declines as rank increases.

J MORE EFFICIENCY EXPERIMENTS

K LIMITATION ANALYSIS

QeRL outperforms 16-bit vanilla LoRA and matches 16-bit full-parameter RL accuracy, with over 2× training speedup relative to vanilla LoRA and QLoRA. The experiments cover 3B–32B models and reasoning benchmarks, leaving performance beyond 70B and on other task types unestablished.

K LIMITATION ANALYSIS

  • Over 2× training speedup relative to both vanilla LoRA and QLoRA accompanies accuracy matching 16-bit full-parameter reinforcement fine-tuning.QeRL also achieves superior performance in RL training compared to 16-bit vanilla LoRA training.Source: K LIMITATION ANALYSIS · p. 21
  • Experiments span 3B–32B models but do not establish whether QeRL maintains performance for models exceeding 70B parameters.The authors identify larger models as future work.Source: K LIMITATION ANALYSIS · p. 21
  • Evaluations cover reasoning benchmarks but not code, other data types, or general-purpose language tasks unrelated to reasoning.The authors note that QeRL can be adapted to richer and more diverse training datasets.Source: K LIMITATION ANALYSIS · p. 21
Loading 2510.11696v1…