PaperlayerView exact revision on arXiv

Source-linked AI summary

QaRL: Rollout-Aligned Quantization-Aware RL for Fast and Stable Training under Training--Inference Mismatch

Hao Gu, Hao Wang, Jiacheng Liu, Lujun Li, Qiyuan Zhu, Bei Liu, Binxing Xu, Lei Wang, Xintong Yang, Sida Lin, Sirui Han, Yike Guo

arXiv:2604.07853v1cs.LGcs.AI

TL;DR

LLM RL is slowed by rollout generation, and quantized rollouts can widen the mismatch with full-precision learning updates. QaRL aligns learner-side training with quantized sampling and uses TBPO to control long-response error tokens. Experiments report performance comparable to BF16 and faster training.

  • Problem

    Rollout generation comprises roughly 70% of RL training time, while low-bit rollout sampling and full-precision updates create a training-inference mismatch.

    Source: §1 · p. 1

  • Method

    QaRL aligns learner-side quantization-aware training with low-bit rollouts, while TBPO uses sequence-level objectives and dual clipping for negative samples to control error-token updates.

    Source: §1 · p. 2

  • Results

    QaRL matches BF16 performance and outperforms quantized-rollout training across math, logic, and code; on Qwen3-30B-A3B-Base it scores 51.2 average math versus 52.1 for BF16 and 45.7 for quantized rollout, with 1.3× speedup over BF16.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The reported results show that rollout-aligned training and TBPO can stabilize quantized-rollout RL while retaining low-bit throughput benefits.

    Source: §1 · p. 2

Abstract

from arXiv · show

Large language model (LLM) reinforcement learning (RL) pipelines are often bottlenecked by rollout generation, making end-to-end training slow. Recent work mitigates this by running rollouts with quantization to accelerate decoding, which is the most expensive stage of the RL loop. However, these setups destabilize optimization by amplifying the training-inference gap: rollouts are operated at low precision, while learning updates are computed at full precision. To address this challenge, we propose QaRL (Rollout Alignment Quantization-Aware RL), which aligns training-side forward with the quantized rollout to minimize mismatch. We further identify a failure mode in quantized rollouts: long-form responses tend to produce repetitive, garbled tokens (error tokens). To mitigate these problems, we introduce TBPO (Trust-Band Policy Optimization), a sequence-level objective with dual clipping for negative samples, aimed at keeping updates within the trust region. On Qwen3-30B-A3B MoE for math problems, QaRL outperforms quantized-rollout training by +5.5 while improving stability and preserving low-bit throughput benefits.

1 Introduction

Quantizing rollouts can accelerate LLM reinforcement learning but widens the gap between the low-bit sampler and full-precision learner. QaRL aligns learner-side training with quantized rollouts, while TBPO addresses long-response error tokens; experiments report performance near BF16 with faster training.

1 Introduction

  • Rollouts account for roughly 70% of RL training time, but low-bit sampling with full-precision updates creates a training-inference mismatch.A standard RL step includes rollout generation, a probability-computing forward pass, and a policy-gradient update.Source: §1 · p. 1
  • QaRL combines low-bit rollouts with rollout-aligned learner-side quantization-aware training to reduce generation cost and align the learner’s policy with the sampling behavior.The learner-side forward pass is aligned to the quantized behavior used for sampling.Source: §1 · p. 2
  • Long generations can accumulate noise into repetitive, garbled off-trajectory tokens whose extreme policy ratios evade standard PPO-style clipping and destabilize training.TBPO applies sequence-level objectives and dual clipping for negative samples, dropping responses that exceed trust bands.Source: §1 · p. 2
  • 51.2 average math score on Qwen3-30B-A3B-Base is close to BF16’s 52.1, exceeds quantized rollout’s 45.7, and comes with a 1.3× training speedup over BF16.Across math, logic, and code benchmarks, QaRL matches BF16 training performance and outperforms quantized-rollout training.Source: §1 · p. 2

2 Preliminaries

The preliminaries establish GRPO’s group-relative policy-gradient objective, explain low-bit quantization, and describe how hybrid rollout and training systems create policy mismatch.

2.1 Group Relative Policy Optimization.

  • GRPO is a PPO-style policy-gradient objective that removes the critic and uses group-relative rewards as the baseline.It samples a group of responses for a query from the old policy and assigns each response a scalar verifiable reward.Source: §2.1 · p. 2

2.2 Quantization

  • Quantization maps full-precision weights to low-bit representations for integer or floating-point computation, with quantized GEMM rescaling its result.Quantization-aware training simulates quantization noise in the forward pass while retaining full-precision master weights; fully quantized training uses actual low-precision operations for forward and backward passes.Source: §2.2 · pp. 2–3

2.3 Training–Inference Mismatch in RL

  • Hybrid RL systems can produce different sampler and learner policies despite shared parameters, so PPO’s trust region may be enforced on the sampler rather than the learner.A mismatch reweighting factor corrects for system-level distribution shift, conceptually orthogonal to PPO’s proximal ratio.Source: §2.3 · p. 3

3 Methods

QaRL aligns learner-side computation with quantized rollouts to reduce training–inference mismatch. TBPO addresses remaining long-response errors with sequence-level objectives and dual clipping for negative samples.

3.1 Rollout-Aligned Quantization Aware Reinforcement Learning

  • QaRL aligns the learner with the quantized sampler by computing forward passes with low-bit GEMM and updating BF16 master weights through STE.On-the-fly quantization narrows the gap between quantized sampler and learner policies.Source: §3.1 · p. 4
  • Low-bit rollout inference accelerates decoding, and W4A16 can provide higher throughput than W8A8 for larger models, especially MoE.The implementation supports both configurations across model regimes.Source: §3.1 · p. 4
  • The learner publishes its already-materialized low-bit tensors to refresh the rollout engine after each optimization step, avoiding redundant re-quantization.This keeps sampler weights in the same low-bit format used by the learner’s forward pass.Source: §3.1 · p. 4

3.2 Trust Band Policy Optimization

  • Long-response quantization errors can produce repetitive, garbled tokens that propagate from corrupted autoregressive states.Token-level clipping may clip initial garbled tokens without clipping repetitive tokens induced later in the response.Source: §3.2 · pp. 5–6, Figure 6 · p. 6
  • Negative error tokens cluster in low-probability regions, where small absolute probability changes can produce extreme policy ratios.For example, a change from p_old = 0.02 to p_θ = 0.05 gives a ratio of 2.5.Source: §3.2 · p. 5
  • For negative-advantage samples, TBPO adds an upper as well as lower bound to limit extreme ratios that can dominate gradients.Positive-advantage samples retain the standard PPO-style upper bound.Source: §3.2 · pp. 5–6
  • Sequence-level objectives and dual clipping for negative samples form TBPO’s trust-region control.Entire responses can be excluded when their sequence-level ratio exceeds bounds; negative-advantage samples use a dual-sided proximal band.Source: §3.2 · p. 6

4 Experiments

The experiments compare QaRL-TBPO with BF16 and quantized-rollout training across math and out-of-distribution benchmarks, then ablate optimization objectives, quantization schemes, and training speed. QaRL-TBPO improves stability and retains performance close to BF16 while preserving speedups over BF16.

4 Experiments

  • The experiments use OpenR1-Math-46K and evaluate in-distribution math and out-of-distribution benchmarks, with GRPO or GSPO and the Muon optimizer.The models include Qwen2.5-Math-1.5B/7B, Qwen3-8B-Base, and Qwen3-30B-A3B-Base.Source: §4 · p. 6

4.1 Main Results

  • On Qwen3-8B, average in-distribution math performance is 48.9% with QaRL-TBPO, versus 51.8% with BF16 and 43.9% with quantized-rollout training.For Qwen3-30B-A3B MoE, TBPO reaches 51.2% average math score against 52.1% for the BF16 baseline; Figure 7 reports rewards nearly identical to BF16.Source: §4.1 · pp. 6–7, Figure 7 · p. 7, Table 1 · p. 7
  • QaRL-TBPO improves out-of-distribution performance over quantized-rollout training while matching BF16, with gains attributed to stable optimization and better generalization rather than overfitting.The supplied text does not report specific out-of-distribution benchmark values.Source: §4.1 · p. 7

4.2 Ablation

  • TBPO shows stable learning, high reward, and tight KL control, while alternative objectives exhibit KL drift, reduced data efficiency, lower reward ceilings, or error-token contamination.GSPO is unstable when negative samples contain error tokens; rejection sampling, discarding negatives, and single updates per batch each have stated trade-offs.Source: §4.2 · pp. 7–8
  • FP8W8A8, W4A16, and W4A8 show similar reward curves and nearly identical final rewards against the BF16 GRPO baseline.The paper uses W4A16 in most experiments for larger-model speed and broad hardware compatibility.Source: §4.2 · p. 8
  • On large-scale MoE models, QaRL achieves a 1.3× speedup over BF16, compared with 1.4× for quantized-rollout training.QaRL incurs modest overhead from in-training quantization.Source: §4.2 · p. 8

5 Related Work

The related work describes quantization approaches for LLMs and recent RL objectives for reasoning models. This paper focuses on stabilizing RL with quantized rollouts and quantization-aware training.

Quantization

  • Post-training quantization avoids retraining but can struggle with reasoning-model distribution shifts and error accumulation, while quantization-aware and fully quantized training incorporate quantization during training.The passage names GPTQ [Frantar et al., 2022], AWQ, LLM-QAT [Liu et al., 2024], and EfficientQAT [Chen et al., 2025].Source: §5 · p. 8

Reinforcement Learning in LLMs

  • Building on GRPO variants such as DAPO, GSPO, and ASPO, this work targets quantization-induced errors during generation under quantized rollouts and QAT.The passage characterizes DAPO by higher-bound clipping, GSPO by sequence-level importance sampling for MoE stability, and ASPO by asymmetric sampling for low-probability tokens.Source: §5 · p. 8

6 Conclusion and Limitations

QaRL addresses training–inference mismatch through rollout-aligned training, while TBPO targets error-token-driven collapse. The conclusion identifies fully quantized RL and more efficient token-level approximations as future directions.

6 Conclusion and Limitations

  • QaRL uses rollout-aligned training to mitigate quantized-rollout mismatch, and TBPO addresses error-token-driven collapse with sequence-level objectives and dual clipping on negative samples.The authors plan to explore fully quantized RL and token-level approximations intended to retain stability while improving efficiency and sample usage.Source: §6 · p. 8

Appdenix

The appendix states that a large language model was used only to edit and polish the manuscript.

A The Use of Large Language Models (LLMs)

B Experiment Setting

The experiments evaluate QaRL across math and out-of-distribution benchmarks, while examining batch-invariant kernels and quantization speed. On Qwen3-30B-A3B MoE, lower-bit schemes bring increasingly large efficiency gains from W8 to W4, leading the authors to use W4 by default.

B Experiment Setting

  • Experiments use Verl for training, vLLM for inference, and 8× NVIDIA H800 GPUs, with evaluation on in-distribution math and out-of-distribution benchmarks.The listed in-distribution datasets include AIME2024, AIME2025, AMC, Math-500, OlympiadBench, and Minerva; out-of-distribution datasets include ARC-Challenge, GPQA-Diamond, LiveCodeBench, and MMLU Pro.Source: B Experiment Setting · p. 11

C Batch invariant kernel

  • Batch-invariant kernels enforce a fixed floating-point summation order for bit-wise reproducibility, at a potential cost to GPU parallelism and performance.The passage attributes output inconsistency across kernel implementations to finite-precision accumulation and non-associativity of floating-point addition.Source: C Batch invariant kernel · p. 11

D.1 Speed on different quant scheme

  • Efficiency gains increase from W8 to W4 on Qwen3-30B-A3B MoE, and W4 is adopted as the default configuration.The comparison reports per-step training latency normalized to the BF16 baseline.Source: §D.1 · p. 12

D.2 More Ablation on TBPO

TBPO is compared with GRPO under BF16 and W4A16 quantized-rollout training. It performs on par with GRPO under BF16 and has a 2.9-point average margin under W4A16.

D.2 More Ablation on TBPO

  • Under W4A16 quantized-rollout training, TBPO delivers an average margin of 2.9 points over GRPO, while remaining on par with GRPO under BF16.The ablation evaluates TBPO against GRPO across the two precision settings.Source: §D.2 · p. 12, Table 3 · p. 12

D.3 Entropy of quantized RL training

Across quantized-rollout training and QAT/QaRL, the reported entropy shows no discernible elevation relative to BF16. A late entropy increase under W4A16 quantized-rollout training is associated with over-optimization of error tokens and repetitive generations.

D.3 Entropy of quantized RL training

  • Quantized-rollout training and QAT/QaRL show no discernible entropy elevation relative to the BF16 baseline during optimization.The authors describe quantization discrepancies as progressively assimilated across optimization.Source: §D.3 · p. 13
  • Entropy rises in the latter stages of W4A16 quantized-rollout training, unlike the reported pattern across the other training paradigms.The passage attributes this late-stage escalation to over-optimization of error tokens.Source: §D.3 · p. 13
  • Over-optimization of error tokens is reported to precipitate repetitive generation patterns alongside the late-stage entropy escalation.Source: §D.3 · p. 13

D.4 Comparsion of soft weighted methods SAPO

SAPO adaptively reweights tokens but remains unable to surpass TBPO in quantized RL training; the authors attribute this to SAPO continuing to assimilate globally off-distribution error-token responses. TBPO instead uses sequence-level objectives and weights to constrain optimization.

D.4 Comparsion of soft weighted methods SAPO

  • SAPO remains unable to surpass TBPO in quantized RL training, with the authors attributing its result to continued assimilation of globally off-distribution error-token responses.SAPO attenuates individual token contributions, but the passage says such sequences remain unsuitable for stable policy learning.Source: §D.4 · p. 13

E Detail of TBPO

  • TBPO defines sequence-level proximity ratios and mismatch weights, with the geometric mean of token-level ratios used to avoid vanishing or exploding gradients in long contexts.The proximity ratio measures drift from the previous learner, while the mismatch weight accounts for distributional shift between the historical sampling policy and the current optimization baseline.Source: E Detail of TBPO · p. 13, Figure 13 · p. 14
  • In the illustrated comparison, TBPO’s sequence-level approach clips garbled tokens without erroneously clipping exploration tokens, unlike GRPO’s token-level approach.The table caption identifies clipped tokens and relative weighting magnitudes through shading and color.Source: Table 6 · p. 14
Loading 2604.07853v1…