PaperlayerView exact revision on arXiv

Source-linked AI summary

Score Centering Stabilizes Off-policy Reinforcement Learning

Martin Marek, Max Ryabinin

arXiv:2609.20807v1cs.LG

TL;DR

RL for LLMs is sensitive to small sampler–trainer differences, or TIM, while eliminating those differences would reduce rollout efficiency. The paper identifies drift as the main instability mechanism and cancels it with additive score centering. Across quantized models from 0.6B to 30B parameters, score centering matches or outperforms importance-sampling methods, and their composition performs best under severe staleness.

  • Problem

    RL training of LLMs can become unstable when sampler and trainer outputs differ because TIM is difficult to eliminate without sacrificing rollout efficiency.

  • Method

    The paper derives score centering, an additive correction that subtracts the sampler’s expected score to cancel drift under TIM.

  • Results

    Across Qwen3 models from 0.6B to 30B parameters, score centering matches or outperforms importance-sampling methods under quantization, while composition with them performs best under severe staleness.

  • Takeaways & Limitations

    Drift is sufficient to explain the instability under severe quantization, and additive score centering provides a practical complement to importance sampling.

  • Takeaways & Limitations

    Score centering leaves a sampler-based covariance mismatch and the headline results use deliberately severe mismatch on short sequences as a proxy for longer training under milder mismatch.

Abstract

from arXiv · show

Reinforcement learning (RL) of large language models is notoriously sensitive to small differences between training and inference engines, often referred to as the training-inference mismatch (TIM). However, completely eliminating TIM is impractical, as it would come at a major cost to rollout efficiency. In this paper, we show that the instability of RL under TIM is primarily caused by drift: a persistent bias between training and inference engines that accumulates with every training step. We derive an additive "score centering" correction term that stabilizes RL under TIM by canceling drift. When training models from 0.6B to 30B parameters, score centering alone matches or outperforms methods based on importance sampling under quantization, with the gap growing as the mismatch becomes more severe. Because the correction is additive, score centering also composes with importance sampling -- their composition outperforms pure importance-sampling baselines in our staleness experiments.

1 INTRODUCTION

RL for LLMs uses policy-gradient updates based on rewarded rollouts, but separate sampler and trainer forward passes create TIM that can degrade training or cause reward collapse. This paper analyzes that instability and introduces score centering, which matches or outperforms importance-sampling stabilization methods in the reported experiments.

  • RL policy-gradient training samples rollouts, assigns rewards, and computes gradients from reward-weighted scores.
  • Separate sampler and trainer forward passes create training-inference mismatch, which can degrade training or cause complete reward collapse.
  • The paper identifies a drift term that vanishes exactly when training and sampling policies match, then introduces score centering to stabilize RL under TIM.
  • Across Qwen3-0.6B Countdown and Qwen3-30B-A3B-Base INTELLECT-2 experiments, score centering matches or outperforms importance-sampling stabilization methods and can combine with them for further gains.

2 BACKGROUND AND RELATED WORK

TIM arises because rollouts are generated by a sampler that can differ from the trainer through numerical, implementation, or staleness effects. Importance sampling can correct this mismatch exactly in principle, but practical ratio control trades variance for bias, motivating score centering as a fundamentally additive alternative.

  • TIM occurs when rollouts come from sampler qθ while gradients are computed through a different trainer pθ.
  • Numerical differences, independent implementations, floating-point non-associativity, and asynchronous stale checkpoints create TIM with varying severity.
  • The practical objective is stable RL under TIM while preserving the high hardware utilization enabled by disaggregated training and inference.
  • Importance sampling corrects TIM by weighting scores with r = pθ(y)/qθ(y), but raw ratios can become arbitrarily large and inflate gradient-estimate variance.
  • Bounding importance ratios stabilizes practice but inevitably introduces bias, while score centering uses no ratios, masking, or clipping and instead applies an additive correction.

3 WHY RL IS SENSITIVE TO MISMATCH

The paper separates offline reward instability from online instability under TIM: offline training favors non-negative rewards, whereas online training is least stable with them. This points to the moving sampler in online training as the key source of mismatch sensitivity.

  • Policy gradient reduces to offline SFT with binary +1/0 advantages, so reward mode and staleness are the two key differences examined between SFT and RL.
  • The controlled experiment perturbs Qwen3-1.7B sampler weights during Countdown training to create intentional policy mismatch.
  • Figure 2 shows offline training is stable only with +1/0 rewards, while online training under TIM is least stable with +1/0 rewards.
  • Offline negative-reward instability is attributed to unbounded negative logprobs and distribution sharpening, but the paper does not investigate it further.
  • Online +1/0 training behaves like distillation from a biased sampler whose weights are refreshed after every step, unlike offline distillation from a fixed teacher.

4 SCORE CENTERING

Under training-inference mismatch, the expected policy-gradient update contains drift toward the sampler; score centering subtracts the sampler’s expected score to cancel that drift, including off-policy.

  • 4 SCORE CENTERING: Drift from training-inference mismatch distills the trainer toward the sampler and compounds through synchronization, making online all-positive-reward training especially unstable.The drift depends on mean reward rather than which rollouts succeeded, while the covariance term carries the learning signal.
  • 4 SCORE CENTERING: Score centering subtracts the sampler’s expected score from each token score, forcing the centered score to have zero mean at every prefix even under mismatch.This cancels drift exactly because both expectations are taken under the sampler distribution.
  • 4 SCORE CENTERING: With score centering, the expected update becomes a covariance between rewards and centered scores, differing from the on-policy update only through the sampling distribution.The covariance identity separates the drift term from the reward-dependent signal term.
  • 4 SCORE CENTERING: Importance sampling also cancels drift but can raise variance and reintroduce drift when large ratios are clipped or masked, whereas score centering is deterministic and additive.Because the corrections act independently, score centering can be composed with importance sampling.
  • 4 SCORE CENTERING: Score centering differs from classical reward baselines because it is a bias correction that shifts the update mean, not merely a variance-reduction device.Its expected score is exactly computable in LLM RL as a sum over next-token logprobs, avoiding a critic.
  • 4 SCORE CENTERING: The practical implementation logs only top-k sampler logprobs and models the remaining tail with the trainer distribution; k = 128 and k = 32 matched full centering in tested settings.The method is implemented as a scalar loss with stop-gradient support and has a generalized importance-weighted version.

5 EXPERIMENTS

Experiments compare correction methods under controlled and realistic training-inference mismatches, finding that score centering is consistently strong and that its composition with importance sampling is especially effective under staleness.

  • 5.1 SETUP: The experiments deliberately amplify mismatch through sampler weight noise, quantization, and staleness because mild mismatch requires longer runs for drift to accumulate and separate methods.The setup compares Qwen3-0.6B-Instruct on Countdown and Qwen3-30B-A3B-Base on INTELLECT-2 math using shared training components and logged sampler probabilities.
  • 5.2 SYNTHETIC WEIGHT NOISE: Under the largest synthetic weight noise, only score centering alone or combined with TIS/MIS trains stably, while other methods collapse earlier as mismatch increases.Across three noise scales, collapse occurs earlier under larger mismatch; DPPO collapses around steps 160, 80, and 20, while TIS collapses around steps 180 and 40 at the two larger noise scales.
  • 5.3 QUANTIZATION AND STALENESS: Score centering alone or combined with TIS/MIS performs best under quantization, while the combinations dominate under severe staleness.The comparison uses a shared REINFORCE objective with group-centered rewards and applies each correction method in isolation or composition.
  • 5.3 QUANTIZATION AND STALENESS: Under staleness, composing score centering with importance sampling is attributed to correcting residual drift after TIS partially corrects the sampling distribution.The paper recommends composition when the trainer can move far from the sampler between synchronization steps.
  • 5.4 SCALING TO 30B: At 30B scale, score centering remains stable under severe sampler quantization, reaching 52% with FP4 KV cache and 30% with an INT4 KV cache.Under FP4 KV cache, MIS collapses late while score centering and TIS remain stable; under INT4 KV cache, TIS reaches 12% and every other method ends below 5%.

6 CONCLUSION

The paper attributes TIM instability to drift toward a periodically refreshed sampler and introduces additive score centering to cancel it. The method supports top-k approximation and composition with importance sampling, while severe staleness and proxy mismatch remain important boundaries.

  • 6 CONCLUSION: Under TIM, drift acts as distillation toward the sampler, compounds through periodic synchronization, and causes earlier collapse as mismatch severity increases.The drift term depends on mean reward rather than rollout success, while the covariance term carries the learning signal.
  • 6 CONCLUSION: Score centering cancels drift by subtracting the expected score as an additive, hyperparameter-free correction expressible through a scalar loss.Its generalized form centers weighted scores when composed with importance sampling.
  • 6 CONCLUSION: Under mild TIM, score centering matches importance sampling; under severe quantization, it is the only method that trains stably, while composition with TIS or MIS performs best under severe staleness.The correction cancels constant-reward drift for any importance-weighting function, and vanilla importance sampling makes the centering term vanish.
  • 6 CONCLUSION: The headline results use deliberately severe mismatch on short sequences as a proxy for long training under milder mismatch, and residual sampler-versus-trainer covariance mismatch matters under severe staleness.The authors therefore report best performance there when score centering is composed with importance sampling.
  • 6 CONCLUSION: Top-k score centering reconstructs the sampler tail from trainer probabilities and computes the correction using only head tokens rather than the full vocabulary.The tail is rescaled by the sampler-to-trainer tail-mass ratio, while the expected tail score is recovered from the zero full-vocabulary expected score.
  • 6 CONCLUSION: Top-k score centering has negligible measured overhead: k = 128 runs finish within 1% of baseline wall-clock time on both tested model sizes.The implementation uses detached sampler probabilities and supports TIS or MIS weighting functions.

A.4 TOP-k ABLATION

Top-k score centering matches full-vocabulary score centering across all tested experimental settings, including the most distorted quantized setting.

  • A.4 TOP-k ABLATION: k = 32 and k = 128 match full score centering in every tested setting, including INT8 W/A + INT4 KV at 30B.With k = 128, the top-k head covers more than 99.9% of sampler mass on average in other settings, 99.45% on average in the hardest setting, and 95.8% in its worst batch.

B.1 TRAINING SETUP

Experiments use REINFORCE with group-centered rewards, SGD, fixed learning rate 10^-2, and batched rollouts with one optimizer step per batch.

  • B.1 TRAINING SETUP: The experiments use REINFORCE with group-centered rewards, where each rollout advantage is centered by the mean reward for its prompt group.The group mean is computed over completions for the same prompt.
  • B.1 TRAINING SETUP: SGD uses a fixed learning rate of 10^-2 across experiments; it was stable and performed comparably to AdamW while saving up to 240GB of memory.Each batch samples eight completions per prompt and takes a single optimizer step.

B.2 CORRECTION METHODS

The correction-method comparison organizes importance-sampling baselines by where corrections apply, how ratios are clipped or masked, and which advantage signs are corrected.

  • B.2 CORRECTION METHODS: Importance-sampling methods vary by token- versus sequence-level correction, ratio clipping or masking, and which advantage signs receive correction.DPPO uses a binary-total-variation masking region but still optimizes with importance-sampling weights.
  • B.2 CORRECTION METHODS: Top-k score-centering ablations compare k = 32 and k = 128 against full-vocabulary score centering, with the number of seeds reported in the figure legend.The ablation visual is Figure 5.
  • B.2 CORRECTION METHODS: All corrections are applied to the same objective, with score centering implemented using the top-k approximation at k = 128.PG is the uncorrected baseline, while other methods use default parameters from their original papers or verl without further tuning.

B.3 COMPUTE RESOURCES

The experiments use varying numbers of random seeds and report approximate compute requirements for reproducing the figures, primarily on 8× NVIDIA H100 SXM GPU nodes.

  • The 0.6B and 1.7B models use 3 seeds across every experiment and correction method in Figures 2–4.
  • The 30B model uses fewer seeds for some experiments because its training cost is substantially higher.
  • Table 2 summarizes seeds per curve and approximate compute required to reproduce the figures on primarily 8× NVIDIA H100 SXM GPU nodes.
Loading 2609.20807v1…