PaperlayerView exact revision on arXiv

Source-linked AI summary

Defeating the Training-Inference Mismatch via FP16

Penghui Qi, Zichen Liu, Xiangxin Zhou, Tianyu Pang, Chao Du, Wee Sun Lee, Min Lin

arXiv:2510.26788v1cs.LGcs.AIcs.CL

TL;DR

RL fine-tuning of LLMs is unstable because training and inference policies can diverge, creating a deployment gap and biased optimization. This paper attributes the mismatch to numerical precision and shows that switching from BF16 to FP16 virtually eliminates it, improving training stability, convergence, and final performance.

  • Problem

    RL fine-tuning is sensitive to instability because the training policy π(·|θ) differs from the inference policy µ(·|θ), creating a deployment gap.

  • Method

    The paper switches RL fine-tuning from BF16 to higher-precision FP16 to reduce rounding discrepancies between training and inference computations.

  • Results

    FP16 virtually eliminates the training-inference mismatch and produces more stable training, faster convergence, and superior performance.

  • Takeaways & Limitations

    FP16 should be reconsidered as a foundational option for robust RL fine-tuning of LLMs.

  • Takeaways & Limitations

    The mismatch remains difficult to reconcile because inference is autoregressive while training is parallel, with parallelization and precision-sensitive operations further complicating alignment.

Abstract

from arXiv · show

Reinforcement learning (RL) fine-tuning of large language models (LLMs) often suffers from instability due to the numerical mismatch between the training and inference policies. While prior work has attempted to mitigate this issue through algorithmic corrections or engineering alignments, we show that its root cause lies in the floating point precision itself. The widely adopted BF16, despite its large dynamic range, introduces large rounding errors that breaks the consistency between training and inference. In this work, we demonstrate that simply reverting to \textbf{FP16} effectively eliminates this mismatch. The change is simple, fully supported by modern frameworks with only a few lines of code change, and requires no modification to the model architecture or learning algorithm. Our results suggest that using FP16 uniformly yields more stable optimization, faster convergence, and stronger performance across diverse tasks, algorithms and frameworks. We hope these findings motivate a broader reconsideration of precision trade-offs in RL fine-tuning.

1 Introduction

RL fine-tuning is unstable partly because training and inference engines produce numerically different policies. The paper argues that BF16 precision causes this mismatch and that switching to FP16 can virtually eliminate it.

  • Motivation: RL fine-tuning is sensitive to hyperparameters and can suffer training collapse, complicating reliable performance improvements.The instability motivates methods that stabilize and streamline RL fine-tuning.
  • Training-Inference Mismatch: Different training and inference engines create numerically different outputs despite being mathematically identical.Precision errors and hardware-specific optimizations contribute to the training-inference mismatch.
  • Prior Solutions: Importance-sampling corrections can prolong or stabilize training but introduce bias or additional computational cost.The cited approaches add an extra forward pass, increasing training cost by approximately 25%.
  • Prior Solutions: Algorithmic corrections optimize against the training engine’s distribution and therefore leave a deployment gap for the inference engine.The resulting parameters may not be optimal for deployment and can cause a tangible performance drop.
  • Root Cause: BF16’s low precision makes it susceptible to accumulating rounding errors that cause training and inference policies to diverge.Although BF16 has a wide dynamic range, its precision is identified as the mismatch’s root cause.
  • Proposed Solution: Switching from BF16 to FP16 during RL fine-tuning virtually eliminates the mismatch without changing the model architecture or learning algorithm.The paper reports improved performance and stability while removing complex algorithmic workarounds and closing the deployment gap.

2 Background

Modern LLM RL systems sample with an inference policy while optimizing a training policy, creating biased gradients and a deployment gap. Existing importance-sampling and engineering fixes address parts of the problem but add variance, overhead, or implementation complexity.

  • Policy Mismatch: Separate inference and training engines create a numerical mismatch between policies µ and π, although they should be mathematically identical.The mismatch arises from subtle computational discrepancies between optimized systems.
  • Biased Gradient: Sampling responses from µ while optimizing π makes the policy-gradient estimator biased when the mismatch is ignored.The prompt x is sampled from pX, responses y receive reward R(x, y), and the gradient uses REINFORCE.
  • Deployment Gap: The deployment gap occurs because π is trained while µ is used for deployment and evaluation, so training-optimized parameters may not be inference-optimal.This mismatch produces a non-trivial performance degradation that algorithmic patches cannot close.
  • Importance Sampling: Sequence-level importance sampling preserves an unbiased gradient in principle, using a probability ratio and an advantage with a variance-reduction baseline.The advantage is A(x, y) = R(x, y) − B(x), where B(x) is the baseline.
  • Importance Sampling: Long response sequences can produce extreme importance ratios, motivating truncated or masked estimators that trade bias for lower variance.Clipping and masking control importance-weight magnitudes to stabilize training.
  • Existing Implementations: GRPO-based corrections are implemented as auxiliary patches rather than strictly principled importance-weighted estimators.Many frameworks are GRPO-centric and do not natively provide the standard estimators.
  • Existing Implementations: Existing GRPO corrections require an additional forward pass, adding approximately 25% computational overhead during training.The extra pass computes π(·|θ′) for off-policy correction, assuming a backward pass costs twice a forward pass.
  • Engineering Attempts: Engineering alignment efforts remain difficult because inference is autoregressive while training is parallel and precision-sensitive operations complicate reconciliation.These differences make engineering workarounds complex and brittle.

3 Revisiting FP16 Precision

This section revisits FP16 and BF16 as competing 16-bit formats, arguing that FP16’s greater precision addresses training–inference mismatch in RL fine-tuning. Offline analysis shows FP16 reduces token- and sequence-level discrepancies, while loss scaling addresses its limited range.

  • Format trade-offs: FP16 allocates 10 mantissa bits and 5 exponent bits, whereas BF16 allocates 7 mantissa bits and 8 exponent bits, trading precision for dynamic range.BF16’s range matches FP32, while FP16 more accurately represents nearby values but is more susceptible to overflow and underflow.
  • Making FP16 practical: Loss scaling multiplies gradients by S before backpropagation and divides them by S before updates, preserving small gradients within FP16’s representable range.Dynamic loss scaling increases S when no overflow is detected and decreases it when overflow occurs.
  • Why mismatch occurs: BF16’s low precision can produce rounding errors that accumulate across different training and inference implementations, causing π and µ to diverge.Autoregressive accumulation can make small kernel- or hardware-specific discrepancies significant at the sequence level.
  • Why FP16 helps: FP16’s 10 mantissa bits provide 8 times more precision than BF16’s 7 mantissa bits, making training and inference outputs more likely to remain numerically aligned.The added precision acts as a buffer against implementation differences and accumulated rounding errors.
  • Offline analysis: FP16 notably reduces token-level mismatch, while sequence-level mismatch under FP16 is approximately 24× smaller than under BF16.The sequence-level BF16 mismatch grows with response length because of cumulative autoregressive errors.

4 A Sanity Test for RL Algorithms

The paper introduces a perfectible-dataset sanity test to distinguish RL algorithm reliability from problem difficulty. In experiments across two frameworks, BF16 methods frequently collapse, while sequence-level correction is stable but costly.

  • Dataset construction: The sanity test filters out trivial and unsolvable problems, retaining questions whose initial accuracy lies between 20% and 80%.The resulting MATH dataset contains 1,460 questions for DeepSeek-R1-Distill-Qwen-1.5B.
  • Sanity criterion: An RL algorithm passes when training accuracy on the perfectible dataset converges above a high threshold, such as 95%.Passing is not a guarantee of universal success, but failing strongly indicates an ill-suited algorithm design.
  • Experimental setup: The evaluation uses DeepSeek-R1-Distill-Qwen-1.5B across VeRL and Oat, with 8 NVIDIA A100 80G GPUs, batches of 64 questions, 8 rollouts per question, and 4 gradient steps.The experiments compare vanilla GRPO, token-level TIS, sequence-level MIS, standard importance-weighted policy gradients, and GSPO.
  • BF16 baselines: Vanilla GRPO collapses after reaching 73% accuracy in VeRL and 84% in Oat, while token-level TIS collapses after reaching 82% and 88%, respectively.These results illustrate instability under the tested BF16 setting.
  • Algorithmic correction: Sequence-level MIS is the only BF16 algorithmic correction that maintains stable training without collapse, but it converges slowly because its sequence-level importance ratio has high variance.The passage also reports a significant deployment gap at peak performance.

AIME 2025

Across algorithms and precision combinations, FP16 reduces training–inference mismatch and improves stability, convergence, and evaluation performance. Using FP16 for both training and inference provides the strongest stability–efficiency trade-off, while algorithmic performance differences largely disappear.

  • AIME 2025: FP16 training runs are significantly more stable, converge faster, and achieve higher final rewards and evaluation scores across tested algorithms.The comparison frames precision-level alignment as more direct and effective than unstable or inefficient algorithmic corrections.
  • AIME 2025: FP16 makes sequence-level importance sampling ratios more concentrated and stable, enabling the classic unbiased policy-gradient estimator without modifications.Under FP16, this simple estimator outperforms existing algorithmic corrections evaluated in BF16.
  • AIME 2025: Algorithms that later collapse show a growing training–inference mismatch beforehand, making the mismatch a potential early-warning signal.The associated policy probabilities can move toward opposite extremes despite using the same weight copy.
  • AIME 2025: FP16 reduces framework-dependent mismatch differences, although VeRL remains more prone to occasional numerical instability than Oat.The initial policy difference is near -0.9 in Oat versus -1.0 in VeRL.
  • AIME 2025: Under FP16, performance differences between RL algorithms become almost indistinguishable, with only mixed evidence for original GRPO across AIME 2024 and AIME 2025.The authors attribute this convergence to a nearly on-policy optimization setting; original GRPO is slightly lower on AIME 2024 but slightly higher on AIME 2025.
  • AIME 2025: FP32 inference stabilizes BF16 training but is nearly three times slower than FP16 or BF16 inference, making that combination impractical for large-scale experiments.Increasing inference precision under BF16 consistently prolongs stability and improves performance before reaching this cost.
  • AIME 2025: FP16 for both training and inference yields the lowest mismatch, the most stable dynamics, and nearly 100% training accuracy without sacrificing speed.This combination reaches nearly 100% training accuracy on the perfectible dataset.

5 Generalization Across Models, Data, and Training Regimes

FP16 generalizes across model architectures, adaptation regimes, model scales, and RL settings. Across these scenarios, it consistently improves stability, training accuracy, rewards, or convergence relative to BF16.

  • 5 Generalization Across Models, Data, and Training Regimes: The broader evaluation moves beyond sanity checks to MoE RL, LoRA RL, larger prompt sets, and alternative model settings.This extension tests whether the precision findings persist across diverse training regimes and model configurations.
  • 5 Generalization Across Models, Data, and Training Regimes: MoE RL presents an especially difficult mismatch setting because training and inference use distinct parallelization strategies and precision-sensitive top-k expert selection.These factors typically produce larger training–inference mismatch than in dense models.
  • 5 Generalization Across Models, Data, and Training Regimes: FP16 improves stability, training accuracy, and validation rewards consistently across three evaluated RL algorithms in MoE reinforcement learning.The experiments report this pattern across the three algorithms tested in the MoE setting.
  • 5 Generalization Across Models, Data, and Training Regimes: In LoRA RL, BF16 training collapses after roughly 600 steps, whereas FP16 maintains stable training throughout.The experiment uses Qwen2.5-Math-1.5B models on MATH with GRPO-Token-TIS.
  • 5 Generalization Across Models, Data, and Training Regimes: On Qwen3-14B-Base with DAPO, FP16 rewards increase much faster and achieve higher AIME 2024 validation accuracy than BF16.The result supports FP16 as a way to mitigate mismatch in larger dense models.
  • 5 Generalization Across Models, Data, and Training Regimes: The study extends beyond Qwen-based models to OctoThinker-3B, examining whether base-model differences affect RL learning dynamics and numerical behavior.The passage identifies base policies as influencing exploration and the numerical range and sensitivity of parameters and activations.

6 Discussions

The discussion argues that FP16 changes the precision tradeoff for RL fine-tuning: it reduces the BF16 mismatch-related bias-variance tension while improving training stability, convergence, and final performance. The paper also states that FP16 is not universally optimal because its limited range can create engineering challenges for extremely large models.

  • Rethinking the Precision Tradeoff in RL Fine-Tuning: FP16 trades BF16’s wide dynamic range for higher precision, yielding more stable RL training, faster convergence, and superior final performance.The discussion presents this as a reason to reconsider BF16 as the default precision for RL fine-tuning.
  • Rethinking the Precision Tradeoff in RL Fine-Tuning: FP16 is not claimed to be universally optimal, and extremely large models may face overflow-related engineering challenges because of its limited range.The paper notes that lower precisions such as FP8 may remain attractive for efficiency.
  • The Bias-Variance Tradeoff under BF16 Precision: BF16 creates a bias-variance trade-off: lower-bias corrections stabilize training but high variance slows convergence, while lower-variance methods can eventually collapse.GRPO, token-level TIS, and GSPO initially converge quickly but become unstable; PG-Seq-IS and GRPO-Seq-MIS are more stable but converge slowly.
  • The Bias-Variance Tradeoff under BF16 Precision: Under FP16, reducing the training-inference mismatch lowers both mismatch-induced bias and importance-sampling variance.This makes the stability-speed tension substantially less critical.
  • The Bias-Variance Tradeoff under BF16 Precision: FP16 enables even the naive policy-gradient estimator to converge efficiently, with all tested algorithms performing well.The paper attributes this training dynamic to the reduced mismatch under FP16 precision.

7 Conclusion

The conclusion identifies numerical precision as the fundamental source of training-inference mismatch in RL fine-tuning and presents FP16 as a simple alternative to algorithmic corrections. Switching from BF16 to FP16 is reported to virtually eliminate the mismatch while improving stability, convergence, and performance.

  • 7 Conclusion: The training-inference mismatch is fundamentally a numerical-precision problem and a major source of instability in RL fine-tuning.This reframes the problem from one requiring compensation to one addressable at the precision level.
  • 7 Conclusion: Switching from BF16 to higher-precision FP16 can virtually eliminate the training-inference mismatch.The conclusion characterizes this as a single efficient change rather than a complex algorithmic fix.
  • 7 Conclusion: FP16 yields more stable training, faster convergence, and superior performance.The conclusion states that these benefits support reconsidering FP16 as a foundational option for robust RL fine-tuning of LLMs.

A Detailed Experimental Settings

The MoE RL experiments use Qwen3-30B-A3B-Base with DAPO-Math-17k training data and evaluate online on AIME 2024 using avg@32 in VeRL. The setup also corrects VeRL’s implementation of Dr.GRPO’s normalization through a named loss aggregation mode.

  • A Detailed Experimental Settings: MoE RL experiments use Qwen3-30B-A3B-Base, DAPO-Math-17k training data, AIME 2024 online evaluation with avg@32, and the VeRL framework.The key hyperparameters are summarized in Table 3.
  • A Detailed Experimental Settings: The corrected VeRL implementation of Dr.GRPO uses the loss aggregation mode “seq-mean-token-sum-norm” instead of the incorrect open-source implementation.Dr.GRPO uses a constant normalizer rather than a token-count-based normalizer.

A.2 RL on Large Dense Models

Experiments on large dense models use Qwen2.5-14B-Base, a 54.4k-sample mathematical training collection, and AIME 2024 online evaluation with avg@8. Table 3 contains the hyperparameters for the large dense model setup.

  • A.2 RL on Large Dense Models: Large dense model experiments use Qwen2.5-14B-Base and a deduplicated, filtered collection of 54.4k mathematical training samples.The collection aggregates reasoning data from OR1, DAPO, and DeepScaler.
  • A.2 RL on Large Dense Models: The experiments evaluate online on AIME 2024 using the avg@8 metric.The training algorithms and hyperparameters follow the setup described in Yu et al. [2025].
  • A.2 RL on Large Dense Models: Table 3 presents hyperparameters for RL training of MoE models and large dense models.

B More Experimental Results

Figure 6 evaluates BF16- and FP16-trained checkpoints across varied frameworks, algorithms, datasets, and training regimes. FP16-trained models generalize to unseen benchmarks, supporting the paper’s claim.

  • Figure 6 compares evaluation results for BF16- and FP16-trained checkpoints across frameworks, algorithms, datasets, and training regimes.
  • FP16-trained models generalize well to unseen benchmarks, further supporting the paper’s claim.
Loading 2510.26788v1…