PaperlayerView exact revision on arXiv

Source-linked AI summary

The Mirage of Optimizing Training Policies: Monotonic Inference Policies as the Real Objective for LLM Reinforcement Learning

Jing Liang, Hongyao Tang, Yi Ma, Yancheng He, Weixun Wang, Xiaoyang Li, Ju Huang, Wenbo Su, Jinyi Liu, Yan Zheng, Jianye Hao, Bo Zheng

arXiv:2606.29526v1cs.LG

TL;DR

Training-inference mismatch means updates that improve the training policy may not improve the inference policy used for deployment. The paper proposes MIPI and MIPU, which improve average reasoning performance and training stability under high mismatch across two model scales.

  • Problem

    Training-inference mismatch creates objective misalignment because training-policy updates may differ from improvements to the inference policy used for deployment.

  • Method

    MIPU implements MIPI through sampler-referenced candidate construction and inference-gap-aware acceptance of synchronized policy updates.

  • Results

    Under FP8-quantized rollout across Qwen3-4B and Qwen3-1.7B, MIPU achieves the best average performance and more stable training dynamics.

  • Takeaways & Limitations

    The results suggest treating training-inference mismatch as an objective-level policy-improvement issue rather than only a low-level system discrepancy.

  • Takeaways & Limitations

    Experiments are limited to moderate-scale models, leaving performance on larger models and more diverse RL training systems for future study.

Abstract

from arXiv · show

Reinforcement learning (RL) has gained growing attention in large language model (LLM) post-training, yet RL training remains fragile and can suffer from instability or collapse. One vital cause is training-inference mismatch: LLM adopts separate inference and training engines for generation efficiency and training precision, which in practice exhibits inconsistent probabilities for the same trajectories on training and inference sides, even with synchronized model parameters. This naturally induces a special type of off-policyness ever existing and poisoning the training. Prior works have made various efforts in addressing the off-policyness to stabilize the training policies under the mismatch. In this paper, we point out the objective misalignment neglected by existing works that an effective update to the policy in the training engine not necessarily ensures the improvement of the inference policy, i.e., the one used in deployment. To this end, we propose a new policy optimization objective for LLM RL, named Monotonic Inference Policy Improvement (MIPI). Following this principle, we introduce Monotonic Inference Policy Update (MIPU), a two-step LLM RL framework that constructs sampler-referenced candidate updates and selectively accepts synchronized candidates using an inference-side gap proxy. Experiments conducted on two model scales under high mismatch show that MIPU improves average reasoning performance and training stability.

1 Introduction

LLM RL separates inference and training engines, creating persistent probability discrepancies that make training-side policy improvement an unreliable proxy for inference-policy improvement. The paper proposes MIPI and its two-step realization, MIPU, which targets monotonic inference-policy improvement and improves performance and stability under high mismatch.

  • Introduction: LLM RL separates rollout generation from gradient computation across inference and training engines, producing discrepancies even when model parameters are synchronized.Inference engines sample responses, while training engines compute log-probabilities and updates.
  • Introduction: Training-side optimization does not guarantee that synchronizing an update produces a better inference policy, revealing an objective-level misalignment.The deployment policy is induced by the inference engine, so updates should not be judged solely by training-policy improvement.
  • Introduction: MIPI establishes monotonic improvement of the inference policy as the LLM RL optimization objective, and MIPU realizes it through two update steps.MIPU constructs sampler-referenced candidate updates and accepts synchronized candidates using an inference-gap-aware criterion.
  • Introduction: Across Qwen3-4B and Qwen3-1.7B under FP8-quantized rollout, MIPU achieves the best average performance and more stable training dynamics.Analysis attributes complementary roles to Step 1 candidate improvement and Step 2 filtering of unreliable synchronized candidates using an inference-side criterion.

2 Preliminaries

This section formalizes LLM reinforcement learning as policy optimization over token-generation MDPs and reviews monotonic policy improvement with PPO-style surrogates. It then models training-inference mismatch by distinguishing the training policy from the inference policy used for rollout and deployment.

  • Reinforcement Learning for LLMs: LLM reinforcement learning models sequential token generation as an MDP, with states containing the prompt and generated prefix and actions selecting the next token.The MDP is defined as M = (S, A, P, R, γ).
  • Reinforcement Learning for LLMs: The LLM is the parameterized policy πθ(at | st), and RL seeks π∗= arg maxπθ J(πθ), where J(π) = Eτ∼π [R(τ)].The policy induces distributions over output sequences, state-action pairs, and states.
  • Monotonic Policy Improvement and Proximal Policy Gradient: Monotonic policy improvement seeks an updated policy πk+1 whose expected performance does not decrease, expressed as J(πk+1) −J(πk) ≥0.Vanilla policy-gradient optimization can be ineffective because performance is sensitive to step size.
  • Monotonic Policy Improvement and Proximal Policy Gradient: Because the exact objective depends on the updated policy’s visitation distribution, TRPO uses an old-policy local surrogate and KL trust region, while PPO uses a clipped probability-ratio objective.GRPO is described as building on PPO.
  • Training-Inference Mismatch in LLM RL: Training-inference mismatch is modeled with distinct policies: πk represents the training engine, whereas µk generates rollouts and is used for deployment despite synchronized parameters.Modern LLM RL pipelines may therefore produce different action distributions across the two engines.

3 Related Works

LLM reinforcement learning has become a prominent post-training paradigm, particularly for reasoning, coding, and long-horizon tasks. Related work includes RLVR-based scalable training recipes and methods addressing training-inference mismatch through sampler-informed updates.

  • RL is increasingly used for LLM post-training, especially to improve reasoning, coding, and long-horizon tasking.
  • RLVR obtains outcome rewards from automatically checkable signals, with GRPO, DAPO, and GSPO developing scalable recipes for mathematical and reasoning tasks.
  • Training-inference mismatch in LLM RL: Training-inference mismatch is recognized as an important source of instability in modern LLM RL systems.
  • Training-inference mismatch in LLM RL: Sampler-informed approaches include TIS, which applies a clipped training-to-sampler probability-ratio correction, and MIS, which filters tokens or sequences with extreme mismatch signals.

4 The New Objective: Monotonic Policy Improvement on the Inference Side

Under training-inference mismatch, improving the training policy may harm the synchronized inference policy, so MIPI targets inference-policy improvement directly. MIPU realizes this objective by constructing sampler-referenced candidates and accepting them only when an inference-gap proxy passes a tolerance test.

  • Motivation: Training-side improvement does not necessarily improve the inference policy when training and inference policies differ.The synchronized inference policy is the one ultimately used for deployment, creating an objective-level misalignment beyond ratio and advantage mismatches.
  • MIPI objective: MIPI defines inference-policy improvement J(µk+1) − J(µk) as the target quantity under training-inference mismatch.Its decomposition separates candidate construction from post-synchronization verification.
  • MIPU Step 1: MIPU constructs a candidate training policy using a sampler-referenced surrogate for J(πk+1) − J(µk).Step 1 aligns the probability ratio and advantage statistic with the rollout sampler, truncates pre-update mismatch correction, and clips only the current-update ratio.
  • MIPU Step 2: MIPU accepts a synchronized candidate when bTpost ≥ −c and otherwise rolls back both trainer and inference states to the previous checkpoint.The tolerance c ≥ 0 permits mild negative proxy values while rejecting candidates with a clear negative signal.
  • MIPU Step 2: After synchronization, MIPU evaluates the post-update inference gap Tpost = J(µk+1) − J(πk+1) using a stable proxy motivated by the reverse identity.A negative gap indicates that inference underperforms the training-side candidate after synchronization.

5 Experiments

Experiments under FP8-quantized rollout show that MIPU improves average reasoning performance and training stability across two model scales. Ablations and rollback controls indicate that sampler-referenced candidate construction and inference-gap-aware acceptance are complementary, with stability depending on calibrated gap-based decisions rather than simply rejecting more updates.

  • Experimental setup: Experiments train Qwen3-1.7B and Qwen3-4B with FP8-quantized rollout and evaluate pass@1 accuracy across five mathematical reasoning benchmarks.AIME24 and AMC23 use avg@16 to reduce evaluation variance; inference-training K3-KL is reported as a mismatch diagnostic.
  • Main results: 66.71% on Qwen3-4B and 53.97% on Qwen3-1.7B are MIPU’s best average scores under FP8-quantized rollout.MIPU’s strongest listed benchmark results are 85.00% on AMC23 and 45.96% on Minerva for Qwen3-4B, and 86.52% on MATH-500 and 59.52% on OlympiadBench for Qwen3-1.7B.
  • Main results: MIPU maintains a stable score trajectory, whereas GRPO, MIS, and LR-decay can reach intermediate peaks before degrading during continued training.The distinction reflects sustained training behavior rather than only the best checkpoint reached during an unstable run.
  • Ablation study: The full Step 1+Step 2 framework outperforms the ablated variants because Step 1 improves candidate updates while Step 2 filters unreliable synchronized candidates using the post-update inference gap.Step 1 constructs sampler-referenced candidates; Step 2 determines whether the synchronized inference policy accepts them.
  • Acceptance-signal analysis: Qwen3-1.7B exhibits larger mismatch and more volatile bTpost than Qwen3-4B, supporting bTpost as an inference-side signal for acceptance decisions.The post-update gap proxy reflects whether the synchronized inference policy remains consistent after the candidate update.
  • Acceptance-signal analysis: Step 2 remains stable despite random rollback rejecting more updates, showing that its benefit comes from conditioning acceptance on bTpost rather than generic update sparsification.Random rollback still collapses after a transient peak, while Step 2 maintains a stable score trajectory and requires calibrated acceptance rather than stricter rejection.

6 Conclusion

The paper reframes training-inference mismatch as an objective-level problem: standard updates optimize the training policy, while deployment depends on the inference policy. It formulates MIPI and realizes it through MIPU, while noting limitations in scale and implementation generality.

  • Training-inference mismatch changes the object of policy improvement because standard RL updates optimize the training policy, whereas deployment depends on the inference policy.
  • MIPU realizes MIPI through sampler-referenced candidate construction and inference-gap-aware candidate acceptance.
  • Limitations: Experiments are limited to moderate-scale models because of computational constraints, leaving larger models and more diverse RL systems for future study.
  • Limitations: Step 2 is a flexible design space, with validation-based post-update gap proxying presented as one effective deployment-side instantiation for evaluating synchronized candidates.

A Implement Details · A.1 Hyperparmeters

The implementation details identify the major hyperparameter choices and present them in Table 3, titled “Hyperparameter setups.”

  • A.1 Hyperparmeters: The paper highlights major hyperparameter choices as part of its implementation details.
  • A.1 Hyperparmeters: Table 3 is the designated location for the paper’s major hyperparameter choices.
  • A.1 Hyperparmeters: The hyperparameter subsection reports implementation-level setup information.
  • A.1 Hyperparmeters: The passage directs readers to Table 3 for the major hyperparameter selections.
  • A.1 Hyperparmeters: No individual hyperparameter values are provided in the supplied passage text.
  • A.1 Hyperparmeters: Table 3 is explicitly labeled “Hyperparameter setups.”

A.2 Implementation Details

The implementation uses dual-clipped GRPO as the baseline and extends it with sampler-referenced candidate updates, inference-side acceptance, and model-specific tolerances. Additional baselines include untruncated importance sampling, mismatch filtering, and learning-rate decay.

  • Baseline: The baseline uses vanilla GRPO with dual-clipped policy loss and ϵ = 0.2, maximizing the average objective over sampled responses.The rollout, reward, and optimization settings match those of other methods.
  • Ours: Step 1 applies token-level truncated training-to-inference importance weights with wmax = 2 after computing the dual-clipped GRPO objective.Vanilla-IS differs by using the untruncated weight instead.
  • Ours: Step 2 synchronizes each candidate to inference, estimates bTpost on a validation batch, and accepts it when bTpost ≥ −ct using a dynamically scheduled tolerance.The larger early tolerance addresses initially low bTpost values under FP8 rollout and avoids over-rejecting useful candidates.
  • Ours: For Qwen3-1.7B, cstart = 4 × 10−3 and cend = 1 × 10−3; for Qwen3-4B, cstart = 1 × 10−3 and cend = 0, with tolerance fixed after 100 steps.The tolerance choices are further analyzed in Appendix C.
  • LR-decay: LR-decay halves the learning rate every decay interval until η∞, using Tdecay = 249 for Qwen3-1.7B and Tdecay = 234 for Qwen3-4B under FP8 rollout.The model parameters are updated using the resulting ηt.

B Analysis of Step 1: Sampler-Referenced Update

Step 1 decomposes sampler-referenced correction from the current training update because directly clipping πθ/µk entangles pre-update mismatch with the trust-region constraint. TIS truncates the mismatch weight, controlling variance while providing the best stability-performance trade-off among Step 1 variants.

  • B Analysis of Step 1: Sampler-Referenced Update: PPO-IS directly clips the trainer-to-sampler ratio, but that ratio mixes pre-update mismatch between πk and µk with the current update from πk to πθ.Because the ratio can deviate from 1 before the current update, clipping becomes overly active and less informative.
  • B Analysis of Step 1: Sampler-Referenced Update: The decomposed formulation separates pre-update mismatch correction from the current training-side update, centering the trust-region constraint on πθ/πk.This avoids making the constraint depend directly on preexisting sampler-training mismatch.
  • B Analysis of Step 1: Sampler-Referenced Update: Vanilla-IS preserves the current-update constraint but leaves its mismatch-correction weight πk/µk unbounded, amplifying gradient variance for low-probability rollout tokens.It recovers a clearer upward learning trend than PPO-IS but still exhibits larger gradient-norm spikes.
  • B Analysis of Step 1: Sampler-Referenced Update: TIS truncates the mismatch-correction weight while preserving decomposition, yielding the best stability-performance trade-off and the Step 1 candidate update used in the full method.It avoids PPO-IS’s overly aggressive clipping and Vanilla-IS’s variance amplification from large mismatch weights.

C Analysis of the Tolerance Paremeter c · D Licenses

The acceptance tolerance c trades off early candidate acceptance against later protection: overly strict rules can slow learning, while MIPU’s dynamic tolerance calibrates this balance. The paper also lists licenses for its datasets, models, and software assets.

  • C Analysis of the Tolerance Paremeter c: c > 0 permits mild negative proxy values, c = 0 enforces proxy non-regression, and c < 0 requires a positive margin.All settings use the boundary bTpost ≥−c.
  • C Analysis of the Tolerance Paremeter c: Stricter acceptance can reject many early candidates because bTpost is initially low under FP8-quantized rollout, slowing learning and lowering the performance plateau.The stricter setting enters persistent rollback after around 280 steps.
  • C Analysis of the Tolerance Paremeter c: Rollback rate is not itself the objective, because excessive rejection can prevent the inference policy from reaching a better region.Rejecting more updates may appear safer but can impede useful policy movement.
  • C Analysis of the Tolerance Paremeter c: MIPU starts with a larger tolerance for useful early candidates and gradually tightens acceptance as the post-update gap stabilizes.This balances early learning efficiency with later-stage protection.
  • D Licenses: AIME24, OlympiadBench, and DAPO-Math-17k use the Apache-2.0 License, while Minerva Math has No License.These are listed as used assets in the paper.
  • D Licenses: DeepMath-103k uses the MIT License, while Qwen3 Models, ROLL, vLLM, and Megatron use the Apache-2.0 License.These licenses are listed for the paper’s used assets.
Loading 2606.29526v1…