PaperlayerView exact revision on arXiv

Source-linked AI summary

VESPO: Variational Sequence-Level Soft Policy Optimization for Stable Off-Policy LLM Training

Guobin Shen, Chenxiao Zhao, Xiang Cheng, Lei Huang, Xing Yu

arXiv:2602.10693v3cs.LGcs.AI

TL;DR

Off-policy LLM training suffers from high-variance importance weights, especially for long autoregressive sequences. VESPO derives a principled sequence-level reshaping kernel with explicit variance control, achieving stable training and 61.8% accuracy at 64× staleness while improving across dense and MoE models.

  • Problem

    Off-policy LLM reinforcement learning needs principled importance-weight reshaping because autoregressive sequence weights can exhibit exponentially growing variance with length.

  • Method

    VESPO formulates importance-weight reshaping as a variance-constrained variational measure change, deriving a closed-form kernel applied directly to sequence-level weights without length normalization.

  • Results

    61.8% accuracy at 64× staleness; VESPO achieves the best average accuracy at every tested staleness ratio and remains stable across dense and MoE models.

  • Takeaways & Limitations

    VESPO provides stable off-policy training under severe staleness and transfers to code generation without retuning in the reported verifiable-reward settings.

Abstract

from arXiv · show

Off-policy updates are inevitable in reinforcement learning (RL) for large language models (LLMs) due to rollout staleness from asynchronous training and mismatches between training and inference engines. Naive importance sampling gives an unbiased correction but suffers from high variance, which is amplified by unbounded ratios and autoregressive generation. Prior remedies either rely on scenario-specific engineering, or trade bias for variance via token-level clipping or sequence-level normalization, yet these approaches remain largely heuristic. We propose Variational sEquence-level Soft Policy Optimization (VESPO). By explicitly incorporating variance reduction into a variational formulation, we derive a principled closed-form reshaping kernel that operates directly on sequence-level importance weights, avoids token-level approximation and length normalization, and admits an explicit variance bound for the deployed kernel. Experiments on math reasoning and code generation show that VESPO maintains stable training under severe off-policy conditions (staleness up to 64x) and delivers consistent gains across both dense and Mixture-of-Experts (MoE) models, outperforming recent reshaping baselines under matched setup. Code is available at https://github.com/FloyedShen/VESPO.

1 Introduction

Off-policy updates arise naturally in LLM reinforcement-learning pipelines, while token-level importance-weight transformations compromise sequence-level correction to control variance. VESPO addresses this through a variance-constrained variational formulation that yields a principled sequence-level reshaping kernel with bounded gradient contributions under arbitrary staleness.

  • Motivation: Off-policy updates arise from sequential mini-batch updates, asynchronous rollout-training decoupling, and training-inference mismatches, especially in MoE routing.These factors make later rollout batches stale relative to the evolving policy.
  • Prior methods: Token-level methods such as GRPO and SAPO transform importance weights to avoid the multiplicative variance explosion of sequence-level weights.GRPO applies PPO-style clipping to per-token ratios, while SAPO uses heuristic transformations.
  • VESPO: VESPO incorporates variance reduction into a variational formulation, yielding a principled closed-form solution for importance-weight reshaping.The formulation recasts reshaping as a measure change to an implicit proposal distribution under a variance constraint.
  • VESPO: VESPO operates directly on sequence-level importance weights without length normalization, preserving inter-token dependencies.The method is designed for sequence-level optimization rather than token-level approximation.
  • VESPO: VESPO admits an explicit variance bound that ensures bounded gradient contributions under arbitrary staleness.This property targets stable off-policy optimization despite stale rollouts.

2 Preliminaries

This section formulates off-policy LLM policy gradients through sequence-level importance weights and general reshaping functions. It explains why autoregressive sequence-level importance sampling has severe variance and why existing token- and sequence-level remedies remain heuristic or biased.

  • Notation: Responses are sampled from a behavior policy µ, while τ = (x, y) denotes a query-response pair receiving sequence-level reward R(τ).
  • Policy Gradient with Off-Policy Correction: Off-policy policy gradients weight each sample by W(τ) = πθ(τ)/µ(τ), and modifying this weight defines a reshaping function ϕ(W).The resulting gradient is Eτ∼µ[ϕ(W(τ)) · R(τ) · ∇θ log πθ(τ)].
  • The Variance Challenge of Sequence-Level IS: Sequence-level importance weights multiply token-level ratios, whereas the log-policy gradient sums across tokens, causing compounded global weighting over sequence length.Even small per-token deviations accumulate multiplicatively across T positions.
  • The Variance Challenge of Sequence-Level IS: The variance of W grows exponentially with T, making naive sequence-level importance sampling impractical for long sequences.
  • Existing Reshaping Methods: GRPO clips token-level updates as a first-order approximation, while GSPO uses length-normalized geometric-mean weights that introduce length-dependent bias.GSPO can assign identical weights to sequences with identical per-token statistics but different lengths despite different true importance weights.

3 VESPO: Variational Sequence-level Soft Policy Optimization

VESPO formulates sequence-level importance-weight reshaping as a variational proposal-design problem, yielding a smooth power-exponential kernel with explicit variance control. The method is bounded, differentiable, computationally practical, and supports asymmetric treatment of positive and negative advantages.

  • Variational formulation: Any sequence-level reshaping function implicitly defines a proposal distribution Q, enabling VESPO to derive the reshaping kernel by specifying desirable proposal properties.The proposal should remain close to the sampling distribution, incorporate the target policy, and support efficient optimization.
  • Kernel derivation: The derived kernel combines a power term W^α with exponential soft suppression exp(−λW), providing smooth differentiability instead of hard-clipping discontinuities.In practice, VESPO uses ϕ(W) = W^c1 exp(c2(1−W)) and shifts the kernel so that ϕ(1) = 1.
  • Surrogate objective: The surrogate objective JVESPO(θ) = Eµ[f(W(θ))A(τ)] saturates as W →∞, while the reshaping kernel peaks near W=1 and decays smoothly.This smooth saturation limits the influence of extreme importance weights in the deployed surrogate.
  • Variance guarantees: K ≈2.46 for (c1, c2) = (2, 3), while K = 1 for (3, 2); finite variance control requires c1 ≥1, whereas c1 < 1 makes the bound vacuous.Under EQ[W] ≤ C, the bound translates control of the first moment under Q into an explicit second-moment guarantee under µ.
  • Stability and implementation: VESPO is uniformly bounded for any c1, c2 > 0, so no single stale sample can create an unbounded gradient contribution; it also adds no forward passes or memory beyond GRPO/GSPO.The implementation computes log W from masked per-token log-probability sums and applies the reshaping in log-space to avoid overflow.

4 Experiments

Experiments evaluate VESPO on mathematical reasoning and code generation under policy staleness, train–inference mismatch, and alternative reshaping methods. VESPO remains stable across severe off-policy conditions, outperforms recent baselines, transfers across reward modalities, and benefits from sequence-level design choices without length normalization.

  • Experimental setup: VESPO is evaluated on math reasoning across three model scales and on code generation, using verifier and execution rewards respectively.Math uses DAPO-Math with AIME 2024/2025, AMC 2023, and MATH-500; code uses PRIME-RL/Eurus-2-RL-Data with HumanEval+, MBPP+, and LiveCodeBench v6.
  • Policy staleness: Across staleness ratios N=4–64, VESPO’s training curves are nearly identical and converge to similar final rewards around 0.7.GRPO saturates early, GSPO degrades as N grows, and SAPO becomes unstable; GSPO also catastrophically collapses around step 1,200 at N=4.
  • Reshaping baselines: VESPO substantially outperforms TOPR, CISPO, and BAPO, scoring +10 on AIME25 over BAPO and +28 over TOPR/CISPO.The reported failure modes include CISPO collapse, BAPO entropy explosion, and TOPR slow convergence with suppressed response length.
  • Cross-domain transfer: VESPO is best on HumanEval+, MBPP+, and LiveCodeBench v6, showing transfer of its variational framework from verifier-based math to execution-based code.Under train–inference mismatch, VESPO without engineering fixes matches GRPO+R2, while VESPO+R2 attains the highest reward and best AIME25.
  • Ablations: Removing length normalization keeps VESPO stable, whereas VESPOlin collapses around step 350 and VESPOsqrt exhibits periodic gradient spikes.The asymmetric hyperparameters c+ = (2, 3) and c−= (3, 2) balance suppression of negative advantages with learning from positive samples.

5 Related Work

Prior LLM policy-gradient methods stabilize off-policy updates through heuristic clipping or normalization, while importance-weight reshaping methods differ by granularity, boundary, and theoretical origin. Token-level reshaping has a sequence-level composition limitation, motivating sequence-level alternatives.

  • Policy Gradient Methods for LLMs: PPO uses a clipped surrogate objective, while GRPO, GSPO, and DAPO apply reward normalization, token-level clipping, sequence-level length normalization, or decoupled clipping with dynamic sampling.These approaches control variance through heuristic clipping or normalization.
  • Importance Weight Reshaping: Importance-weight reshaping methods can be organized by granularity, boundary, and theoretical origin.The passage distinguishes token-level hard-boundary, token-level smooth-gating, and sequence-level approaches.
  • Importance Weight Reshaping: Per-token reshaping does not compose into importance sampling under any single sequence-level proposal, limiting token-level methods such as CISPO, BAPO, GPPO/KLEAR, M2PO, and SAPO.SAPO uses token-level smooth gating and shares VESPO’s philosophy while retaining the token-level limitation.

6 Conclusion

VESPO provides a principled sequence-level importance-weight reshaping kernel with explicit variance guarantees, remaining stable under severe staleness and transferring across model types and code generation without retuning.

  • 6 Conclusion: VESPO reframes importance-weight reshaping through measure change and derives a sequence-level smooth kernel with explicit variance guarantees.The kernel is principled, though selecting specific (c1, c2) values remains a user choice.
  • 6 Conclusion: VESPO remains stable under staleness up to 64× and improves consistently across dense and MoE models.The reported experiments cover verifiable-reward settings in math and code, using models up to 30B-scale.
  • 6 Conclusion: VESPO transfers to code generation without retuning.The conclusion reports this transfer alongside results on math and code tasks.

Supplementary Material

The supplementary material contains derivations and supporting analyses for VESPO, including proposal distributions, variance guarantees, and experimental details.

  • Supplementary Material: It analyzes the implicit proposal distributions of existing methods.
  • Supplementary Material: It derives the proposal distribution used by VESPO.
  • Supplementary Material: It proves the variance bound and uniform boundedness.
  • Supplementary Material: It provides experimental details.

A Implicit Proposal Distributions of Existing Methods · B Derivation of the Proposal Distribution

The analysis shows that token-level weighting lacks a coherent sequence-level proposal, while length normalization and hard clipping introduce distinct biases or optimization issues. VESPO derives a smooth sequence-level proposal and surrogate objective from a constrained variational formulation, with an on-policy-preserving shifted form.

  • A Implicit Proposal Distributions of Existing Methods: Token-level weighting assigns different weights within one trajectory, so it cannot represent importance sampling toward a single proposal distribution Q.This inconsistency breaks coherent sequence-level credit assignment for tokens sharing a common outcome.
  • A Implicit Proposal Distributions of Existing Methods: Token-level importance sampling keeps only diagonal first-order terms, discarding cross-token interactions that create approximation error.The approximation is most reasonable near on-policy and for short sequences, conditions often violated in off-policy LLM reasoning tasks.
  • A Implicit Proposal Distributions of Existing Methods: GSPO’s geometric-mean weighting makes its proposal depend explicitly on sequence length, and as T →∞, QGSPO →µ.Longer sequences therefore receive vanishingly small corrections toward the target policy.
  • A Implicit Proposal Distributions of Existing Methods: Sequence-level hard clipping truncates trajectories with W > c and introduces a discontinuity at W = c that can hinder optimization.Standard PPO clips at the token level, combining token-wise weighting with hard truncation issues.
  • B Derivation of the Proposal Distribution: The constrained variational problem yields a proposal with log Q*(τ) = log µ(τ) + α log W(τ) −λW(τ) + const.The equivalent dual-KL form interprets Q as interpolating between µ at α=0 and π at α=1.
  • B Derivation of the Proposal Distribution: VESPO’s reshaped gradient is the gradient of an implicit surrogate objective Eτ∼µ[f(W) A(τ)], where f′(W) = ϕ(W)/W.For ϕ(W) = W^α exp(−λW), f′ integrates to the lower incomplete gamma function.
  • B Derivation of the Proposal Distribution: The resulting kernel is smooth, infinitely differentiable, and saturates as W →∞, providing a principled soft alternative to hard clipping.The practical shifted form ϕ(W) = W^c1 exp(c2(1 −W)) satisfies ϕ(1) = 1, so on-policy samples receive unit weight.

C Proof of the Variance Bound and Uniform Boundedness

The proof establishes a finite variance bound for the deployed kernel when c1 ≥ 1 and proves uniform boundedness for all positive c1, c2. Together, these results control second moments under the moment constraint and bound each sample’s gradient contribution regardless of staleness.

  • Variance Bound: Eµ[ϕ(W)^2] ≤ Z · K · C follows from the variational representation, the definition K = supw>0 ϕ(w)/w, and EQ[W] ≤ C.This gives the stated second-moment control under the moment constraint.
  • Variance Bound: K is finite if and only if c1 ≥ 1; for c1 = 1, K = exp(c2), while for c1 > 1 it is attained at w∗ = (c1 −1)/c2.The ratio ϕ(w)/w diverges as w →0+ when c1 < 1 and decays to zero as w →∞ for c2 > 0.
  • Uniform Boundedness: ϕ(W) is uniformly bounded for any c1, c2 > 0, with its maximum attained at W ∗∗ = c1/c2.The maximum value is ϕmax = (c1/c2)c1 exp(c2 −c1).
  • Practical Settings: For (c1, c2) = (2, 3), K = (1/3)e2 ≈2.46 and ϕmax = (2/3)2e ≈1.21.These are numerical values at a practical kernel setting.
  • Practical Settings: For (c1, c2) = (3, 2), K = 1 and ϕmax = (3/2)3e−1 ≈1.24.The propositions imply bounded gradient contribution from any single off-policy sample regardless of staleness.

D Experimental Details … I Algorithm Pseudocode

The paper evaluates VESPO under controlled dense and MoE training setups, including severe simulated staleness, and analyzes length-normalization bias, baseline failures, code-generation dynamics, and the algorithm’s sequence-level implementation. Across these analyses, VESPO uses fixed asymmetric reshaping parameters, avoids length normalization, and maintains stable training where several baselines fail.

  • D Experimental Details: Experiments use 32 NVIDIA H20 GPUs, 1,500 gradient steps, maximum sequence length 16,384 tokens, and 8 responses per query across dense and MoE models.Rollouts use vLLM 0.11.0, with FSDP for dense models and Megatron for the MoE model.
  • D Experimental Details: Primary experiments use staleness ratio N=8, with ablations spanning N ∈{4, 8, 16, 32, 64}; VESPO keeps asymmetric parameters (2.0, 3.0) for A>0 and (3.0, 2.0) for A<0.The same VESPO parameters are applied across all models and training tasks without retuning between math and code.
  • E Length Normalization Introduces Bias: As T →∞, GSPO’s normalized weight converges to exp(E[log ρt]), a trajectory-independent constant that dissipates signal and conflates distinct sequences.Sequences with identical per-token statistics but different lengths can receive identical weights despite different true importance weights.
  • E Length Normalization Introduces Bias: VESPO avoids this bias with ϕ(W) = W c1 exp(c2(1−W)), which depends only on sequence-level importance weight W and preserves length-sensitive discrimination without normalization.For equal average per-token log-ratio but different lengths, the longer sequence has W = eT ¯r and receives an appropriately transformed weight.
  • G Training Dynamics for Recent Baseline Comparison: CISPO collapses around step 280, BAPO exhibits entropy explosion after step 1,100, TOPR converges slowly with suppressed response length, and VESPO alone remains stable across all metrics.These failures occur under the matched setup used for the recent importance-weight reshaping comparison.
  • H Code Generation Training Details and Dynamics: On PRIME-RL/Eurus-2-RL-Data, VESPO is best on HumanEval+, MBPP+, and LiveCodeBench v6 using pass@10 with zero hyperparameter tuning.The code reward is execution-based, unlike the verifier-based math reward, while VESPO retains the same asymmetric parameters used for math.
  • H Code Generation Training Details and Dynamics: VESPO maintains the highest training reward with stable entropy and response length throughout 1,500 steps, whereas SAPO collapses around step 1,250 and GRPO and GSPO converge slowly to lower rewards.The dynamics are reported for Qwen3-30B-A3B-Base under N=8.

J Limitations and Future Directions

VESPO’s limitations concern user-selected kernel hyperparameters, untested reward modalities, and limited model-scale evaluation. Future work includes adaptive hyperparameter selection, broader reward settings, and direct verification at frontier scales.

  • Hyperparameter specification: The kernel form is principled, but users must choose (c1, c2); c1 ≥1 defines the inverse-temperature regime where the variance bound is non-vacuous.The study uses (c+1, c+2) = (2, 3) and (c−1, c−2) = (3, 2) without retuning across math and code.
  • Hyperparameter specification: Adaptive scheduling, per-task search, or data-driven learning of (c1, c2) could improve performance and remains open.
  • Reward modality scope: Experiments cover verifier-based math rewards and execution-based code rewards, leaving preference-based, shaped or dense, and open-ended creative settings unevaluated.The paper identifies direct empirical validation in these reward regimes as future work.
  • Model scale: 30B total parameters is the largest evaluated scale, while direct verification beyond 100B remains future work.Larger gains on Qwen3-30B-A3B-Base than on the dense Qwen3-8B-Base or Llama-3.2-3B-Instruct suggest favorable scaling behavior, but do not verify frontier-scale performance.
Loading 2602.10693v3…