ReSPO: Reshaped Sequence Policy Optimization for Gradient Starvation in Off-Policy Learning

1 Introduction

• Rollout reuse addresses the cost of slow generation but creates mismatch between the current and rollout-generating policies.The sequence importance ratio W measures this mismatch for a prompt and response.Source: §1 · p. 1
• Clipping starves gradients in opposite ways: under-generated positive responses receive little signal, while over-generated negative responses can dominate.For positive responses, low token weights suppress updates; for negative responses, high weights remain unclipped and unbounded.Source: §1 · p. 1
• ReSPO replaces clipping with a smooth, sign-dependent sequence-level kernel derived from an α-divergence objective and exponential variance-control tilt.Its positive branch preserves weight near W = 0, while its negative branch suppresses both low and high tails.Source: §1 · p. 2

Policy optimization for LLM reasoning

• GRPO, DAPO, and REINFORCE++ use value-free or normalized-advantage approaches, while this paper studies sequence-level gradient starvation.REAL frames gradient starvation as classification; ReSPO retains importance sampling to make off-policy staleness explicit.Source: §2 · p. 2

Sequence-level and off-policy optimization

• GSPO uses a length-normalized sequence ratio, while VESPO derives a reshaping kernel from a KL variational objective.ReSPO generalizes the VESPO framework to an α-divergence family with sign-dependent branches.Source: §2 · p. 2

Trust regions and divergence measures

• Prior trust-region methods use KL bounds or clipping, while ReSPO derives sequence-level reshaping from separable power-Bregman geometry.Its branches allocate gradient according to advantage sign and importance-weight tail behavior.Source: §2 · p. 2

3.1 Preliminaries: Token-Level and Sequence Policy Optimization

• ReSPO applies a soft kernel to sequence-level importance weights because token-wise ratios do not provide exact sequence-level correction for sequence rewards.GSPO aggregates token ratios but its length normalization can introduce length-dependent bias, while hard clipping retains gradient starvation.Source: §3.1 · p. 3

3.2 Gradient Starvation

• Gradient starvation is the sign-dependent failure in which low-ratio positive updates vanish while high-ratio negative updates can grow without bound.ReSPO addresses these failures by reshaping the sequence-level weight W.Source: §3.2 · p. 3

3.3 ReSPO: Reshaped Sequence Policy Optimization

• The derivation treats reshaping as a change of measure, derives an unconstrained kernel from an α-divergence objective, then applies an exponential tilt and chooses α by advantage sign.This three-stage construction is used to satisfy the required tail behavior.Source: §3.3 · p. 3

3.3.1 Weight Reshaping as Measure Change

• Replacing W with a reshaped weight ϕ(W) defines an unnormalized measure τ = µϕ, whose normalized form is τ divided by its total mass.The design strategy solves for τ first, reads off ϕ, then checks its tails against the gradient requirements.Source: §3.3.1 · p. 4

3.3.2 Variational Derivation of the Reshaping Kernel

• The variational objective balances proximity to the behavior policy, movement toward the target policy, and variance control, with β setting the interpolation trade-off.The α-divergence choice determines a power-mean interpolation and its tail behavior.Source: §3.3.2 · p. 4

3.3.3 Exponential Tilt for Variance Control

• A KL projection under a moment constraint produces a smooth exponential tilt of the unconstrained kernel rather than a hard boundary.The α-divergence sets the interpolation and tails; the KL projection supplies the multiplicative tilt for variance control.Source: §3.3.3 · p. 5

3.3.4 Tail Behavior and the Two-Branch Design

• No single α meets both sign-specific tail requirements, so ReSPO uses α(+) > 1 and the α(−) = 1 limit.The positive branch retains nonzero weight at W = 0, whereas the negative branch vanishes at both W = 0 and W → ∞; both branches satisfy ϕ(1) = 1.Source: §3.3.4 · p. 5

3.3.5 Hyperparameter Selection

• The tail analysis fixes α by advantage sign; the authors set β(±) = 0.5 and choose λ(±) to shape the local kernel.The selected branches match in value and slope at W = 1; Figure 1 presents kernel shapes and tail limits for ε = 0.2.Source: §3.3.5 · p. 6, Figure 1 · p. 6

4.1 Experimental Setup

• The experiments use Qwen3-1.7B-Base and Qwen3-30B-A3B-Base on DAPO-MATH-17k, varying rollout reuse N across 8, 16, and 32.Each run uses 1,024 policy updates; evaluation covers six mathematical benchmarks.Source: §4 · p. 6, §4.1 · p. 7

4.2 Training and Evaluation Performance

• ReSPO has the highest observed late training mean in five of six settings and exceeds the highest clipped baseline by 4.2–8.4 pp in all six.At N = 32, the margins over the highest clipped baseline are 7.9 pp on 1.7B and 8.4 pp on 30B.Source: §4.2 · p. 7, Figure 2 · p. 8
• Across both model scales and N values, ReSPO has the highest observed early training peak in five of six settings.Its early-peak margins over the highest baseline at N = 16 and 32 are 1.0 and 3.0 pp on 1.7B and 4.2 and 5.4 pp on 30B.Source: §4.2 · p. 7, Figure 2 · p. 8
• In shared response-length bins, ReSPO has the highest pooled accuracy in 24 of 30 bins, with its largest stated N = 32 margins spanning Q1–Q7.Those margins are 8.4, 16.0, 24.4, 21.6, 20.8, 15.1, and 8.5 pp, respectively.Source: §4.2 · p. 7, Figure 3 · p. 8
• At N = 16 and 32, ReSPO has the highest observed evaluation mean in all four model–N settings, with gains over the highest baseline of 0.8 and 1.3 pp on 1.7B and 3.2 and 1.9 pp on 30B.At N = 8, macro averages are similar relative to evaluation sampling uncertainty.Source: §4.2 · p. 7, Table 1 · p. 8

4.3 Discussion

• ReSPO retains more coefficient mass for low-W positive responses as their mean length grows faster, indicating early learning from longer positive reasoning trajectories.On 30B, its early advantage over the strongest baseline is 4.2 and 5.4 pp at N = 16 and 32 despite the soft length penalty.Source: §4.3 · p. 9
• At N = 32, three-run last-128 means are 27.07% on Qwen3-1.7B and 57.92% on Qwen3-30B-A3B, with standard deviations of 0.54 and 0.95 pp.The passage reports that standard deviations are substantially smaller than ReSPO’s performance gains over every baseline.Source: §4.3 · p. 9
• In a matched 30B ablation at N = 16, adding Routing Replay raises the last-128 mean penalized training accuracy from 56.32% to 58.16%.The comparison also reports improved late-stage training-score smoothness.Source: §4.3 · p. 9

• Clipping suppresses under-generated positive responses at low importance weights while allowing severely over-generated negative responses to dominate at high weights.Source: §5 · p. 9
• ReSPO uses separate positive- and negative-advantage reshaping branches derived from an α-divergence variational objective.The positive branch preserves a nonzero limit as W approaches zero, while the α(−) = 1 limit suppresses both negative-branch extremes.Source: §5 · p. 9
• Across both Qwen3 scales and N values, ReSPO has the highest observed early and late training scores in five of six settings, with differences generally largest at N = 32.It also has the highest observed final-checkpoint evaluation averages at N = 16 and 32 on both scales, and leads in 24 of 30 response-length bins.Source: §5 · p. 9
• Routing Replay complements ReSPO by improving late-stage training score and smoothness, while the paper identifies compute budget and hyperparameter search as limitations.Source: §5 · p. 9

Reproducibility Statement

• The objective, gradient, kernel-selection conditions, stable log-space implementation, algorithm, and experiment launch configurations are documented across Section 3.3, Appendices A–B, and the code artifact.Source: Reproducibility Statement · p. 10
• The reproducibility materials specify training data, checkpoints, baselines, evaluation, uncertainty estimation, and detailed comparisons and diagnostics.Source: Reproducibility Statement · p. 10

A.1 α-Bregman divergence and first-order condition

• Optimizing over an unnormalized measure yields a smooth kernel depending on W alone, avoiding normalized-distribution dependence on behavior mass and a positivity projection that can set weights to zero.The derivation recovers the unconstrained kernel before normalizing the resulting measure.Source: §A.1 · p. 13

A.2 Positive branch: exponential tilt derivation

• The KL projection of the unconstrained kernel applies an exponential tilt, while the two-step procedure uses alpha-divergence for reshaping and KL for variance control.Using the proxy kernel makes the variance constraint linear in the measure; the resulting tilt is multiplicative rather than introducing behavior-mass dependence or a possible hard cutoff.Source: §A.2 · pp. 13–14

A.3 Hyperparameter derivations

• The derivations set alpha(+) = 2 and lambda(+) = 2, and choose lambda(-) = 2 so the branch kernels match in value and first derivative at W = 1.The positive-branch choice gives a stationary maximum at W = 0 with phi(+)(0) = 1; the negative-branch parameter matches the on-policy derivative.Source: §A.3 · p. 15

A.4 Continuous limit as α →1

• As alpha approaches 1, the reshaped kernel tends to W^beta exp(lambda(1 − W^beta)), the form used by ReSPO’s negative branch.VESPO’s gamma-IS kernel instead tilts the raw importance weight W rather than the reshaped weight.Source: §A.4 · p. 15

A.5 On-policy limit

• At W = 1, ReSPO assigns unit reshaping weight, recovering standard on-policy training without importance-ratio weighting.Source: §A.5 · p. 16

A.6 Length-dependent bias from sequence normalization

• GSPO’s 1/T normalization makes off-policy correction depend on response length, whereas ReSPO applies one fixed kernel to the unnormalized sequence weight and intentionally trades bias for variance.Different-length responses with the same average log-ratio can receive the same GSPO weight despite different sequence-level shifts.Source: §A.6 · p. 16

B Log-Space Implementation of the Two-Branch Kernel

• ReSPO computes the sequence-level log-ratio, selects a branch by advantage sign, and uses the resulting detached coefficient to scale the token-level loss.The implementation clamps log W to [−20, 20] before nonlinear operations and uses numerically stable branch computations.Source: B Log-Space Implementation of the… · pp. 16–17

C Detailed Hyperparameters

• The appendix lists shared and method-specific hyperparameter tables and uses batch-wide token-mean normalization, which preserves relative ReSPO weights within each batch.The VESPO settings use its published best-performing sign-specific configuration, while ReSPO uses analytically motivated parameters.Source: C Detailed Hyperparameters · pp. 17–18, Table 2 · p. 18, Table 3 · p. 18

D Evaluation Implementation

• The evaluation-implementation section describes the training reward implementation and evaluation scoring procedure.Source: D Evaluation Implementation · p. 19

D.1 DAPO-MATH training reward implementation

• The training verifier assigns a signed strict-box score, with the 30B runs additionally receiving a length penalty that is zero through 4,096 tokens and reaches −1 at 8,192 tokens.The dense-model runs disable the penalty and use only the signed score.Source: §D.1 · p. 19

D.2 Math-Verify scoring implementation

• Math-Verify gives credit when a parsed completion is mathematically equivalent to the ground truth; unparseable answers, failed matches, and scoring timeouts receive zero credit.The reported pass@1 is the mean binary correctness indicator over samples and problems.Source: §D.2 · p. 19

D.3 Bootstrap evaluation uncertainty

• The evaluation error bars use a paired cluster bootstrap that resamples problems within each benchmark and keeps completions for each sampled problem together.The resulting uncertainty is conditional on the fixed final checkpoint and its stored generations.Source: §D.3 · pp. 19–20

E.1 Detailed training and evaluation comparisons

• ReSPO has the highest early peak in five of six training settings and the highest last-128-step mean in five of six, exceeding the highest clipped baseline by 4.2–8.4 pp in every setting.At N=32, ReSPO exceeds the highest clipped baseline by 7.9 and 8.4 pp on 1.7B and 30B, respectively.Source: §E.1 · p. 20
• ReSPO's held-out macro mean is highest in all four N=16/32 settings, exceeding the highest baseline by 0.8 and 1.3 pp on 1.7B and 3.2 and 1.9 pp on 30B.Its largest observed differences occur on difficult competition benchmarks; on AIME25 it beats the highest baseline in all six model–N settings, by 0.7–7.1 pp.Source: §E.1 · p. 20

E.2 Accuracy across response-length ranges

• At N=32, ReSPO has the highest accuracy in eight of ten pooled response-length bins; its largest reported margins over the highest baseline are 24.4 pp in Q3 and 21.6 pp in Q4.Q10 contains responses at the 16,384-token cap; accuracy generally declines toward the long tail.Source: §E.2 · p. 21, Figure 4 · p. 22

E.3 Positive-response weight and length dynamics

• In the first 16 rollout batches, ReSPO assigns 38.5% of positive-branch coefficient mass to log W < −1, where 22.8% of its positive responses fall, an amplification of 1.69×.By batch 16, mean log W is −0.385 for ReSPO and −0.064 for VESPO, while mean positive-response length grows 43% and 10%, respectively; the diagnostic connects the allocation difference to the branches’ contrasting low-W behavior.Source: §E.3 · pp. 21, 23, 24

E.4 MoE training and Routing Replay

• In the matched Qwen3-30B-A3B N=16 run, adding Routing Replay leaves the early peak essentially unchanged, raises late score by 1.84 pp, and reduces temporal standard deviation from 2.11 to 0.75 pp.Responses also shorten and response-cap hits are nearly eliminated in this window, while approximate KL rises slightly.Source: §E.4 · p. 24

Compute resource limitation

• Compute limits leave longer-context, longer-horizon Qwen3-30B-A3B training untested; the reported study uses an 8,192-token response limit and soft length penalty.Extending context and training horizon could require several weeks on 8×H200 GPUs.Source: F Limitations · p. 25
• Multiple ReSPO seeds are run only at N = 32, where separation from baselines is largest and robustness is most important to verify.Source: F Limitations · p. 25

Hyperparameter optimality

• The study does not exhaustively search α, β, and λ, so other choices may improve particular model scales, reuse ratios, or response-length regimes.Hyperparameters are selected primarily from analytical tail requirements and derivative matching at W = 1.Source: F Limitations · p. 25

Training-statistic calculations

• Late statistics interpolate curves over [896, 1024] and integrate their first and second moments for time-weighted mean and standard deviation.The early statistic is the maximum logged value through step 256 and is reported without an error bar.Source: G Detailed Training Statistics and… · p. 25

Training-curve reporting

• Curves are truncated at 1024 policy updates, plotted against training/global step×N, and transform signed DAPO-MATH scores as (score + 1)/2.Each rollout batch supplies N policy updates.Source: G Detailed Training Statistics and… · p. 25

Additional diagnostics and evaluation

• Figures 5 and 6 add AIME25 evaluation accuracy, response length, approximate KL, and entropy; the 1.7B figure reports best@4 and the 30B figure best@2.The 30B figure uses penalized training accuracy and best@2, the largest common best-of-k metric for all four methods.Source: G Detailed Training Statistics and… · p. 25, Figure 5 · p. 26, Figure 6 · p. 26