Source-linked AI summary
Slow-Fast Policy Optimization: Reposition-Before-Update for LLM Reasoning
Ziyan Wang, Zheng Wang, Xingwei Qu, Qi Cheng, Jie Fu, Shengpu Tang, Minjia Zhang, Xiaoming Huo
TL;DR
Single-step GRPO updates can be unstable and underuse rollout batches. SFPO adds fast inner updates, repositioning, and slow correction; on math reasoning benchmarks, it improves stability, sample efficiency, and convergence speed over GRPO.
Problem
GRPO’s single stochastic update per rollout batch can be unstable under high-variance rewards and can discard useful gradient information across inner steps.
Source: §2 · p. 3Method
SFPO combines multiple fast updates on the same batch, repositioning to control off-policy drift, and a slow correction while leaving the base loss and rollout collection unchanged.
Source: §1 · p. 2Results
SFPO consistently improves stability, sample efficiency, and convergence speed over GRPO on math reasoning benchmarks.
Source: §1 · p. 2Takeaways & Limitations
SFPO achieves higher reasoning accuracy and fewer generated tokens without added wall-clock cost.
Source: §7 · p. 10The fast-trajectory intuition assumes a sufficiently small step size and neighborhoods dominated by positive-curvature directions.
Source: §3.1 · p. 4
Abstract
from arXiv · showhide
Reinforcement learning (RL) has become central to enhancing reasoning in large language models (LLMs). Yet on-policy algorithms such as Group Relative Policy Optimization (GRPO) often suffer in early training: noisy gradients from low-quality rollouts lead to unstable updates and inefficient exploration. We introduce Slow-Fast Policy Optimization (SFPO), a simple yet efficient framework to address the above limitations via decomposing each step into three stages: a short fast trajectory of inner steps on the same batch, a reposition mechanism to control off-policy drift, and a final slow correction. This reposition-before-update design preserves the objective and rollout process unchanged, making SFPO plug-compatible with existing policy-gradient pipelines. Extensive experiments demonstrate that SFPO consistently improves stability, reduces number of rollouts, and accelerates convergence of reasoning RL training. Specifically, it outperforms GRPO by up to 2.80 points in average on math reasoning benchmarks. It also achieves up to 4.93\texttimes{} fewer rollouts and an up to 4.19\texttimes{} reduction in wall-clock time to match GRPO's best accuracy. Project website is available at https://slow-fast-po.github.io/.
1 INTRODUCTION
SFPO addresses fragile, inefficient GRPO updates by restructuring training into fast, reposition, and slow stages. The paper reports improved stability, sample efficiency, and convergence over GRPO on math reasoning benchmarks.
1 INTRODUCTION
- RL fine-tunes LLMs with rewards tied to semantic correctness and solution quality; cited examples report GRPO math gains and RL-induced reasoning behaviors across domains (Shao et al., 2024).Source: §1 · p. 1
- Early GRPO training can be unstable because weak rollouts produce high-variance gradients, while one-shot updates underuse batches and direct off-policy reuse can degrade performance as training proceeds.Source: §1 · p. 1
- SFPO reuses each rollout batch through fast inner updates, interpolation toward the starting policy to control off-policy drift, and a final slow correction.Source: Figure 1 · p. 2, §1 · p. 2
- Up to 2.80 points in average on math reasoning benchmarks, SFPO outperforms GRPO; it also uses up to 4.93× fewer rollouts and cuts wall-clock time by up to 4.19× to reach GRPO’s best accuracy.Source: §1 · p. 2
2 PRELIMINARIES
GRPO uses group-relative reward information rather than an explicit value function, alongside policy clipping and KL regularization. Its single stochastic update per rollout batch can remain noisy and limit data efficiency.
2 PRELIMINARIES
- GRPO dispenses with an explicit value function, normalizes rewards across responses to one prompt, and uses relative differences plus an explicit KL term between learned and reference policies.Source: §2 · p. 2
- The token-level importance ratio compares new- and old-policy token probabilities; ε sets the clipping range, β controls KL regularization strength, and πref is the fixed reference policy.Source: §2 · p. 3
- A single stochastic gradient step per rollout batch can be unstable under reward randomness, and group normalization only partially dampens the fluctuations.Source: §2 · p. 3
- Restricting each batch to one update discards potentially useful gradient information across inner steps, limiting variance reduction and efficient data use.Source: §2 · p. 3
3 METHOD
SFPO replaces one-shot policy updates with fast inner steps, repositioning, and a slow correction. The stages aim to stabilize updates while controlling off-policy drift and reusing rollouts without changing the underlying objective or rollout process.
3.1 STAGE I: FAST TRAJECTORY
- Multiple inner updates on the same rollout batch replace a noisy one-shot gradient with a trajectory intended to stabilize the update direction.Each step refines the direction using the same data, making the endpoint less sensitive to randomness in any single gradient estimate.Source: §3.1 · pp. 3–4
3.2 STAGE II: REPOSITION
- Reusing rollouts moves the fast-trajectory endpoint off-policy, so repositioning interpolates toward the starting point to control distribution mismatch.The interpolation factor α trades proximity to the on-policy iterate against reliance on the fast trajectory.Source: §3.2 · p. 4
3.3 STAGE III: SLOW CORRECTION
- After repositioning, a slow correction produces a predictor–corrector structure, with the correction aligned to local curvature at the update point.The slow correction is described as mitigating overshooting while preserving progress along the stabilized trajectory direction.Source: §3.3 · pp. 4–5
- SFPO adapts α rather than K because changing α adjusts trajectory use without changing runtime or discarding computed inner steps.An entropy-triggered schedule sets α to zero after a threshold fluctuation, reverting to pure on-policy updates near convergence.Source: §3.3 · pp. 5–6
3.5 UNIFIED SFPO UPDATE
- SFPO unifies fast updates, repositioning, and slow correction as a plug-compatible replacement that improves stability and sample efficiency on math reasoning benchmarks.It preserves the policy-gradient objective and rollout procedure with minimal engineering overhead.Source: §3 · p. 3, §3.5 · p. 6
4 EXPERIMENTS
Experiments compare SFPO with GRPO across models, training datasets, math benchmarks, and training behavior. They report higher benchmark accuracy, faster convergence, more controlled response lengths, and improved rollout and time efficiency.
4.1 EXPERIMENTAL SETTINGS
- Experiments use five base models, two training datasets, and six math benchmarks, reporting average Pass@1 accuracy across repeated evaluations.Source: §4.1 · p. 6
4.2.1 MATH REASONING BENCHMARKS
- +2.80 average-accuracy gain, from 47.73 to 50.53, is reported for DS-distilled-Qwen-1.5B with SFPO versus GRPO.Source: §4.2.1 · p. 6, Table 1 · p. 7
4.2.2 TRAINING DYNAMICS.
- SFPO converges faster and maintains higher validation accuracy across base models; Qwen3-4B-Base reaches a higher stable accuracy within 150 steps than GRPO does after 400.Source: §4.2.2 · p. 7, Figure 2 · p. 8
- SFPO stabilizes DeepSeek-R1-Distill-Qwen-7B responses around 2700 tokens, while GRPO gradually collapses to overly short responses.Source: §4.2.2 · p. 7, Figure 3 · p. 8
- SFPO’s lower entropy loss is interpreted as more efficient exploration: unproductive search paths are eliminated while reasoning trajectories remain sufficiently broad.Source: §4.2.2 · p. 7
- SFPO achieves higher and more stable reward than GRPO throughout training.Source: §4.2.2 · p. 8, Figure 3 · p. 8
4.2.3 EFFICIENCY ANALYSIS.
- 4.93× fewer rollouts and 4.19× faster training time are reported for DS-Qwen-7B to reach GRPO’s same best accuracy.Source: §4.2.3 · p. 8
5 ANALYSIS AND ABLATION STUDY
The ablations show how α and K interact with off-policy drift, and assess the contributions of slow correction and entropy control to training stability.
Impact of α and K
- At K=3, SFPO remains stable across α values and outperforms GRPO; at K=7, smaller α mitigates fast-weight drift and restores stability.The study attributes this interaction to larger α pulling slow weights toward unstable fast trajectories.Source: §5 · pp. 8–9, Figure 5 · p. 9
Interpolation Against Off-Policy Overfitting
- Interpolation counters off-policy overfitting: with α=1, performance rises initially then declines, while overly small α with small K slightly slows early improvement.For large K, smaller α is preferable because it counteracts severe drift from repeated inner steps.Source: §5 · p. 9
The Importance of Stage III: Slow Correction
- Adding Stage III slow correction improves stability and accuracy over GRPO; without it, repositioning can leave the iterate away from the true descent direction.The paper describes slow correction as a curvature-aware adjustment that realigns updates with the optimization trajectory.Source: §5 · p. 9, Figure 5 · p. 9
Necessity of Entropy Control
- Without entropy control, DS-Qwen-7B accuracy drops after roughly 100 steps as entropy loss rapidly diverges; the implementation resets α to zero after the trigger.After the reset, SFPO reverts to standard GRPO for subsequent steps.Source: §5 · pp. 9–10, Figure 6 · p. 9
Strategies for Decaying α
- Directly setting α to zero after the z-score trigger and gradually decaying it produce similar accuracy and stability curves.The study adopts the direct reset for simplicity and wall-clock efficiency.Source: Figure 7 · p. 9, §5 · p. 10
6 RELATED WORKS
The related work situates SFPO among methods that stabilize GRPO, improve data efficiency, and study off-policy updates. It distinguishes SFPO by restructuring policy updates while retaining the on-policy objective and data pipeline.
RL for LLM Reasoning
- SFPO is presented as orthogonal to GRPO-stabilization methods such as DAPO and Dr. GRPO because it leaves the objective unchanged while restructuring the update.The cited approaches address entropy collapse or remove normalization terms, respectively.Source: §6 · p. 10
Data Efficiency for LLM Reasoning
- Unlike data-filtering approaches that choose which examples to use, SFPO changes how policy updates are performed without changing the data pipeline.The paper describes this update restructuring as complementary to data filtering.Source: §6 · p. 10
Off-policy GRPO
- Unlike off-policy GRPO modifications, SFPO preserves the on-policy GRPO update and restructures one-step optimization into a fast–reposition–slow framework.The cited work examines off-policy variants, reward-improvement conditions, and clipped surrogate objectives.Source: §6 · p. 10
7 CONCLUSION
The conclusion presents SFPO as a three-stage update mechanism for stabilizing early training, with reported accuracy and efficiency gains. Its modular structure is described as a basis for possible curriculum or meta-learning extensions.
7 CONCLUSION
- SFPO combines a fast trajectory, repositioning, and slow correction to stabilize early training, with higher reasoning accuracy, fewer generated tokens, and no added wall-clock cost reported.The conclusion also points to curriculum or meta-learning extensions and richer adaptive rules as future directions.Source: §7 · p. 10
REPRODUCIBILITY STATEMENT
The statement reports that the results are reproducible and that code and experiment scripts will be released upon acceptance; it also specifies the limited role of LLMs in preparing the paper.
REPRODUCIBILITY STATEMENT
- The authors state that all results in Section 4 are fully reproducible and that code and experiment scripts will be released upon acceptance.The passage makes a release commitment conditional on acceptance.Source: REPRODUCIBILITY STATEMENT · p. 11
A LLM USAGE
- LLMs were used only to improve writing, not to generate ideas, design experiments, conduct analyses, or produce scientific content.The authors attribute the research contributions, technical claims, and conclusions to themselves.Source: A LLM USAGE · p. 16
B GPU MEMORY PROFILING RESULTS
SFPO uses an interpolation-based reposition step to stabilize multi-step updates, with similar GPU memory consumption to GRPO during a training step.
B GPU MEMORY PROFILING RESULTS
- SFPO and GRPO show similar GPU memory consumption during one training step.SFPO stores one copy of the model weights rather than heavy optimizer states and parameters, which the authors say adds no significant overhead, especially with GPU sharding.Source: Figure 8 · p. 16, B GPU MEMORY PROFILING RESULTS · p. 16
C.1 IMPACT OF INTERPOLATION SCHEME IN STAGE II
- At K=7, omitting Stage II causes off-policy drift and training collapse, while interpolation remains stable; at K=3, omission degrades toward GRPO.The reposition step is described as crucial for stabilizing multi-step updates and turning inner updates into consistent gains.Source: §C.1 · pp. 16–17
C.2 SFPO TRAINING ON CODING TASKS
On coding and mathematical reasoning benchmarks, SFPO improves on GRPO and comparison methods. The coding ablations also show that the preferred α depends on the inner-update budget K.
C.2 SFPO TRAINING ON CODING TASKS
- On LiveCodeBench, SFPO peaks at Pass@1 (Avg 4) 21.77 at 350 steps, versus GRPO’s 18.91 at 400 steps.The experiment used DeepSeek-R1-Distill-Qwen-1.5B, trained on 14.1k Skywork-OR1 Code examples, and evaluated with a 32K response length.Source: Figure 10 · p. 17, §C.2 · p. 17
- For coding, α = 0.8 performs best with K = 3, while the smaller α = 0.2 is preferred with K = 7 to mitigate stronger distribution drift.The reported α ablations use values 0.2, 0.8, and 1.0.Source: Figure 10 · p. 17, §C.2 · p. 17
C.3 INTERPOLATION IN STAGE-II AGAINST KL PENALTY
- Interpolation outperforms the KL-penalty variant at K=3, reaching 45.44 versus 40.64 at step 200; at K=7 it reaches 46.14 while KL collapses after step 80.The comparison uses Qwen3-4B-Base on mathematical reasoning benchmarks.Source: §C.3 · p. 17, Figure 11 · p. 18
- On both tested models, SFPO exceeds DAPO throughout training and reaches target performance roughly 225 steps earlier on DS-R1-Distill-Qwen-1.5B and 120 steps earlier on Qwen3-4B-Base.Best average accuracy is 50.56 for SFPO versus 49.30 for DAPO on DS-R1-Distill-Qwen-1.5B, and 47.87 versus 46.48 on Qwen3-4B-Base.Source: Figure 12 · p. 18, §C.3 · p. 18
- SFPO also improves on DAPO when dynamic sampling already mitigates zero-advantage batches, indicating its gains are not primarily from addressing that issue.The authors characterize SFPO as an orthogonal optimization layer that can be combined with GRPO, DAPO, or GRPO-family variants.Source: §C.3 · p. 18
C.5 COMPUTE COST AND WALL-CLOCK EFFICIENCY OF SFPO VS GRPO
Although SFPO adds per-step compute, its matched-time and fixed-accuracy comparisons show compute-efficiency gains over GRPO. The method preserves the base objective and rollout procedure, and “fast” and “slow” refer to update frequency.
C.5 COMPUTE COST AND WALL-CLOCK EFFICIENCY OF SFPO VS GRPO
- With K = 3 and α = 0.8, average steps take about 410 s for SFPO versus 300 s for GRPO, or approximately 1.37× longer.Rollout generation takes about 240 s per step for each method; the actor update takes about 150 s for SFPO and 40 s for GRPO.Source: §C.5 · p. 19
- Across about 5–33 hours, SFPO is typically 1–3.6 accuracy points higher at matched time and reaches the same accuracy in 3–6.6× less wall-clock time.The reported comparison is for DeepSeek-R1-Distill-Qwen-1.5B; SFPO has slightly higher per-step compute.Source: Figure 13 · p. 19, §C.5 · p. 19
D.1 NOVELTY
- SFPO’s three-stage pipeline reaches GRPO’s best performance earlier, reducing rollout use and training budget while preserving the original objective and rollout process.It operates as a higher-level mechanism that can combine with GRPO-family methods without changing their objective or KL regularization.Source: §D.1 · p. 19
D.2 NAMING OF SFPO
- “Fast” and “slow” describe update frequency: Stage I performs K inner updates, while Stage III performs one correction update.Source: §D.2 · p. 20
D.3 MORE ENTROPY ANALYSIS
The authors interpret SFPO’s lower entropy as more efficient exploration, connecting it to more stable update directions. Their analysis relates drift to K and α and treats it as a residual for the slow update to compensate.
D.3 MORE ENTROPY ANALYSIS
- SFPO’s lower entropy is interpreted as more efficient exploration: multiple gradient steps stabilize the update direction, while the GRPO entropy bonus remains in place.The authors also cite stable diversity metrics and sustained accuracy gains in support of this interpretation.Source: §D.3 · p. 20
D.4 INFORMAL DEFINITION AND ANALYSIS OF F(K, α) IN SEC. 3.3.
- The analysis assumes a quadratic loss with H ⪰ 0 and zero-mean, bounded-variance gradient noise in both the fast trajectory and the Stage III query.Source: §D.4 · p. 20
- In the analysis, Stage II repositions the fast trajectory, and the drift magnitude increases with K and quadratically with α.For fixed α, F(K, α) is non-decreasing in K; for fixed K, it grows quadratically in α.Source: §D.4 · pp. 20–21
- The descent bound treats F(K, α) as residual error for the one-step slow update to compensate, favoring small K and moderate α.Source: §D.4 · p. 21