PaperlayerView exact revision on arXiv

Source-linked AI summary

FIPO: Eliciting Deep Reasoning with Future-KL Influenced Policy Optimization

Chiyu Ma, Shuo Yang, Kexin Huang, Jinda Lu, Haoming Meng, Shangshang Wang, Bolin Ding, Soroush Vosoughi, Guoyin Wang, Jingren Zhou

arXiv:2603.19835v3cs.LG

TL;DR

GRPO-style reinforcement learning uses uniform trajectory-level advantages, limiting credit assignment for long reasoning. FIPO introduces discounted Future-KL-based token reweighting, raising AIME 2024 Pass@1 from 50.0% to a peak of 58.0% while extending average chain-of-thought length from 4,000 to over 10,000 tokens. The paper also identifies computational cost and math-only training scope as boundaries of the approach.

  • Problem

    GRPO’s uniform outcome-based advantage cannot distinguish critical reasoning tokens from trivial tokens, creating a coarse credit-assignment problem.

  • Method

    FIPO incorporates discounted Future-KL divergence into policy updates to create dense, token-level supervision based on subsequent trajectory behavior.

  • Results

    58.0% AIME 2024 Pass@1 was achieved at peak, while average chain-of-thought length increased from 4,000 to over 10,000 tokens.

  • Takeaways & Limitations

    Dense advantage formulation can be realized within the efficient GRPO framework to support longer-chain reasoning without requiring complex critic models.

  • Takeaways & Limitations

    Training used only the DAPO dataset and math data, leaving scalability to broader datasets and generalization beyond mathematical domains unexplored.

Abstract

from arXiv · show

We present Future-KL Influenced Policy Optimization (FIPO), a reinforcement learning algorithm designed to overcome reasoning bottlenecks in large language models. While GRPO style training scales effectively, it typically relies on outcome-based rewards (ORM) that distribute a global advantage uniformly across every token in a trajectory. We argue that this coarse-grained credit assignment imposes a performance ceiling by failing to distinguish critical logical pivots from trivial tokens. FIPO addresses this by incorporating discounted future-KL divergence into the policy update, creating a dense advantage formulation that re-weights tokens based on their influence on subsequent trajectory behavior. Empirically, FIPO enables models to break through the length stagnation seen in standard baselines. Evaluated on Qwen2.5-32B, FIPO extends the average chain-of-thought length from roughly 4,000 to over 10,000 tokens and increases AIME 2024 Pass@1 accuracy from 50.0% to a peak of 58.0% (converging at approximately 56.0\%). This outperforms both DeepSeek-R1-Zero-Math-32B (around 47.0%) and o1-mini (approximately 56.0%). Our results suggest that establishing dense advantage formulations is a vital path for evolving ORM-based algorithms to unlock the full reasoning potential of base models. We open-source our training system, built on the verl framework.

1 Introduction

FIPO targets the coarse credit assignment of GRPO-style reinforcement learning by reweighting tokens according to future trajectory behavior. On Qwen2.5-32B-Base, it lengthens reasoning chains and improves AIME 2024 performance under a controlled comparison.

  • Results: FIPO’s performance gain was accompanied by significantly longer average responses than the compared baselines.The comparison includes reproduced DAPO, Deepseek-R1-Zero-32B, and o1-mini.
  • Motivation: GRPO-style training distributes outcome-based rewards uniformly across tokens, treating critical reasoning steps and trivial tokens equally.This coarse-grained credit assignment is associated with intermediate-length reasoning plateaus.
  • FIPO: FIPO incorporates Future-KL divergence into policy updates to reweight current-token advantages using subsequent trajectory behavior.The method also uses influence-weight clipping and filtering to maintain training stability.
  • Results: 58.0% AIME 2024 Pass@1 was reached by FIPO, compared with 50.0% for DAPO.The result was obtained on Qwen2.5-32B-Base using the publicly released DAPO training dataset.
  • Results: Over 10,000 tokens became the deep-reasoning chain length achieved by FIPO, up from a 4,000-token baseline.The model progressively expanded its reasoning chains while accuracy increased.

2 Related Work

Related work spans reasoning-oriented reinforcement learning and open-source efforts to scale it transparently. FIPO is positioned against methods that either revert to PPO or rely on value models with Long-CoT supervision.

  • Reinforcement learning for reasoning: Recent reasoning-focused RL work includes OpenAI’s o-series and DeepSeek-R1, while open-source efforts pursue transparent, scalable implementations.These efforts build on reinforcement learning approaches aimed at improving reasoning capabilities.
  • Open-source RL recipes: DAPO provides a large-scale GRPO-style reproduction on clean base models and serves as FIPO’s primary baseline.The paper adopts DAPO to compare algorithmic changes in a transparent setting.
  • Comparison with prior methods: FIPO refines GRPO directly, whereas Open-Reasoner-Zero reverts to PPO and VC-PPO, VAPO, and T-PPO rely on value models pretrained with Long-CoT data.The paper treats Long-CoT-supervised value models as a potential external knowledge prior that can confound evaluation.

3 Preliminary

PPO uses token-specific advantages from a learned value function, whereas GRPO removes the value network by standardizing group rewards and broadcasting each sequence advantage across its tokens. DAPO further modifies clipping and sampling for stable updates.

  • Proximal Policy Optimization: PPO stabilizes policy optimization with a clipped surrogate objective that constrains updates relative to the old policy.Its token-level probability ratio compares the current and old policies, while the advantage is estimated using a learned value function.
  • Proximal Policy Optimization: PPO’s Generalized Advantage Estimation provides distinct, token-specific signals for temporal credit assignment.This creates dense supervision at each step rather than broadcasting only the final outcome.
  • Group Relative Policy Optimization: GRPO estimates a sequence-level advantage by standardizing rewards across multiple sampled outputs, avoiding the computational burden of a value network.The group statistics are the empirical mean and standard deviation of sampled rewards.
  • Group Relative Policy Optimization: GRPO adds a per-token KL penalty to its clipped objective while using group-based advantages.The probability ratio compares each current token’s probability with its old-policy probability.
  • Group Relative Policy Optimization: GRPO broadcasts each sequence advantage identically across all tokens, assigning uniform credit regardless of individual contribution.This differs from PPO’s token-specific advantage signals.
  • DAPO: DAPO removes the explicit KL penalty, uses asymmetric clipping, and dynamically samples groups containing both positive and negative examples.These changes are intended to amplify advantageous updates and maintain non-trivial optimization gradients.

4 FIPO

FIPO assigns token-level credit using Future-KL, which measures how current decisions influence subsequent trajectories. Masking, discounting, and clipping mechanisms control variance and stabilize this forward-looking policy update.

  • Probability Shift: ∆log p: Positive probability shifts indicate reinforcement of specific reasoning steps, whereas negative shifts indicate that the updated policy suppresses those tokens.The shift is defined as the log-space difference between current and old policies.
  • Future-KL Estimation: FIPO re-weights current-token advantages using cumulative probability shifts across subsequent trajectories rather than instantaneous policy drift alone.Future-KL is interpreted as a sample-based KL estimate over the future sequence.
  • Future-KL Estimation: 13 tokens exceeding the Dual-Clip threshold are masked from future accumulation to remove a primary source of instability.The binary mask excludes harmful outliers whose gradients are already clipped.
  • Future-KL Estimation: Vanilla FutureKL can destabilize training when unregulated negative signals cause low-clip spikes, Policy KL divergence, gradient explosions, and response-length collapse.The observed unstable run occurs around Step 70.
  • Soft Decay Window: The discount factor γ ∈(0, 1] reduces the influence of distant tokens because causal dependence weakens over longer horizons.The soft decay window concentrates credit assignment on the immediate reasoning chain while retaining a continuous rather than hard cutoff.
  • Target Loss: Influence-weight clipping constrains future modulation to [1 − ϵflow, 1 + ϵfhigh], limiting variance from extreme accumulated log-probability shifts.The clipped multiplicative weight controls the magnitude of advantage modulation in the policy update.

5 Experiment

Experiments use controlled DAPO settings on Qwen2.5-32B-Base and evaluate reasoning reliability on AIME benchmarks. FIPO improves average Pass@1 while gains in coverage remain more modest.

  • Experimental Setup: FIPO and DAPO are trained with VeRL on the public DAPO-17K dataset under matched optimization settings and a 20,480-token maximum response length.Each batch contains 512 prompts with 16 sampled responses per prompt; the Future-KL horizon is set to τ = 32.
  • Evaluation: AIME 2024 and AIME 2025 evaluations repeat the DAPO protocol 32 times and report Pass@1 averaged over 32 samples.Inference uses temperature 1.0 and top-p 0.7.
  • Main Results: 6.0%: FIPO improves Pass@1 (Avg@32) by roughly 6.0% over DAPO across both AIME 2024 and AIME 2025.The authors prioritize Avg@32 as the most robust indicator of reasoning reliability.
  • Main Results: FIPO’s coverage improvement is more modest for Pass@32, especially on AIME 2025, than its reliability improvement on Avg@32.The paper attributes this boundary-setting difficulty to reinforcement learning’s reliance on the model’s existing internal knowledge without external augmentation or tools.

6 Analysis

FIPO expands reasoning length while improving the training signals and optimization stability associated with sustained reasoning growth. Its length-weighted advantages rise as DAPO’s decline, alongside more controlled policy evolution and smoother gradients.

  • 6.1 The scaling of length and performance: Over 10,000 tokens: FIPO’s median response length rises from an initial 200, while DAPO plateaus near 4,000 tokens.All reported length percentiles shift upward under FIPO rather than only isolated outliers increasing.
  • 6.1 The scaling of length and performance: FIPO’s response-length expansion tracks accuracy positively across training stages, whereas DAPO reaches a performance bottleneck.The reported correlation slopes vary somewhat by phase but remain consistently positive.
  • 6.2 The dynamics of advantage and sustained reasoning growth: DAPO maintains higher raw rewards because its shorter responses incur smaller overlong penalties, making raw reward an unreliable comparison of reasoning quality.The paper characterizes this behavior as convergence toward a local optimum in a restricted search space.
  • 6.2 The dynamics of advantage and sustained reasoning growth: DAPO’s increasing sampled-batch requirement indicates increasingly non-discriminative training samples and potential overfitting.More aggressive sampling is needed to obtain sufficient effective data for optimization.
  • 6.2 The dynamics of advantage and sustained reasoning growth: FIPO’s length-weighted mean advantage rises throughout training, while DAPO’s declines and reduces the incentive to extend derivations.DAPO’s declining signal is associated with longer positive samples being increasingly dominated by negative samples.
  • 6.3 Smooth Policy Drift, Exploration and Gradient Update: FIPO exhibits structured policy drift, low consistent gradient norms, and rising entropy, while DAPO shows volatile gradients and noisy entropy.These trends are presented as evidence of broader, steadier exploration toward complex reasoning.

7 Conclusion

FIPO addresses coarse-grained credit assignment in standard GRPO by using discounted Future-KL divergence to provide dense token-level supervision. On Qwen2.5-32B-Base, it extends chain-of-thought length beyond 10,000 tokens and raises AIME 2024 performance from 50.0% to a peak of 58.0%.

  • 7 Conclusion: FIPO incorporates discounted Future-KL divergence into policy updates to address coarse-grained credit assignment in standard GRPO.The method transforms sparse outcome-based rewards into dense, token-level supervision.
  • 7 Conclusion: 58.0% peak AIME 2024 accuracy: FIPO rises from the 50.0% baseline while extending average chain-of-thought length from 4,000 to over 10,000 tokens.Accuracy later converges at 56.0%.
  • 7 Conclusion: FIPO demonstrates that dense supervision can be realized within the more efficient GRPO framework without requiring complex critic models.The authors release training code and recipes for further research.

8 Limitations and Future Work

The paper identifies limitations in computational efficiency, task generalization, training-data scope, model scope, and the performance gap between self-evolution and distillation.

  • Cost and Efficiency: FIPO’s reasoning sequences exceeding 10,000 tokens substantially increase training and inference overhead, leaving efficiency optimization for future work.The paper frames long reasoning as a first stage before converting reasoning paths into more concise forms.
  • Task Generalization: Evaluations are primarily conducted on mathematical reasoning benchmarks, leaving behavior in open-ended or less structured domains for future validation.The paper argues mathematics is a rigorous proxy for deep reasoning but does not report broader-domain validation.
  • Training Data Scope: Training exclusively on the DAPO dataset leaves FIPO’s scalability on larger, higher-quality, or more diverse data distributions unexplored.The paper also states that its advantage over o1-mini is domain-specific and may not generalize beyond mathematics.
  • Limited Model Scope: Restricting the study to clean base models with no prior Long-CoT exposure significantly limits the selection of suitable backbone models.Most contemporary open-source reasoning models have already undergone Long-CoT supervised fine-tuning or distillation.
  • Performance Gap vs. Distillation: RL-based self-evolution remains less efficient than direct distillation because teacher models provide denser supervision and stronger heuristics.The paper describes a persistent performance gap between self-trained and distilled variants.

9 Contributions

The experiments compare FIPO with DAPO and other RL-based methods across Qwen2.5-32B-Base and Qwen2.5-7B-MATH, using configurations designed for reproducibility and fair comparison. On Qwen2.5-7B-MATH, FIPO leads the reported AIME results.

  • Reproducibility: Most infrastructure and optimization settings were kept identical between FIPO and DAPO to support fair comparisons, with complete code and training scripts planned for release.The paper also documents hyperparameter configurations for both Qwen2.5-32B-Base and Qwen2.5-7B-MATH.
  • Experimental setup: Training used a global batch size of 512, group size G = 16, learning rate 1 × 10−6, weight decay 0.1, and Future-KL settings of 32.0 with a safety threshold of 10.0.Policy updates used asymmetric clipping ratios of [0.2, 0.28].
  • Experimental setup: Qwen2.5-7B-MATH served as a pilot model for early explorations and ablations before scaling experiments to the 32B parameter regime.The 7B model was used extensively because 32B training was computationally expensive.
  • Experimental setup: The 7B pilot increased group size to G = 32 and used a stricter advantage clipping threshold of 3.0 to stabilize volatile and inconsistently reproducible training gains.These adjustments targeted stability across independent runs.
  • Experimental results: 40.0% AIME 2024 Pass@1 is achieved by FIPO (7B), compared with 22.0% for GRPO (7B) and 36.0% for DAPO (7B).FIPO also maintains a leading 19.0% on AIME 2025, although the performance gap narrows.

B.2 Result Analysis

The analysis finds that FIPO’s length and entropy behavior differs substantially by model scale. The 7B model remains short and low-entropy, whereas length-control modifications on the 32B model increase response length with limited efficiency or performance benefits.

  • Length dynamics: Around 1200 tokens, the Qwen2.5-7B-MATH mean response length remains stable despite FIPO training.This contrasts with the sustained response-length growth typically observed for the 32B model.
  • Length dynamics: A restricted 4K context window and a strong code-based reasoning bias likely constrain the 7B model’s reasoning depth.The analysis characterizes this length stagnation as a capacity and training-prior effect rather than an optimization failure.
  • Entropy dynamics: Lower entropy under FIPO indicates that Qwen2.5-7B-MATH converges toward more deterministic reasoning traces during reinforcement learning.The analysis contrasts this with larger models’ reliance on broader exploration and sustained entropy growth.
  • Ablations: Using the 32B model’s Future-KL clipping range on the 7B model causes entropy to increase continuously, revealing scale-dependent sensitivity to influence-weight clipping.The 32B model instead maintains superior performance with influence weights clipped between 1.0 and 1.2.
  • Ablations: Increasing the 32B maximum response length to 25K tokens triggers rapid early response-length growth but lowers training efficiency and yields only marginal performance gains.The ablation compares default FIPO with a 25K maximum response length.
  • Ablations: Raising the clip-high ratio to 1.4 sharply escalates policy entropy, while both length-control changes are associated with superficial reasoning behaviors.Reported behaviors include repetition, task-irrelevant LaTeX formatting, and premature self-reflection.

C.2 Extreme Value Filtering

Extreme-value filtering stabilizes Future-KL influence weights and improves reliable training, while clipping and decay-horizon choices regulate exploration pressure in 7B models.

  • Extreme Value Filtering: 40.0% AIME2024 peak performance supports filtering over the unfiltered configuration.The unfiltered version can score slightly higher on AIME2025 in one instance, but its AIME2024 gain and overall reliability are suboptimal.
  • Extreme Value Filtering: Filtering extreme importance-sampling ratios makes Future-KL weights more compact and policy updates more stable.Without filtering, disproportionately large influence weights increase aggressive updates and clipping frequency.
  • Influence Weight Clipping: [0.8, 1.2] reaches 40.0% on AIME 2024 versus 36.0% for [1.0, 1.2].The wider exploration pressure of [1.0, 1.2] increases entropy, but 7B models appear sensitive to excessive exploration noise.
  • Decay Horizon: τ = 256 produces the largest influence-weight fluctuations and highest entropy, whereas τ = 8 keeps weights close to baseline.Longer horizons increase variation and clipping frequency, potentially destabilizing policy updates; restrictive horizons limit the weights.

D Case Study

The case study contrasts DAPO’s linear, single-pass reasoning with FIPO’s progression toward self-reflection and systematic multi-pass verification, alongside training-stability differences across mini-batch settings.

  • D Case Study: DAPO remains a linear executor that terminates after the first result, explaining its response-length stagnation.Its reasoning can reach ground truth through standard CoT but lacks self-verification.
  • D Case Study: FIPO’s intermediate stage uses expanding response length for spontaneous self-reflection and alternative-method verification.The model cross-validates an initial result by switching between approaches such as algebraic and geometric reasoning.
  • D Case Study: FIPO’s late stage performs systematic audits through repeated symbolic re-derivation and granular arithmetic verification.This compute-heavy strategy extends beyond simple reflection toward multi-pass checking.
  • Training Stability: Mini-batch 32 frequently shows irreproducible length stagnation, while removing the overlong penalty does not resolve the bottleneck.A successful mini-batch 32 run exists, but its outcome is not reliably reproduced.
  • Training Stability: Mini-batch 64 naturally reduces clipping and stabilizes optimization through steadier gradients and sustained entropy.The successful mini-batch 32 trial reaches approximately 60% Mean@32 and 70% Cons@32, comparable to the mini-batch 64 baseline.

F More details on Training Cost

Future-KL adds quadratic temporal aggregation and memory demands, so FIPO uses chunked computation to avoid dense-matrix memory failures while preserving GPU-parallel processing.

  • Computational Cost: A naive Future-KL implementation requires O(L^2) memory and can cause out-of-memory errors on long trajectories.The dense temporal decay matrix scales with response length L.
  • Memory-Efficient Computation: Chunking partitions the response into fixed-size blocks and computes decay-weighted contributions through parallel matrix multiplications.The block operations use shapes (B, K) × (K, L).
  • Computational Cost: Future-KL increases temporal aggregation from standard GRPO’s O(B · L) element-wise cost to O(B · L^2).The implementation vectorizes this overhead with dense GPU matrix multiplication, producing a relatively limited empirical wall-clock slowdown.
  • Reasoning Evolution: FIPO’s qualitative stages progress from superficial planning and linear execution to self-reflection and systematic deep reasoning.The later stages use alternative derivations, symbolic auditing, and detailed arithmetic verification.
Loading 2603.19835v3…