PaperlayerView exact revision on arXiv

Source-linked AI summary

ESPO: Early-Stopping Proximal Policy Optimization

Zihang Li, Rui Zhou, Yingcheng Shi, Wenhan Yu, Zhewen Tan, Zixiang Liu, Zeming Li, Binhua Li, Yongbin Li, Tong Yang, Jieping Ye

arXiv:2605.29860v1cs.LGcs.AI

TL;DR

Long-horizon RL rollouts continue after early reasoning failures, wasting tokens and adding post-failure noise to credit assignment. ESPO detects failure from actor logits and critic values, stops rollouts early, and treats truncation as absorbing failure; on 7B mathematical reasoning, it beats PPO across three benchmarks while saving over 20% of rollout tokens.

  • Problem

    Standard RL algorithms force LLMs to generate to the horizon after irrecoverable reasoning errors, wasting computation and polluting advantage estimates with post-failure noise.

  • Method

    ESPO uses surrogate regret from the actor’s logits, value-gated stopping, and an absorbing failure penalty without auxiliary reward models or human annotations.

  • Results

    ESPO outperforms PPO and DAPO across almost all benchmarks while consuming substantially fewer rollout tokens, including 73.17% average accuracy at 7B versus PPO’s 71.20%.

  • Takeaways & Limitations

    Early stopping concentrates negative learning signals near detected failures and removes uninformative post-failure tokens from rollout buffers.

  • Takeaways & Limitations

    ESPO can miss confidently wrong trajectories, falsely stop some high-entropy correct steps, and requires manual truncation-rate adjustment across models and tasks.

Abstract

from arXiv · show

When a large language model under reinforcement learning commits a wrong reasoning step early in a trajectory, standard algorithms force it to keep generating until the maximum horizon, spending compute on tokens that never receive positive reward and polluting advantage estimates with post-failure noise. We propose ESPO (Early-Stopping Proximal Policy Optimization), which detects trajectory failure on-the-fly and terminates rollouts early. At each generation step, ESPO computes a surrogate regret using only the logits already computed during sampling, and terminates when the smoothed cumulative regret significantly exceeds its estimated values. Truncated trajectories are treated as absorbing failure states with a terminal reward, concentrating negative temporal-difference (TD) errors near the detected failure step without any additional reward model or human annotation. On DeepSeek-R1-Distill-Qwen-7B trained for mathematical reasoning, ESPO surpasses PPO on AIME~2024 (46.28% vs. 45.25%), AMC~2023 (85.83% vs. 82.94%), and MATH-500 (87.42% vs. 85.43%), while saving more than 20% rollout tokens cumulatively.

1 Introduction

ESPO addresses wasted computation and noisy credit assignment after early reasoning failures by stopping rollouts using signals already available to PPO. It combines regret and value signals with terminal failure handling, improving benchmark performance while reducing rollout-token use.

  • Motivation: Standard PPO continues generating to the fixed horizon after an irrecoverable reasoning error, wasting tokens and contaminating credit assignment.The eventual reward is zero or negative, while post-failure tokens receive no positive reward.
  • Method: ESPO reuses the policy’s logits and critic value estimate to detect likely trajectory failure with negligible additional computation.It avoids a separately trained termination module and additional reward-model or human-annotation requirements.
  • Method: ESPO computes per-step surrogate regret from the logit gap and normalizes it with an exponential moving average.The normalized signal remains comparable to the critic’s value estimate during training.
  • Method: A value-gated rule stops rollouts when normalized cumulative regret significantly exceeds estimated remaining value, treating truncation as absorbing failure.This concentrates a negative TD error at the stopping step instead of introducing per-step reward shaping.
  • Results: 59.09% average accuracy and 927.96M cumulative tokens at 1.5B show ESPO outperforming PPO and DAPO while using fewer rollout tokens.The random-truncation ablation scores only 42.4% on AIME 2024 despite a similar average rollout length.

2 Related Work

Prior work improves credit assignment, termination, or inference efficiency, but PPO-style training still commonly rolls out after irrecoverable errors. ESPO targets this training-time inefficiency and can be layered onto PPO-style advantage estimators.

  • PPO-based LLM alignment and reasoning: PPO-style methods such as GRPO and DAPO improve advantage estimation and stability but still roll out trajectories to a fixed horizon.ESPO is described as orthogonal to these methods.
  • Process reward models and step-level credit assignment: Process reward models provide step-level supervision but require costly human annotation, whereas ESPO requires no step-level annotation.Outcome reward models still depend on full trajectories.
  • Learned termination and the options framework: Option-Critic learns a dedicated termination function jointly with the policy, while ESPO maps forced truncations to absorbing failure states.This contrasts learned termination with ESPO’s failure-transition treatment.
  • Inference-time early stopping for reasoning models: Inference-time methods such as ESTAR and TERMINATOR stop reasoning generation after monitoring predictions or learning exit points, unlike ESPO’s training-time rollout termination.TERMINATOR reports 14–55% length reduction on reasoning benchmarks.
  • Efficiency-oriented RL for reasoning: Efficiency-oriented RL methods address compute or length-related learning signals through mechanisms distinct from ESPO’s rollout stopping criterion.The related-work passage identifies DRPO and Latent-GRPO as examples.

3 Preliminaries

The paper models autoregressive LLM generation as a finite-horizon MDP with token actions and deterministic state transitions. Standard actor-critic PPO uses a clipped policy objective, critic values, and GAE to estimate token-level learning signals.

  • Token-level RL for LLM generation: Autoregressive decoding is formulated as a finite-horizon MDP whose state contains the prompt and prior tokens, with the next token as the action.Transitions are deterministic, and episodes end at EOS or the maximum horizon Tmax.
  • Token-level RL for LLM generation: The task uses sparse outcome rewards, with R(x, y1:T) denoting the final task reward such as binary correctness.Intermediate token steps do not receive the final outcome reward directly.
  • Actor-critic PPO: PPO updates the policy by maximizing a clipped surrogate objective based on the current and old policy probabilities.The importance ratio compares those probabilities for the sampled action.
  • Actor-critic PPO: GAE estimates advantages from temporal-difference errors, while the critic supplies a state-dependent estimate of remaining return.These signals allow early-termination effects to propagate backward to preceding tokens.

4 ESPO: Early-Stopping Proximal Policy Optimization

ESPO changes rollout collection rather than PPO’s objective by using a logit-derived deviation signal, frozen EMA normalization, and value-gated stopping. It converts truncated prefixes into absorbing failure transitions so standard GAE and PPO can train on shorter trajectories.

  • Overview: ESPO computes a token-level deviation signal from policy logits, smooths it over time, and compares cumulative deviation with a critic-derived threshold during generation.The signal is obtained from logits already computed for decoding, adding negligible computation.
  • 4 ESPO: Early-Stopping Proximal Policy Optimization: ESPO retains the standard PPO objective and applies GAE to shorter trajectories produced by its rollout-collection procedure.Algorithm 1 samples tokens, checks the stopping condition during generation, and returns a trajectory ending with the failure reward.
  • 4.2 Normalized cumulative stopping statistic: Frozen exponential moving averages normalize the deviation signal using batch statistics updated only at training-batch boundaries.This prevents future information from the current rollout from influencing termination decisions.
  • 4.3 Value-gated early termination: ESPO stops when normalized cumulative regret exceeds a value-dependent threshold, granting larger tolerance to states with higher predicted future return.A proportional controller can adjust the threshold multiplier to maintain a stable stopping frequency, and stopping is disabled during critic warmup.
  • 4.4 Failure transition and PPO training: A stopped trajectory becomes an absorbing failure transition with a terminal penalty, and no further decoding or bootstrap occurs beyond the stopping state.The stopping event creates a negative TD signal at termination that propagates backward through PPO training.
  • Limitations: False positives can terminate locally uncertain but globally recoverable trajectories, introducing inherent objective bias.Critic warmup is used as the primary safeguard against aggressive early truncation before value estimates become informative.

5 Experiments

Experiments evaluate ESPO against PPO and DAPO on held-out mathematical reasoning benchmarks at 1.5B and 7B scales. ESPO improves accuracy across the reported aggregate results while using fewer rollout tokens, and its benefit is complementary to DAPO’s credit-assignment improvements.

  • Models and Benchmarks: Experiments use DeepSeek-R1-Distill-Qwen-1.5B and 7B models across AIME24, AMC23, and MATH500 held-out benchmarks.All methods are trained under identical data, reward, sampling, and evaluation settings.
  • 5.2 Main Results: At 7B, ESPO reaches 73.17% average accuracy versus PPO’s 71.20% and DAPO’s 71.76% across the three benchmarks.ESPO also uses 839.24M cumulative rollout tokens, roughly 22% fewer than PPO and 19% fewer than DAPO.
  • 1.5B Scale Results: At 1.5B, ESPO achieves 59.09% average accuracy versus PPO’s 57.03% and DAPO’s 58.29%, while consuming 927.96M cumulative tokens.This is 24% fewer tokens than DAPO and 13% fewer than PPO; ESPO’s AIME 2024 score is marginally below DAPO’s.
  • Comparison with DAPO: ESPO improves on DAPO across almost all benchmarks while using fewer rollout tokens, indicating compatibility with improved advantage-estimation methods.The paper attributes this complementarity to removing uninformative post-failure tokens from the rollout buffer.

6 Ablation Studies

Ablations show that ESPO’s adaptive warmup, terminal failure penalty, and combined value-regret stopping criterion each materially contribute to performance and efficient truncation.

  • Component ablations: AIME24 improves by 2.1 points with critic warmup, which stabilizes the value baseline before value-gated truncation begins.The reported comparison is 46.3 versus 44.2.
  • Component ablations: Removing the terminal failure penalty reduces AIME24 by 2.6 points and increases average training rollout length from 3278 to 3522 tokens.The penalty concentrates negative TD error at the stopping step for more precise credit assignment.
  • Stopping criterion: Value-only and regret-only stopping score 44.0 and 44.8, respectively, both below full ESPO because the signals provide complementary information.Value-only stopping depends on critic scale, whereas regret-only stopping lacks value-based recovery allowance.
  • Stopping criterion: Random truncation scores 42.4 on AIME24 despite matching ESPO’s stopping rate, showing that stopping must use policy confidence or value estimates.Variant F replaces surrogate regret with a random stop signal.

7 Analysis

ESPO reduces executed response lengths while preserving the underlying length distribution, maintains higher policy entropy, and accepts a small false-positive truncation cost.

  • 7.1 Response Length: ESPO’s actual trajectories are significantly shorter than PPO’s and its untruncated length curve, while its original lengths track PPO’s response-length distribution.This indicates that early stopping reduces executed tokens without destroying response-length behavior.
  • 7.2 Policy Entropy and Diversity: ESPO maintains higher entropy throughout training and slows entropy decay relative to PPO, mitigating premature entropy collapse.The analysis attributes this to removing spurious gradient pressure from post-failure tokens rather than directly penalizing low-probability tokens.
  • 7.3 False-positive truncation: 2.7% of trajectories per batch are correct-yet-truncated false positives, representing a small training cost that the authors report is outweighed by gains over PPO.The false-positive rate measures trajectories that would have recovered under full rollout.

8 Limitations and Future Work

ESPO is limited when incorrect reasoning is highly confident, can falsely stop some high-entropy correct steps, and requires manual truncation-rate adjustment across settings.

  • Limitations and Future Work: Highly confident incorrect models can delay detection because their surrogate regret approaches zero on confidently wrong reasoning branches.The paper also notes false kills on some high-entropy but correct steps and sensitivity to manually adjusted truncation rates.

9 Conclusion

ESPO is a regret-aware rollout termination method for LLM reasoning that removes post-failure noise while reducing rollout-token consumption. Across mathematical reasoning benchmarks, it outperforms PPO and DAPO while reducing cumulative training rollout tokens by more than 20%.

  • Conclusion: ESPO combines actor-logit surrogate regret with a dynamic value-gated threshold to detect and truncate failing trajectories on-the-fly.Truncation applies an implicit failure penalty at termination without auxiliary reward models or human annotations.
  • Conclusion: More than 20% fewer cumulative training rollout tokens accompanies ESPO’s superior performance over PPO and DAPO on multiple mathematical reasoning benchmarks.The conclusion presents this as evidence of a scalable, compute-efficient framework for long-horizon reasoning training.

A Training Details

ESPO training uses specified optimization, rollout, and termination-control hyperparameters.

  • Training Configuration: The learning rate is 1 × 10^-6, with a maximum rollout length of 8192 tokens, global batch size of 64, and 8 rollouts.These settings define the core training and rollout configuration.
  • ESPO Parameters: ESPO sets the failure reward to −1.0, EMA α_ema to 0.99, normalization α_s to 0.9, and initial β to 7.0.These values configure failure penalization, smoothing, normalization, and threshold control.
  • Termination Control: A β adjustment rate of 0.1 targets a termination rate of 0.25.The adjustment maintains the intended frequency of early rollout termination.

B Adaptive Critic Warmup Details

ESPO disables stopping during critic warmup, then activates the stopping criterion after critic convergence or a bounded warmup period.

  • Adaptive Critic Warmup: During warmup, the stopping criterion is disabled and the critic uses only the base PPO objective.This separates initial critic training from adaptive rollout termination.
  • Convergence Criteria: Warmup exits when critic loss magnitude falls below 0.5 or adjacent-step differences stay below 0.1 for three consecutive steps.These conditions indicate critic convergence.
  • Warmup Bound: If convergence is not reached after 10% of total training steps, warmup ends unconditionally and stopping activates.The bounded warmup prevents indefinite delay of the stopping criterion.
  • Critic-Loss Visualization: Figure 3 presents critic loss for the 1.5B model on the left and the 7B model on the right.Compare critic-loss behavior across the two model scales.
Loading 2605.29860v1…