PaperlayerView exact revision on arXiv

Source-linked AI summary

SRPO: Self-Reflective Policy Optimization for Long-Horizon Reasoning

Jialong Liu, Yuling Shi, Ning Yang, Xiaodong Gu, Zuchao Li

arXiv:2608.23493v1cs.AI

TL;DR

Long-horizon post-training is limited by sparse outcome supervision and difficult credit assignment, while existing reflection mechanisms have training limitations. SRPO uses hindsight reflections and reflection-conditioned teacher scoring on on-policy student rollouts to create dense token-level supervision, achieving strong results across mathematical and agentic benchmarks with high data efficiency.

  • Problem

    Current post-training methods struggle to scale to long-horizon tasks because terminal feedback provides sparse supervision for many intermediate decisions.

  • Method

    SRPO analyzes completed trajectories, creates compact reflection patches, and uses the same model as a reflection-conditioned teacher to score student on-policy tokens densely.

  • Results

    SRPO achieves state-of-the-art results across mathematical reasoning and long-horizon agentic benchmarks, including 73.3% on AIME’24 and approximately 3.8× fewer total FLOPs than GRPO.

  • Takeaways & Limitations

    Reflection-guided on-policy training converts episode-level outcomes into dense supervision while steering the student within its own policy space.

  • Takeaways & Limitations

    Self-generated reflections include generic advice, incorrect diagnoses, and problems beyond the model’s capability.

Abstract

from arXiv · show

Self-reflection is a powerful mechanism for credit assignment in human learning, converting sparse outcome feedback into actionable guidance. However, its potential for post-training Large Language Models (LLMs) remains underexplored. We propose Self-Reflective Policy Optimization (SRPO), a framework that internalizes this capability. SRPO enables LLMs to analyze their own completed trajectories, synthesize errors into concise "reflection patches," and use reflection-conditioned teacher scores on student on-policy rollouts as dense token-level training signals. This process effectively transforms sparse terminal supervision into dense, token-level learning signals without requiring external critics, separate reward models, or larger teacher models. We demonstrate that SRPO achieves state-of-the-art performance across mathematical reasoning and long-horizon agentic benchmarks with exceptional data efficiency. Using a Qwen3-8B base model, SRPO attains 73.3% on AIME'24 using only 8% (0.08x) of the training FLOPs required by scaled supervised fine-tuning, while significantly improving success rates on WebShop (64.7%), ALFWorld (76.8%), and SWE-Bench-Lite (31.2%). Code is available at https://github.com/Galleons2029/SRPO

1. Introduction

Long-horizon tasks expose a credit-assignment bottleneck because standard RL supplies only sparse terminal feedback. SRPO addresses this by using hindsight reflection to generate dense token-level supervision and reports strong, compute-efficient results across reasoning and agentic benchmarks.

  • Current post-training methods scale poorly to long-horizon tasks requiring coherent reasoning across dozens or hundreds of intermediate decisions.
  • Terminal success or failure provides only O(1) bits per episode, producing high-variance gradients and sample-inefficient learning.The passage identifies sparse supervision and credit assignment as the root cause of this bottleneck.
  • Existing reflection methods can suffer semantic drift, weak solution-space exploration, and disrupted or excessively verbose reasoning.
  • SRPO treats self-reflection as dense reward generation, using the model as its own teacher for reflection-conditioned scoring of student rollouts.
  • SRPO extracts O(T) bits through per-token reverse KL supervision, converting hindsight reflections into dense learning signals while remaining on-policy.The method uses reflection patches with a clean-state reset to preserve task specification fidelity.
  • 73.3% on AIME’24 and state-of-the-art results across mathematical and agentic benchmarks were achieved with approximately 3.8× fewer total FLOPs than GRPO.The reported benchmarks include WebShop, ALFWorld, and SWE-Bench-Lite.

2. Related Work

Related work spans reinforcement learning for language policies, inference-time reflection, and self-distillation. These approaches motivate SRPO while highlighting the computational and distributional challenges of long-horizon learning.

  • PPO, GRPO, and related reinforcement-learning methods are widely used to align language policies with human preferences or objective correctness.
  • Long-horizon agentic tasks add instruction drift, context management, and error propagation over extended sequences.
  • Large-scale asynchronous RL and system-level optimizations address these complexities but can demand massive computational resources.
  • Self-Refine, Reflexion, and CRITIC use model-generated critiques or verbal feedback to mitigate errors, primarily at inference time.
  • Self-distillation methods learn from high-quality generations, while on-policy distillation uses the model’s current distribution to reduce off-policy mismatch.

3. Methodology

SRPO uses self-reflection to turn sparse outcomes from long-horizon rollouts into dense token-level supervision. It generates a compact reflection patch, reconstructs a clean prompt for a reflection-conditioned teacher, and distills that teacher into the base policy using on-policy student rollouts.

  • Problem setup: SRPO studies sequential text decision-making where policies generate token actions over trajectories and rewards are often sparse or terminal.The setup seeks high expected reward while remaining close to a reference policy through KL regularization.
  • Stage 1: Reflection-guided state augmentation: The method first samples an on-policy rollout, obtains an outcome signal, and uses the completed trajectory to generate a concise reflection patch.The patch contains diagnostic analysis and actionable guidance about failures, decisions, constraints, subgoals, or pitfalls.
  • Stage 1: Reflection-guided state augmentation: SRPO prepends the reflection patch to the original prompt, resetting generation while preserving the original task specification and using reflection as a guiding prior.This reset-with-memory design avoids modifying the environment state and reduces semantic entanglement from iterative reflection appending.
  • Stage 2: On-policy self-distillation: The reflection-conditioned policy serves as a teacher, while the base policy generates student on-policy rollouts from the original prompt.Teacher-forcing scores the student’s tokens under the reflection-augmented prompt rather than replacing them with teacher-generated trajectories, preserving on-policy distillation.
  • Stage 2: On-policy self-distillation: SRPO transfers reflection-enhanced behavior into the base policy through dense token-level supervision and removes the need for explicit reflection at inference.The framework combines on-policy sampling with reflection-conditioned teacher scores and uses group-relative normalization without a learned value function.

4. Experiments

SRPO improves mathematical reasoning and long-horizon agentic performance across the evaluated benchmarks while using substantially less training compute. Its gains are linked to dense self-reflective supervision, on-policy self-distillation, compact actionable reflections, and reset-based training.

  • Experimental Setup: SRPO was evaluated on mathematical reasoning, out-of-distribution generalization, and long-horizon agentic benchmarks across multiple model scales.The benchmark suite includes AIME’24, MATH-500, GSM8K, DeepScaleR, WebShop, ALFWorld, and SWE-Bench-Lite.
  • Mathematical Reasoning: 73.3% on AIME’24, improving 3.3 percentage points over standard on-policy distillation and 5.3 points over GRPO.SRPO achieved these gains with only 8% of the training FLOPs required by SFT-2M extrapolation.
  • Method Analysis: SRPO converts hindsight reflections into dense token-level supervision through on-policy distillation, using the model’s own reflection-conditioned outputs rather than an external teacher.The method compares student and reflection-augmented teacher distributions with reverse KL, yielding O(T) rather than O(1) supervision bits per episode.
  • Long-Horizon Agentic Tasks: 64.7% on WebShop, 76.8% on ALFWorld, and 31.2% on SWE-Bench-Lite, with SRPO achieving the highest success rate on all three tasks.The reported improvements are +7.9 over SFT on WebShop, +5.6 over Reflexion on ALFWorld, and +4.4 over Reflexion on SWE-Bench-Lite.
  • Long-Horizon Agentic Tasks: SRPO’s shortest average episode length was 10.2 steps, indicating improved execution efficiency alongside higher agentic success rates.The analysis links this result to internalized planning strategies rather than trial-and-error exploration.
  • Inference-Time Scaling: Each additional inference-time refinement iteration improved SRPO performance by +1.8% on average, versus +1.2% for GRPO and +0.9% for SFT.At three iterations, SRPO reached 78.5% on AIME’24.

5. Conclusion

SRPO converts episode-level outcomes into hindsight-guided dense supervision and reports gains across reasoning and agentic benchmarks. The framework also removes reliance on larger teacher models and targets continual-learning settings.

  • SRPO converts episode-level outcomes into hindsight-guided dense, reflection-augmented supervision to address sparse-reward RL inefficiency.
  • Across 10 benchmarks spanning math, code, agent, science, and logic, SRPO yields reproducible gains with multi-seed bootstrap confidence intervals and p < 0.005.
  • SRPO matches or exceeds larger-teacher distillation while requiring ∼3.8× fewer total FLOPs than GRPO.
  • SRPO self-distillation outperforms distillation from Qwen3-72B by 0.8 points while using 9× fewer teacher FLOPs.

Impact Statement

The impact statement describes SRPO as a method for improving the reliability and sample efficiency of long-horizon reasoning and agentic behavior. It identifies both broader access benefits and misuse risks requiring safeguards.

  • SRPO is intended to improve the reliability and sample efficiency of long-horizon reasoning and agentic behavior in LLMs.
  • Potential positive impacts include better performance with lower compute and fewer human labels, which can broaden access to capable models.
  • More effective autonomous agents could be misused, including for scalable cyber abuse, without appropriate safeguards.
  • The authors recommend careful evaluation, monitoring, and staged release for high-stakes or open-ended agentic settings.

A.1. Mathematical Reasoning Results

The appendix describes the mathematical-results setup, theoretical motivation, implementation costs, and robustness analyses. It emphasizes dense token-level supervision, on-policy training, and consistent gains across mathematical subdomains.

  • Mathematical Reasoning Results: The tables compare mathematical and agentic benchmarks under controlled training settings, including identical initialization and reported five-seed means.
  • Information-Theoretic Perspective: On-policy distillation with token-level KL rewards provides O(T) bits per episode instead of O(1), and distillation-based methods achieve equivalent performance with 10–100× fewer gradient steps than sparse-reward RL.
  • Why Self-Distillation Works: Reflection patches expose downstream outcome information at a suboptimal decision, making the decision more informed before self-distillation transfers the resulting behavior into the unconditional policy.
  • Why Self-Distillation Works: The reflection is generated before the rethinking rollout, diverse prompts encourage generalization, on-policy sampling follows visited states, and compact patches limit verbose memorization.
  • Mathematical Reasoning Results: Reflections generalize across problem types, and the learned policy maintains strong performance when reflection is removed at inference time.
  • Complete FLOPs Breakdown: Updating the reflection cache every 2–3 epochs reduces overhead by an additional 30–40%, while a leaner variant reduces total cost from 5.4 to 4.2 × 10^18 FLOPs.
  • Complete FLOPs Breakdown: SRPO requires ∼7.5× less wall-clock time than GRPO on 8×H100 GPUs, using ∼8 GPU-hours versus ∼60+ GPU-hours.
  • Mathematical Reasoning Results: MATH-500 improvements are consistent across all seven subdomains, ranging from +6.4 in algebra to +9.6 in number theory.

B.6. Data Contamination Audit

The contamination audit tests 8-gram overlap between OpenThoughts-3 and evaluation benchmarks. Removing the three flagged MATH-500 instances barely changes performance, while DeepScaleR provides a contamination-free reference.

  • The audit flags benchmark instances using 8-gram overlap between the OpenThoughts-3 training set and evaluation problems.
  • The three flagged MATH-500 overlaps were common mathematical phrases rather than problem-level contamination.
  • Removing the three flagged MATH-500 instances changes performance from 81.2 to 81.0, a 0.2-point shift.
  • On DeepScaleR, which has 0% overlap across 1,200 unseen competition problems, SRPO reaches 59.7 versus 55.4 for OPD-32B and 55.8 for OPD-72B.

B.7. Multi-Verifier and Human Evaluation of Reflection Quality

The evaluation checks whether reflection-quality measurements are reliable and whether SRPO’s semantic-control conclusions survive matched experimental conditions. Cross-verifier, human, training-dynamics, and ablation evidence support stable, task-relevant reflections while isolating semantic content from token-length and format effects.

  • Evaluation Reliability: 92.4%–93.2% pairwise agreement was observed among three independent LLM evaluators scoring 500 AIME’24 reflections.Agreement was defined as identical scores or scores within ±1.
  • Evaluation Reliability: 68–74% of 100 expert-rated reflections were Effective, while 18–22% were Redundant and 8–10% Detrimental.Annotators used a ternary scale, with Effectiveness requiring an actionable, problem-specific diagnosis.
  • Training Stability: 3.72 to 3.79 mean helpfulness across iterations 100–500 showed no quality collapse in the self-teacher.The reported means were 3.72 ± 0.41, 3.76 ± 0.38, and 3.79 ± 0.34.
  • Experimental Controls: 66.5% versus 65.8% on AIME’24 shows that equal-length random patches perform like no reflection, not like semantic reflection.The semantic-control experiment isolates reflection content from additional context tokens.
  • Solution Diversity: 2.3 versus 4.1 unique paths and 8.4% versus 15.7% novel-correct rates show the reflection-conditioned self-teacher changes solution diversity relative to OPD-72B.The contrast was consistent across random seeds and embedding models, with ≤0.3-cluster variation.

C.2. Quantitative Results

Quantitative ablations show that SRPO gains depend on task-specific semantic content rather than reflection format or additional tokens. Matched reflections outperform generic, outcome-only, mismatched, and shuffled alternatives, while token lengths remain controlled.

  • Quantitative Results: 66.5% versus 65.8% on AIME’24 shows mismatched reflections perform nearly identically to no reflection.The 0.7-point difference falls within error margins, rejecting extra-token and format-induced explanations.
  • Quantitative Results: Matched reflections outperform template-only, outcome-only, mismatched, and no-reflection conditions, establishing a hierarchy of semantic usefulness.Template-only provides +2.0 over baseline, whereas task-specific reflections provide +7.5.
  • Experimental Controls: Token-length distributions are statistically indistinguishable across controlled conditions, with Kolmogorov–Smirnov p > 0.85.This isolates semantic content as the principal changed factor in the comparison.
  • Semantic Coherence: Shuffled words and shuffled sentences perform comparably to mismatched reflections, showing that disrupting coherent semantic structure removes most of the benefit.The result indicates that the model relies on coherent reasoning structure rather than merely extracting keywords.

C.3. Content Analysis

Content analysis links SRPO’s performance gains to reflection content that is relevant to the current task and coherent enough to guide correction. Training curves and controlled analyses further support semantic alignment as the operative signal.

  • Content Analysis: 78–91% of matched reflections contain task-relevant information, compared with 8–15% for mismatched reflections.The categories include task-specific numbers, constraints, error diagnoses, and generic advice.
  • Qualitative Example: A mismatched reflection can be mathematically valid for its source problem yet provide no actionable guidance for the current geometry problem.The example demonstrates why grammatical validity does not imply task relevance.
  • Training Dynamics: 70% accuracy is reached in approximately 250 iterations with matched reflections, while mismatched and no-reflection conditions plateau near 66% after more than 500 iterations.The mismatched condition does not recover with extended training.
  • Implications: The ablations support the conclusion that semantic alignment is essential and that format, token count, teacher forcing, and future-information exposure are insufficient explanations.The authors characterize aligned reflections as performing soft credit assignment by highlighting decision points and error sources.

D.2. Initial Responses

The initial response explores competing interpretations of the averaging period and whether melting affects the count. It derives both 11 and 6 as possible third-minute additions, with corresponding totals of 20 and 15.

  • Under the three-minute interpretation, the total is set to 15, yielding x = 6 and 15 cubes by the end of the third minute.The response explicitly computes 4 + 5 + x = 15 and then 4 + 5 + 6 = 15.
  • The wording leaves the averaging interval and the effect of frying on melting unspecified, making the answer ambiguous.The response considers whether frying lasts three or four minutes and whether cubes remain intact.
  • The response models the four-minute interpretation as 4 + 5 + x + 0 = 20, yielding x = 11.Under this interpretation, 11 cubes are added at the start of the third minute.
  • The response distinguishes cubes added during the third minute from the total number present at its end.It notes that 11 added cubes would imply 20 total cubes if melting is ignored.

E. Continual Learning and Catastrophic Forgetting

The continual-learning evaluation tests whether SRPO can acquire coding skills after mathematical training while retaining prior mathematical performance. SRPO preserves more of the original capability than GRPO and SFT.

  • Table 13 summarizes a math-to-code continual-learning test in which SRPO preserves original capabilities while acquiring new skills.The table evaluates AIME’24 performance before code adaptation and SWE-Lite adaptation afterward.
  • The evaluation measures both new coding-capability acquisition and retention of mathematical reasoning.The setup trains Qwen3-8B on mathematical reasoning before adapting it to coding tasks.
  • SRPO achieves 95.2% retention of mathematical reasoning after code adaptation, compared with 87.2% for GRPO and 80.3% for SFT.Models are first trained on mathematics and then adapted to SWE-Bench-Lite coding tasks.
  • The paper attributes SRPO’s improved retention to on-policy learning from the model’s own distribution.The accompanying discussion connects this behavior to reduced imitation of out-of-distribution behaviors.
Loading 2608.23493v1…