Source-linked AI summary
Self-Evolution Reasoner: Continuous Optimization via Policy-Intrinsic Exploration and Exploitation
Wenhang Shi, Yiren Chen, Zhe Zhao, Jinhao Dong, Wei Lu, Xiaoyong Du
TL;DR
RLVR and on-policy distillation face exploration bottlenecks, while external demonstrations bring off-policy and distribution-mismatch challenges. SEPO uses labels to elicit in-distribution reasoning and distills it back into the policy; across scientific, tool-use, and logical reasoning tasks, it outperforms strong baselines in efficiency and convergence, including settings where baselines fail.
Problem
RLVR and on-policy distillation can lack effective supervision when the policy finds no valid solution, while external demonstrations introduce off-policy and distribution-mismatch challenges.
Source: §1 · pp. 1–2Method
SEPO uses ground-truth labels to elicit correct reasoning from the policy, then distills predictions conditioned on that in-distribution reasoning back into the policy.
Source: §1 · p. 2Results
Across scientific, tool-use, and logical reasoning tasks, SEPO outperforms strong RLVR and self-distillation methods in convergence performance and training efficiency, including when baselines underperform or fail.
Source: §1 · p. 2Takeaways & Limitations
SEPO demonstrates continuous improvement across diverse reasoning tasks and challenging settings, including complex tasks and smaller models where baselines may fail.
Source: §1 · p. 2Self-generated-context distillation has no learning signal on problems where the policy samples no correct solution.
Source: §A.2 · pp. 14–15
Abstract
from arXiv · showhide
Reinforcement learning with verifiable rewards (RLVR) enables large language models (LLMs) to continuously self-improve their reasoning capabilities. Compared to RLVR's scalar rewards, recent self-supervised on-policy distillation methods leverage denser supervision signals. However, both paradigms suffer from exploration bottlenecks, struggling to escape local optima when the policy fails to sample valid solutions. Relying on external demonstrations can bypass this issue; however, the off-policy nature hinders genuine self-improvement and the mismatched-distribution reasoning can not be fully exploited. In this work, we propose Self-Evolution Policy Optimization (SEPO), a framework that drives continuous reasoning capability evolution via guided exploration and in-distribution exploitation. Specifically, SEPO utilizes labels to elicit correct reasoning from the policy, unlocking the exploration potential. By treating current policy conditioned on this in-distribution reasoning as a self-teacher, SEPO distills the newly discovered solution back into the policy. Through this iterative cycle of policy-intrinsic exploration and exploitation, SEPO achieves robust and continuous capability evolution. Across scientific, tool-use, and complex logical reasoning domains, SEPO consistently achieves the best performance under identical training time or steps, demonstrating superior efficiency and convergence ceilings. Notably, SEPO maintains stable improvements even in scenarios where baselines severely underperform or fail entirely, such as on complex tasks or with small-scale models. Our code is available at https://github.com/Eric8932/SEPO.
1 Introduction
RLVR and on-policy self-distillation support reasoning improvement but depend on unguided policy trials, while external solutions introduce off-policy and distribution-mismatch problems. SEPO uses labels to elicit in-distribution reasoning and distills it back into the policy; experiments report stronger performance and efficiency across reasoning domains.
1 Introduction
- RLVR and on-policy self-distillation can improve reasoning, but when the policy samples no valid solution they cannot form effective supervision and optimization may stagnate.On-policy self-distillation uses the policy’s correct rollouts as a self-teacher.Source: §1 · p. 1
- External expert solutions can bypass exploration failures, but their off-policy nature and distribution mismatch hinder internalization and impose a performance ceiling.The introduction identifies this as a dual deficit in exploration and exploitation.Source: §1 · p. 2
- SEPO uses ground-truth labels to elicit correct reasoning from the policy, then distills predictions conditioned on that in-distribution reasoning back into the policy.This shifts exploration from generating solutions from scratch to guided deduction toward correct answers.Source: §1 · p. 2
- Across scientific, tool-use, and logical reasoning tasks, SEPO outperforms strong RLVR and self-distillation methods, with better convergence performance and training efficiency.It surpasses baseline performance in less time or fewer steps and maintains improvements when baselines underperform or fail.Source: §1 · p. 2
- The paper presents SEPO as an on-policy self-distillation framework that iteratively uses label-guided exploration and in-distribution exploitation to improve reasoning.Its contributions also include theoretical analysis of exploration and exploitation deficits.Source: §1 · p. 2
2 Background
The background contrasts sparse outcome-reward learning with denser on-policy context distillation. Both depend on context quality and exploration: self-generated contexts can be unavailable, while external contexts create off-policy distribution shifts.
2.1 Reinforcement learning with verifiable rewards
- RLVR gives a scalar, typically binary, reward based on whether a sampled solution’s final prediction matches the ground-truth label.In GRPO, a group of sampled solutions is used to form group-relative advantages applied uniformly across each solution’s tokens.Source: §2.1 · p. 2
- RLVR provides sparse outcome-level supervision without direct credit assignment for intermediate reasoning, and no correct solution in a rollout group can leave it without an effective learning signal.That condition can cause optimization to stagnate.Source: §2.1 · p. 2
2.2 On-policy context distillation
- OPCD conditions the student on a privileged context to create a stronger self-teacher, then distills token-specific predictions over the student’s own generated prefixes.The context can be a valid reasoning path from the student or from external experts; the distillation objective uses KL divergence.Source: §2.2 · pp. 2–3
2.3 RLVR and OPCD
- OPCD is framed as a proxy objective for RLVR, using a teacher that approximates the optimal reward-maximizing policy and denser supervision than sparse-reward methods.Its effectiveness and stability depend on the quality and compatibility of the privileged context.Source: §2.3 · p. 3
- Student-generated contexts preserve on-policy learning but can disappear when the policy finds no correct solution; external contexts bypass that bottleneck but introduce support mismatch and limit self-improvement.The text links the mismatch to difficult exploitation of expert reasoning patterns.Source: §2.3 · p. 3
3 Method: Self-Evolution Policy Optimization
SEPO pairs label-guided reasoning exploration with distillation into the unhinted policy. Its analysis characterizes improvement as depending on the exploration advantage exceeding residual exploitation error, with training approaching a bound as that gap narrows.
3 Method: Self-Evolution Policy Optimization
- The framework’s stated aim is to combine guided reasoning exploration with in-distribution exploitation in an online training loop.Its motivation is to use a context that is both reliably accessible and strictly on-policy.Source: §3 · p. 3
3.1 The SEPO framework
- SEPO elicits label-informed reasoning from its teacher, retains correct reasoning as privileged context, and distills context-conditioned predictions over the student’s rollout prefixes.The student’s unhinted reasoning is pulled toward its own label-guided potential.Source: §3.1 · pp. 3–4
- The teacher is initialized from the student and updated by exponential moving average, so its context evolves with the student while its smoothed target is intended to stabilize learning.The text says smoothing also helps prevent confirmation bias.Source: §3.1 · p. 4
- SEPO uses bounded, symmetric Jensen-Shannon divergence to balance mode-covering and mode-seeking behavior, and top-K truncation to reduce memory footprint while preserving performance.The stated motivation is to avoid the instability associated with extremes of forward and reverse KL.Source: §3.1 · p. 4
3.2 Theoretical analysis of SEPO
- The analysis defines an exploration gap from label-conditioned posterior reasoning and states that it gives the teacher a capability advantage over the unhinted student.The text describes posterior conditioning as pruning the search space.Source: §3.2 · p. 4
- The expected-accuracy bound implies improvement when the exploration gap exceeds the O(√ϵJSD) exploitation penalty; training converges to its empirical upper bound as the gap approaches that error.Synchronous teacher evolution is described as regenerating a substantial gap during training.Source: §3.2 · p. 5
- SEPO’s analysis says it optimizes both exploration advantage and exploitation error: guided exploration raises the ceiling beyond autonomous discovery, while on-policy contexts reduce mismatch and optimization friction.The comparison is with methods using student rollouts or external solutions as context.Source: §3.2 · p. 5
4 Experiments
SEPO is evaluated across scientific, tool-use, and logical reasoning tasks, where it delivers strong short-term and converged performance. Its advantages persist on difficult cold-start tasks and across model scales.
4.1 Experimental settings
- Evaluation spans science, tool-use, and logic tasks, with train-test splits, varied model scales, three-seed averages, peak avg@16, and tuned GRPO and OPCD baselines.Tasks include SciKnowEval science subsets, ToolAlpaca, and SynLogic’s Arrow_Maze and Dyck_Language.Source: §4.1 · p. 5
4.2 Main results
- Across science QA and tool-use benchmarks, SEPO is optimal or near-optimal at 1 and 10 hours, combining stronger early efficiency with a higher convergence ceiling.GRPO lags, while the Expert baseline can gain rapidly but plateaus by 10 hours; Self and GRPO also finish behind SEPO.Source: Table 1 · p. 6, §4.2 · p. 6
4.3 SEPO overcomes severe exploration bottlenecks
- On Arrow Maze, where Qwen3-8B starts at pass@16 of zero, SEPO improves continuously while GRPO and Self fail and Label and Expert plateau quickly.The training curves show SEPO maintaining top performance even where baselines fail.Source: §4.3 · p. 6, Figure 2 · p. 7
4.4 SEPO achieves scale-invariant self-improvement
- After 10 hours on Chemistry, SEPO performs best across Qwen3 scales from 0.6B to 14B, including where smaller models’ OPCD baselines underperform GRPO.The passage attributes SEPO’s small-model advantage to guided exploration and in-distribution targets with a low assimilation threshold.Source: §4.4 · p. 7
5 Analysis
The analysis links SEPO’s concise reasoning to preserved exploration, illustrates self-correction, and examines how distillation and teacher-update choices affect performance. The reported advantages persist across rollout group sizes and teacher-update strategies.
5 Analysis
- After 10h of Chemistry training, SEPO consistently performs best, achieving the highest avg@16 across Qwen3 scales and uniquely outperforming GRPO at 0.6B and 1.7B.Source: Figure 3 · p. 7
5.1 SEPO compresses reasoning and maintains exploration diversity
- At convergence, SEPO’s reasoning is more than 5× shorter than GRPO’s, while it gives up only 2 points in best@16 for a 10-point avg@16 advantage.SEPO also has the highest best@k among OPCD methods at every k, indicating that shorter reasoning does not eliminate its exploration diversity.Source: §5.1 · pp. 7–8, Figure 4 · p. 8
5.2 SEPO cultivates self-reflective reasoning
- In a Chemistry example, SEPO revisits its SMILES interpretation, corrects its cyclopropane atom counts, and arrives at the correct formula and answer.The authors suggest that distilling reasoning-rich teacher contexts may help students internalize self-correction beyond matching final answers.Source: §5.2 · p. 8, Figure 5 · p. 8
5.3 Ablation study
- On 10-hour Chemistry performance, JSD performs best; Forward KL beats Reverse KL, and top-K performance plateaus at K ≥100 while preserving distillation efficacy.Top-K truncation also avoids the memory overhead of storing full teacher and student logits.Source: Figure 6 · p. 8, §5.3 · pp. 8–9
- Across rollout group sizes, smaller groups improve 1-hour performance but lower the 10-hour ceiling, while every tested SEPO configuration beats baselines at both times.SEPO does not strictly require large rollout groups to produce effective training.Source: Figure 7 · p. 9, §5.3 · p. 9
- Across three tasks, EMA improves SEPO and OPCD (Self) over fixed teachers, while SEPO beats OPCD (Self) under both fixed and EMA updates.A fully dynamic teacher performs substantially worse, consistent with an unstable distillation target.Source: §5.3 · p. 9
6 Related Work
Related work covers RLVR and external-expert approaches as well as on-policy context distillation. SEPO is contrasted with methods that rely on external supervision or are limited by exploration and distribution mismatch.
6 Related Work
- Table 2 reports that EMA performs best and SEPO outperforms OPCD (Self) under both fixed and EMA teacher updates.Source: Table 2 · p. 9
6.1 Reinforcement learning with verifiable rewards
- External-expert approaches use expert solutions for auxiliary supervised fine-tuning or to guide RL exploration, whereas SEPO uses self-evolution distillation.The cited approaches include expert rollouts in GRPO groups and expert prefixes as initial contexts.Source: §6.1 · p. 9
6.2 On-policy context distillation
- On-policy context distillation builds a stronger teacher from augmented policy context, but student-rollout methods remain exploration-limited and expert-context methods underexploit off-policy solutions.SEPO instead uses self-explored, in-distribution reasoning in its context.Source: §6.2 · p. 10
7 Conclusion
SEPO is reported to improve training efficiency and final performance across diverse reasoning benchmarks, including settings where baselines fail. The appendix frames RLVR as optimizing toward an optimal policy over successful reasoning paths and OPCD as distilling a context-built proxy for that policy.
7 Conclusion
- SEPO achieves superior training efficiency and final performance across diverse architectures, scales, and reasoning benchmarks, with continued gains where baselines fail.The cited conclusion names complex logical tasks and lightweight models as examples.Source: §7 · p. 10
A.1 Equivalence Between RLVR and OPCD
- Maximizing the regularized RL objective is equivalent to minimizing the current policy’s divergence from π*, because Z(x) is constant with respect to policy parameters.The temperature η controls the KL penalty.Source: §A.1 · p. 14
- With binary rewards and η approaching zero, π* converges to the reference policy normalized over successful trajectories, the conditional success distribution.The appendix characterizes RLVR’s underlying objective as pulling the model toward its intrinsic distribution of successful reasoning paths.Source: §A.1 · p. 14
- OPCD uses a privileged context to construct a teacher approximating π* and distills it with dense token-level supervision.The proxy’s validity depends on the context faithfully approximating the natural conditional success distribution; exploration unavailability, external distribution shifts, or non-causal shortcuts can break the equivalence.Source: §A.1 · p. 14
A.2 Theoretical Limitations of OPCD Baselines
Self-rollout distillation can lose its learning signal when no correct context is sampled, while static expert contexts impose exploitation ceilings. The analysis bounds improvement under residual distillation error and identifies expert-choice and elicitation-cost qualifications.
A.2 Theoretical Limitations of OPCD Baselines
- When no correct self-rollout is sampled, the conditional teacher is undefined and the distillation gradient is zero, removing the learning signal for that problem.The empirical success probability is estimated by sampling G rollouts.Source: §A.2 · pp. 14–15
- Static expert contexts cannot co-evolve, and their out-of-distribution support mismatch distorts student behavior, limiting transfer and imposing a performance ceiling.Forward KL can force inclusion of isolated expert actions, while reverse KL can suppress student-intrinsic actions.Source: §A.2 · p. 15
A.3 SEPO’s Continuous Improvement
- Proposition 1 bounds updated expected accuracy recursively under residual JSD error, with the teacher’s exploration gap offset by an exploitation-error penalty.The proof connects the distillation objective to an accuracy bound through total variation and Jensen–Shannon divergence.Source: §A.3 · pp. 15–16
B.1 Expert Selection in Baselines
- Among GPT-OSS-120B, Qwen3-235B, and DeepSeek-V3.2, DeepSeek-V3.2 was selected for highest self-distillation performance; expert strength or alignment may alter the baseline ceiling.Source: §B.1 · p. 16
B.2 Computational Overhead of the Elicitation Pipeline
- SEPO adds a label-conditioned teacher-elicitation generation step, though reported efficiency and convergence remain superior; streamlining this pipeline is future work.Source: §B.2 · p. 16
B.3 Evaluation at Larger Model Scales
The experiments evaluate models only up to 14B parameters, leaving larger-scale performance unverified. The broader-impact discussion notes potential benefits from reducing reliance on external supervision as well as risks from malicious use.
B.3 Evaluation at Larger Model Scales
- Whether SEPO’s advantages persist beyond the evaluated 14B scale, including at 32B or 72B, remains unverified.The authors identify evaluation at these scales as future work.Source: §B.3 · p. 16
C Broader Impacts
- SEPO may reduce reliance on expensive external supervision while supporting domain-specific reasoning systems.The passage frames these as potential broader impacts.Source: C Broader Impacts · p. 17
- Malicious actors could redirect autonomous self-improvement toward harmful domains such as malicious code or dangerous chemical compounds.The authors describe this as a broader systemic risk of generalized capability-enhancement algorithms, not a risk unique to SEPO.Source: C Broader Impacts · p. 17
D.1 SEPO’s Implementations
SEPO’s implementation uses label-informed teacher reasoning and contextual distillation, with enhanced GRPO and OPCD baselines for comparison. Across fixed-step, stability, and thinking-mode evaluations, SEPO retains strong performance.
D.1 SEPO’s Implementations
- Each prompt receives eight student rollouts and four EMA-teacher candidates from a label-informed prompt; retained candidates guide contextual distillation.Candidates must pass the verifier with reward at least 1; the implementation uses JSD with λ = 0.5 and top-K = 100 shared token support.Source: §D.1 · pp. 17–18
D.2 Enhanced GRPO Implementation
- The enhanced GRPO baseline incorporates asymmetric clipping, avoids biased normalization, and applies off-policy data correction.The implementation follows prior work and includes improvements cited as,, and.Source: §D.2 · p. 18
D.3 Baseline Implementations
- OPCD baselines use self, expert, or label-conditioned contexts, with DeepSeek-V3.2 selected as the primary external expert.For controlled comparison, SEPO and the OPCD baselines use EMA-updated teachers initialized from the same student weights.Source: §D.3 · p. 18
D.4 Meta-Prompts
- The implementation appendix presents separate prompts for student rollouts, label-conditioned teacher reasoning, and contextual distillation.The prompt tables are identified as Tables 3, 4, and 5.Source: §D.4 · p. 18, Table 3 · p. 19, Table 4 · p. 19
E.1 Off-Policy vs. On-Policy GRPO
- The off-policy GRPO variant performs better than the on-policy variant in most evaluated cases.The comparison uses the same experimental settings as Table 1.Source: §E.1 · pp. 18–19
E.2 Results under Fixed Training Steps
- SEPO achieves the highest avg@16 across both 50- and 400-step budgets, supporting its efficiency and higher convergence ceiling.Table 7 reports peak avg@16 for science QA and tool-use benchmarks under those training-step budgets.Source: §E.2 · p. 19
E.3 Standard Deviations Across Independent Runs
- Standard deviations across methods fall between 0.2 and 1.2, which the authors attribute primarily to stochastic trajectory-sampling fluctuations.The reported standard deviations correspond to three independent runs.Source: §E.3 · p. 19
E.4 Thinking Mode
- In thinking mode, SEPO achieves the best performance at both 1-hour and 10-hour milestones on Chemistry and Physics.This evaluation uses Qwen3-8B.Source: §E.4 · p. 19
E.5 Results on the Dyck Language Task
On Dyck Language, SEPO outperforms methods even when the task’s nonzero initial performance lets standard baselines train normally. A Chemistry trace illustrates how label-conditioned teacher reasoning can provide intermediate guidance.
E.5 Results on the Dyck Language Task
- SEPO shows greater sample efficiency at 1 hour and the highest final convergence at 10 hours on Dyck Language, where standard methods train normally.Qwen3-8B begins with a nonzero avg@16 of 12.5 on this task.Source: §E.5 · pp. 19–20
F.1 Example of Intrinsic Teacher Context
- A Chemistry teacher trace revises an initial candidate list and correctly identifies two hydrogen-bond acceptors, illustrating intermediate reasoning and corrective steps.The trace excludes the terminal amide nitrogen because its lone pair is delocalized into the adjacent carbonyl group.Source: §F.1 · p. 20
F.2 Comparison of Student Reasoning Trajectories
A Chemistry example contrasts SEPO’s structural correction with errors or repeated incomplete reasoning in other policies. SEPO revisits the molecular representation, corrects its atom counts, and selects the correct answer.
F.2 Comparison of Student Reasoning Trajectories
- After initially misreading the molecule, SEPO revisits its SMILES representation, corrects the ring and atom counts, and reaches the correct answer.It derives C7H11NO2 and a molecular weight of 141.168 g/mol; the table identifies option B as correct.Source: §F.2 · p. 20, Table 11 · p. 23
- GRPO makes inconsistent atom counts, while Self and Expert derive incorrect formulas and Label repeats an incomplete calculation without correction.Expert selects an option inconsistent with its own calculation, and Self’s calculated mass matches an incorrect option.Source: §F.2 · p. 20, Table 11 · p. 23
- The comparison characterizes SEPO’s trajectory as spontaneous corrective reflection rather than merely matching the final answer.The table legend distinguishes this from mechanical reasoning in GRPO and repeated reflection in Label.Source: §F.2 · p. 20, Table 11 · p. 23