PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Self-Evolving Reasoning

Zihan Liu, Shun Zheng, Xumeng Wen, Yang Wang, Jiang Bian, Mao Yang

arXiv:2510.17498v1cs.CL

TL;DR

Small open-weight reasoners may have weak or unstable verification and refinement on hard tasks, leaving their benefit from self-evolving approaches unclear. DSER models iterative reasoning as a probabilistic Markov process and runs parallel long-horizon processes; on AIME 2024–2025 it solved 5 of 9 hard problems and surpassed its 600B-parameter teacher’s Pass@1 through majority voting.

  • Problem

    It remains unclear how much small and medium-sized open-weight models can benefit from self-evolving reasoning, given weak or unstable verification, refinement, and instruction following on hard tasks.

    Source: §1 · p. 2

  • Method

    DSER models iterative reasoning as a Markov chain and runs multiple independent long-horizon self-evolving processes, with convergence to correct solutions when improvement probability exceeds degradation probability.

    Source: §1 · p. 2

  • Results

    5 of 9 hard AIME problems were solved, and the 8B model’s majority-voting accuracy surpassed its 600B-parameter teacher’s Pass@1.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The findings suggest that smarter inference-time processes can extend smaller models’ reasoning rather than relying only on building larger models.

    Source: §5 · p. 9

    Verification-induced rejecting conditions can prematurely terminate exploration, while accepting conditions risk cementing false-positive solutions.

    Source: §3.4 · p. 6

Abstract

from arXiv · show

Long-form chain-of-thought reasoning has become a cornerstone of advanced reasoning in large language models. While recent verification-refinement frameworks have enabled proprietary models to solve Olympiad-level problems, their effectiveness hinges on strong, reliable verification and correction capabilities, which remain fragile in open-weight, smaller-scale models. This work demonstrates that even with weak verification and refinement capabilities on hard tasks, the reasoning limits of such models can be substantially extended through a probabilistic paradigm we call Deep Self-Evolving Reasoning (DSER). We conceptualize iterative reasoning as a Markov chain, where each step represents a stochastic transition in the solution space. The key insight is that convergence to a correct solution is guaranteed as long as the probability of improvement marginally exceeds that of degradation. By running multiple long-horizon, self-evolving processes in parallel, DSER amplifies these small positive tendencies, enabling the model to asymptotically approach correct answers. Empirically, we apply DSER to the DeepSeek-R1-0528-Qwen3-8B model. On the challenging AIME 2024-2025 benchmark, DSER solves 5 out of 9 previously unsolvable problems and boosts overall performance, enabling this compact model to surpass the single-turn accuracy of its 600B-parameter teacher through majority voting. Beyond its immediate utility for test-time scaling, the DSER framework serves to diagnose the fundamental limitations of current open-weight reasoners. By clearly delineating their shortcomings in self-verification, refinement, and stability, our findings establish a clear research agenda for developing next-generation models with powerful, intrinsic self-evolving capabilities.

1 INTRODUCTION

The introduction asks whether smaller open-weight reasoners with weak, unstable verification can benefit from iterative reasoning. It presents DSER’s probabilistic, parallel self-evolution and reports recovery of hard AIME problems, while identifying self-verification and refinement as future research targets.

1 INTRODUCTION

  • Small and medium-sized open-weight models’ benefits from self-evolving reasoning remain unclear because hard-task verification, refinement, and instruction following can be weak or unstable.These behaviors can lead to unexpected terminations under Huang & Yang’s framework.Source: §1 · p. 2
  • DSER models each iteration as a Markov transition and guarantees convergence to a correct-dominated stationary distribution when improvement probability exceeds degradation probability.The approach runs multiple independent self-evolving processes over sufficiently long iterations.Source: §1 · p. 2
  • Unlike Huang & Yang’s framework, DSER does not allocate extra reasoning budget to verification or add specific loop-exit conditions; parallel voting can still yield a correct majority when degradation is more likely.Correct solutions converge to the same ground truth while incorrect solutions diverge into different results.Source: §1 · p. 2
  • 5 of 9 previously unsolved AIME problems were solved by DSER; its majority-voting accuracy also exceeded the 600B-parameter teacher DeepSeek-R1-0528’s Pass@1.The reported gains were 6.5% on AIME 2024 and 9.0% on AIME 2025 Pass@1.Source: §1 · p. 3
  • The authors suggest DSER could support GRPO exploration and reduce deployment cost, while calling for models with stronger verification, constructive feedback, and correction.The cost reduction is framed as a possibility rather than a reported result.Source: §1 · p. 3

2 RELATED WORK

Related work covers iterative self-critique, feedback-dependent correction, specialized training, and tool-guided refinement. This paper distinguishes its approach through a probabilistic Markov-chain account of how iterative reasoning may improve despite imperfect verification and refinement.

2 RELATED WORK

  • Earlier self-critique, self-correction, and self-reflection methods often depend on feedback quality; reliable external signals support successful self-correction (Kamoi et al., 2024).Seed-Prover combines iterative reasoning with formal verification for IMO 2025 problems (Chen et al., 2025).Source: §2 · p. 3
  • Related directions include training intrinsic self-verification and self-improvement, and using multi-turn tool interactions guided by external tools and environments.These developments are described as part of a broader shift toward self-evolution beyond single-turn reasoning.Source: §2 · p. 3
  • DSER recasts verification-refinement as a Markov process, providing a theoretical basis for convergence to correct solutions under a marginal statistical bias toward improvement.The authors frame this as a way for LLMs to progressively solve previously intractable problems.Source: §2 · p. 3

3 METHODOLOGY

DSER models iterative solution refinement as a stochastic process, using a Markov-chain formulation to analyze when repeated reasoning can converge toward correct answers. Its analysis contrasts this approach with verification-dependent cycles, whose absorbing states can constrain exploration on hard problems.

3.1 THE SELF-EVOLVING PROCESS

  • DSER repeatedly transforms an initial candidate through verification feedback and refinement until a termination condition, such as a fixed iteration count, is met.The initial solution is s(0); each cycle generates feedback on s(n) and uses it to produce s(n+1).Source: §3.1 · pp. 4–5

3.2 MARKOV CHAIN FORMULATION

  • The Markov model tracks correct and incorrect solution states, with improvement and degradation probabilities governing transitions under a consistent transition matrix.The formulation assumes each next state depends on the current solution’s correctness, not the history of earlier solutions.Source: §3.2 · p. 5

3.3 STATIONARY DISTRIBUTION AND CONVERGENCE

  • When pIC > pCI, sufficiently long self-evolution favors a majority of correct solutions; parallel voting may still succeed when pIC < pCI if improvement is not too rare.The paper attributes the latter possibility to correct solutions sharing the ground truth while incorrect solutions diverge.Source: §3.3 · p. 5
  • For challenging problems, low pIC can still yield exponential convergence when pCI is also small, at a rate determined by |λ2| = |1−pIC−pCI|.The convergence speed depends on the second-largest eigenvalue magnitude; the ideal case pCI → 0 and pIC → 1 gives πC → 1.Source: §3.3 · p. 5
  • The framework suggests optimizing pIC and pCI directly and using deep self-evolution during reinforcement-learning exploration to find more solutions for hard tasks..Source: §3.3 · p. 6

3.4 COMPARISON WITH VERIFICATION-DEPENDENT SELF-EVOLVING

  • Verification-dependent cycles use absorbing acceptance and rejection conditions, while DSER marginalizes verification outcomes and relies on improvement relative to degradation.In the comparison framework, five consecutive passes accept a solution and ten consecutive failures reject it; such absorbing states can halt exploration or cement a false positive.Source: §3.4 · p. 6

4 EXPERIMENTS

Across AIME 2024 and 2025, DSER improves DS-8B’s overall Pass@1 and solves hard problems beyond its baseline reach. Results vary by question, and some hard problems retain suboptimal stationary correctness.

Evaluation Setup

  • Evaluation used 60 AIME 2024–2025 problems and up to 64K response tokens per reasoning call; Avg@K estimates single-process Pass@1, while Cons@K measures majority-vote accuracy.Cons@K is computed from a majority vote over K trial outputs.Source: §4 · p. 6

Extended Reasoning Limit for DS-8B

  • DSER solved five of the nine hard AIME questions on which DS-8B failed to produce a correct solution using majority voting over 128 parallel trials.The paper classifies those questions as unsolvable by the base reasoning model under that majority-voting condition.Source: §4 · pp. 6, 9

Overall Improved Performance

  • Pass@1 rose from 82.8% to 89.3% on AIME 2024 (+6.5%) and from 74.4% to 83.4% on AIME 2025 (+9.0%) with DSER.Figure 4 reports overall performance over iterations on the full AIME 2024 and 2025 benchmarks.Source: Figure 4 · p. 7, §4 · p. 7

Per-Question Convergence Analysis

  • Cons@K rises rapidly within the first ten iterations for most solved problems, while Avg@64 continues improving after 80 iterations.Per-question convergence varies: some problems stabilize quickly at high correctness, while others converge slowly or retain many incorrect solutions.Source: §4 · pp. 7–8
  • Suboptimal stationary distributions on the bottom three AIME 2025 questions expose limits in maintaining correct solutions for some hard problems.The per-question patterns include fast convergence that still retains a significant portion of incorrect solutions.Source: §4 · pp. 8–9

Comparison with “Verification-Dependent” Self-Evolving

  • On the same 9 hard AIME questions, verification-dependent self-evolving solved 2, compared with DSER’s 5.Figure 6 presents the verification-dependent approach’s per-question performance and exit ratios for the five problems DSER solved.Source: Figure 6 · p. 8, §4 · p. 9

5 CONCLUSION

DSER reframes iterative reasoning as a Markov chain whose correctness can converge under a marginal statistical bias toward improvement. The paper identifies future directions for stronger self-correction and more efficient DSER, and presents inference-time computation as a potential trade-off with model scale.

Conclusion

  • DSER frames iterative reasoning as a convergent Markov chain whose long-term correctness depends on a marginal statistical bias toward improvement, not flawless steps.The conclusion presents this as a way to extend reasoning boundaries even when open-weight models have weak verification and refinement capabilities.Source: §5 · p. 9

Future Directions

  • Future work could train for robust self-critique and constructive self-correction, refine DSER with better search or learnable verification, and apply it to GRPO exploration.The proposed GRPO application could help discover high-quality reasoning traces for challenging problems.Source: §5 · p. 9

Conclusion

  • The paper presents smarter inference-time processes as a possible route to stronger reasoning alongside building larger models.It characterizes DSER’s results as a promising trade-off between model scale and test-time computation.Source: §5 · p. 9

A.1 THEORETICAL ANALYSIS FOR VERIFICATION-DEPENDENT SELF-EVOLVING

The appendix models verification-dependent self-evolution as a four-state Markov process, then shows that difficult, over-confident cases can limit the chance of stabilizing in a correct solution.

Theoretical analysis

  • The analysis removes the ten-consecutive-fail rejection condition from Huang & Yang’s framework, but the simplified process still depends critically on reliable verification for effective self-evolution.The authors analyze this simplified variant using a Markov transition graph.Source: §A.1 · p. 11, Figure 7 · p. 11

Markov transition model

  • The model has two ongoing states—correct and incorrect—and two absorbing states for terminated correct and incorrect solutions.The labels are S1–S4, respectively.Source: §A.1 · p. 12
  • Five consecutive verification passes move an ongoing correct solution to termination, while the incorrect solution has an analogous absorbing termination state.Transitions between ongoing states are governed by refinement probabilities at each iteration.Source: §A.1 · p. 12

Theoretical result

  • When the model is over-confident on a difficult problem and has limited improvement ability, the probability of reaching the correct solution will not exceed 0.5.The stated condition is that α is at least the improvement probability.Source: §A.1 · pp. 12–13

A.2 CASE STUDIES IN DEEP SELF-EVOLVING

The case studies trace an 80-iteration DSER solution attempt for an AIME 2025 geometry problem, while the supplied reasoning passages also contain conflicting candidate answers and checks that reject a claimed minimizer.

Case study

  • The 80-iteration case study presents a purported solution with answer 60, while another supplied solution gives 62 and a separate check rejects its claimed minimizer.The passages do not establish which candidate answer is correct.Source: §A.2 · pp. 13–15
  • The traced problem asks for the minimum sum of distances from a point X to the five vertices of a specified convex pentagon.The pentagon has side lengths 14, 7, 24, 13, and 26, with angles B and E equal to 60°.Source: §A.2 · p. 15
  • A different worked solution gives 38 + 21√3 and answer 62, but its check says the proposed point is not a minimum.The check notes that another point has a smaller function value.Source: §A.2 · pp. 14–15
  • One worked solution claims the minimum occurs at the midpoint of BE and gives 38 + 19√3, yielding 60.Its accompanying check reports that the midpoint value is greater than a value found elsewhere on BE.Source: §A.2 · p. 15

A.3 ADDITIONAL EXPERIMENTAL RESULTS

Table 1 is described as a case-by-case comparison of the base model and DSER on nine challenging AIME 2024–2025 problems, using separate run and voting metrics.

Table 1

  • 9 challenging AIME 2024–2025 problems are covered in a case-by-case comparison of DS-8B and its DSER enhancement.The supplied text gives the table’s scope, not its cell values.Source: §A.3 · p. 16, Table 1 · p. 17
  • Avg@128 is the base model’s average accuracy over 128 independent runs.Source: Table 1 · p. 17
  • Avg@640 reports accuracy across 640 DSER solutions, while Cons@640 indicates whether their majority vote is correct.The 640 solutions aggregate the final ten self-evolving iterations from each of 64 independent DSER runs.Source: Table 1 · p. 17
Loading 2510.17498v1…