PaperlayerView exact revision on arXiv

Source-linked AI summary

$\textbf{Re}^{2}$: Unlocking LLM Reasoning via Reinforcement Learning with Re-solving

Pinzheng Wang, Shuli Xu, Juntao Li, Yu Luo, Dong Li, Jianye Hao, Min Zhang

arXiv:2603.07197v1cs.AI

TL;DR

LLMs can struggle to recover from misguided early reasoning even after generating many more tokens. Re2 uses pure reinforcement learning to let models abandon an unproductive path and restart. The paper reports gains over standard RLVR across domains and difficulty levels, and improved test-time scaling performance.

  • Problem

    When early reasoning is misguided, LLMs often struggle to recover and reach the correct answer, even after generating substantially more reasoning steps and tokens.

    Source: §1 · p. 2

  • Method

    Re2 uses pure reinforcement learning to train models to choose between finalizing an answer and abandoning the current trajectory to re-solve from scratch.

    Source: §1 · p. 2, §4 · p. 5

  • Results

    Re2 consistently outperforms standard RLVR methods across benchmarks of varying domains and difficulty levels, while raising the performance upper bound under test-time scaling.

    Source: §6 · pp. 10–11

  • Takeaways & Limitations

    Re2 achieves a superior test-time compute–performance trade-off curve compared with majority voting.

    Source: §1 · p. 3

    Inference lacks an explicit mechanism to control the probability of re-solving, which may affect efficiency.

    Source: G LIMITATIONS AND FUTURE WORK · p. 23

Abstract

from arXiv · show

Reinforcement learning with verifiable rewards (RLVR) has shown promise in enhancing the reasoning performance of large language models (LLMs) by increasing test-time compute. However, even after extensive RLVR training, such models still tend to generate unnecessary and low-quality steps in their chain-of-thought (CoT), leading to inefficient overthinking and lower answer quality. We show that when the initial direction or quality of the CoT is suboptimal, the model often fails to reach the correct answer, even after generating several times more tokens than when the initial CoT is well-initialized. To this end, we introduce Reinforcement Learning with Re-solving (Re$^2$), in which LLMs learn to flexibly abandon unproductive reasoning paths and restart the solution process when necessary, rather than always committing to a final answer. Re$^2$ applies pure reinforcement learning without any preliminary supervised fine-tuning, successfully amplifying the rare redo behavior in vanilla models from only 0.5% to over 30%. This leads to substantial performance gains over standard RLVR under the same training compute budget, and also demonstrates notable improvements in test-time performance as the number of samples increases.

1 INTRODUCTION

The introduction argues that misguided early reasoning can leave LLMs unable to recover despite generating many more steps, motivating a way to restart unproductive paths. It presents Re2, trained with pure reinforcement learning, and reports gains over RLVR and majority voting.

1 INTRODUCTION

  • Prior work links longer chains of thought and test-time compute with stronger reasoning, while post-training RL has helped produce longer chains and deepen reasoning.The introduction also notes that overthinking and underthinking can generate unnecessary or low-quality steps that degrade efficiency and performance.Source: §1 · pp. 1–2
  • When initial reasoning is suboptimal, LLMs can struggle to reach the correct answer even after generating significantly more reasoning steps and tokens.The introduction motivates restarting when a reasoning path proves unproductive.Source: §1 · p. 2
  • Re2 lets models choose between finalizing an answer and re-solving, rewarding restarts by expected success from scratch and final answers by correctness.It uses pure reinforcement learning without preliminary supervised fine-tuning, raising redo behavior from 0.5% to over 30%.Source: §1 · p. 2
  • Re2 significantly improves reasoning over recent RLVR methods under the same training budget and has a better test-time compute–performance trade-off than majority voting.The evaluation spans diverse reasoning benchmarks and five models from 3B to 14B parameters.Source: §1 · p. 3

2 RELATED WORK

The related-work section situates Re2 among methods for improving LLM reasoning and scaling test-time compute. It distinguishes its learned ability to abandon and restart a reasoning trajectory from single-chain, supervised, decoding, and sampling approaches.

2 RELATED WORK

  • The prevailing RLVR paradigm encourages longer chains of thought but retains a single trajectory, while prior backtracking or termination methods rely on supervised fine-tuning or decoding rather than RL.The authors describe Re2 as enabling models to abandon unproductive paths and re-solve through reinforcement learning.Source: §2 · p. 3
  • Test-time scaling includes iterative self-correction, majority voting, and tree search, whereas Re2 scales compute by abandoning unpromising trajectories and re-solving when needed.The section also places test-time scaling among methods that increase inference-time compute and improve reasoning performance.Source: §2 · p. 3

3 DIFFICULTY OF RECOVERING FROM SUBOPTIMAL EARLY REASONING

The analysis examines how response length relates to accuracy and whether models recover when continued from incorrect prefixes. It finds that early reasoning quality matters and motivates restarting paths that become unproductive.

3 DIFFICULTY OF RECOVERING FROM SUBOPTIMAL EARLY REASONING

  • The analysis uses Qwen2.5-7B-Instruct on AMC23 and DeepScaleR-1.5B-Preview on AIME25, matching dataset difficulty to each model's capabilities.The models are selected to represent an instruction-tuned LLM and a long-CoT reasoning model extensively trained with RLVR.Source: §3 · p. 3

3.1 CORRELATION BETWEEN RESPONSE LENGTH AND PERFORMANCE

  • The study finds a clear negative correlation between CoT length and accuracy across the dataset and within individual problems.It samples 128 responses per problem; longer responses are often associated with early mistakes or overanalysis, with recovery unlikely despite additional tokens.Source: §3.1 · p. 4

3.2 IMPACT OF INITIAL REASONING QUALITY ON FINAL ACCURACY

  • As the incorrect prefix grows, the relative accuracy drop compared with reasoning from scratch becomes larger.The experiment truncates incorrect responses at 20%, 40%, 60%, or 80% and tracks cases exceeding 25% or 75% drop thresholds.Source: §3.2 · p. 4, Figure 4 · p. 5
  • For most incorrect responses, a significant accuracy drop is already present when only the first 20% is used as the prefix.The analysis compares continuation accuracy from incorrect prefixes with reasoning from scratch.Source: §3.2 · p. 4, Figure 4 · p. 5

3.3 TAKEAWAYS

  • For a given problem, shorter responses with smoother reasoning tend to be more accurate, while longer responses are often associated with lower accuracy.The section concludes that early reasoning quality is crucial to final accuracy.Source: §3.3 · p. 4, §3.1 · p. 4

4 RE2: REINFORCEMENT LEARNING WITH RESOLVING

Re2 trains models to continue reasoning or restart from a partial trajectory, using rewards that estimate whether restarting is worthwhile. It groups continuations by prefix, then computes advantages and updates the policy using a DAPO-style procedure.

4 RE2: REINFORCEMENT LEARNING WITH RESOLVING

  • Re2 trains models to abandon an unpromising reasoning path and restart instead of always committing to a final answer.The method uses reinforcement learning to teach this choice during reasoning.Source: §4 · p. 5, §4.1 · p. 5

4.1 PREFIX GROUP GENERATION

  • Training samples n full responses per query, truncates them at random proportions from 0 to 0.8, and generates m continuations for each resulting prefix.Continuations from the same prefix are grouped for advantage calculation.Source: §4.1 · pp. 5–6

4.2 REWARD STRATEGY WITH RE-SOLVING

  • Correct final answers receive reward 1, incorrect answers receive 0, and a resolve action receives the estimated expected accuracy of restarting from scratch.The estimate uses continuations from other prefix groups, with at most R redo rounds allowed.Source: §4.2 · p. 6

4.3 ADVANTAGE COMPUTATION AND PARAMETER UPDATE

  • Group-wise advantages normalize each continuation’s reward by its group mean and standard deviation before policy updates follow DAPO.Groups whose continuations all have the same outcome are filtered because their advantages are zero and gradients vanish.Source: §4.3 · pp. 6–7

5 EXPERIMENTS

Across the experiments, Re2 improves on DAPO across benchmarks and model types. Its test-time scaling, training behavior, and difficulty-level results show when resolving is used and where the gains are most pronounced.

5.1 EXPERIMENTAL SETUP

  • Training uses 17K DAPO-Math problems and compares methods at matched generated-token budgets across five models and five reasoning benchmarks.The benchmarks span mathematical and graduate-level scientific reasoning.Source: §5.1 · pp. 7–8

5.2 MAIN RESULTS

  • Re2 improves reasoning performance across all five benchmarks and five model types, outperforming DAPO for each model.The evaluations include base, instruction-tuned, and reasoning-optimized models ranging from 3B to 14B parameters.Source: Table 1 · p. 8, §5.2 · p. 8

5.3 PERFORMANCE UNDER TEST-TIME SCALING

  • Above 64 samples, Re2 consistently surpasses RLVR-trained models and continues improving as test-time compute increases.At small sample counts, redo attempts on hard problems can reduce accuracy by lowering the share of valid final answers.Source: §5.3 · p. 9, Figure 6 · p. 9

5.4 TRAINING DYNAMICS OF RE2

  • In the first 20 training steps, Re2’s resolve behavior rises sharply, then gradually declines as training continues.Average reward and resolve reward rise; incorrect answers drop substantially while correct final answers increase slowly.Source: §5.4 · p. 9, Figure 7 · p. 9

5.5 ANALYZING THE MECHANISMS BEHIND RE2’S EFFECTIVENESS

  • 51.2% to 81.7%: Re2’s largest accuracy gain is on DAPO’s occasionally solved Group 4 questions; its resolve rate also falls as questions get easier.For difficult but solvable Group 2 questions, Re2 achieves more than twice DAPO’s accuracy; for problems the base model cannot solve, Re2 almost always resolves rather than answering.Source: Figure 8 · p. 10, §5.5 · p. 10

6 CONCLUSION

The conclusion presents Re2 as a pure-RL approach that lets LLMs restart unpromising reasoning and reports consistent gains over standard RLVR across varied benchmarks. The paper also notes risks inherent to LLM research and describes its open-source basis and implementation documentation.

6 CONCLUSION

  • Re2 uses pure reinforcement learning to let language models restart reasoning when a trajectory appears unpromising, and consistently outperforms standard RLVR across benchmarks of varying domains and difficulty.The conclusion also reports a higher upper bound under test-time scaling.Source: §6 · pp. 10–11

ETHICS STATEMENT

  • Research involving large language models carries risks including misinformation spread and propagation of societal biases.Source: ETHICS STATEMENT · p. 11

REPRODUCIBILITY STATEMENT

A IMPLEMENTATION DETAILS

Re2’s training template elicits an explicit decision to abandon a reasoning trajectory and restart. Reward assignment marks redo actions separately from final-answer completions.

A IMPLEMENTATION DETAILS

  • The Re2 template prompts the model to explicitly signal when it abandons its current reasoning trajectory and restarts the problem.Figure 9 is identified as the template illustration.Source: A IMPLEMENTATION DETAILS · p. 14, Figure 9 · p. 14
  • When a redo phrase precedes a boxed final answer, reward assignment truncates the response there and appends <eos> to distinguish redo actions from final-answer completions.This is intended to support consistent policy learning.Source: A IMPLEMENTATION DETAILS · p. 14
  • A prompt asks the model to solve step by step and enclose any final answer in boxed{}.Source: A IMPLEMENTATION DETAILS · p. 14

B DERIVATION OF THE REWARD OF RE2

The appendix derives resolve rewards from the probability of eventually reaching a correct outcome within a bounded number of redo rounds. It also describes five-benchmark comparisons and reports that Re2 scales better at test time than competing approaches.

B DERIVATION OF THE REWARD OF RE2

  • With at most R redo rounds, the expected reward for choosing resolve is the probability that the first non-resolve outcome is correct.Each round draws from empirical correct, incorrect, and resolve outcomes estimated from out-of-group completions.Source: B DERIVATION OF THE REWARD OF RE2 · p. 15

C.1 MAIN RESULTS

  • Table 2 reports DAPO and Re2 accuracy comparisons with 95% confidence intervals across five reasoning benchmarks.The supplied passages identify the comparison and table scope but do not give cell values.Source: §C.1 · p. 16, Table 2 · p. 16

C.2 PERFORMANCE UNDER TEST-TIME SCALING

  • Re2 achieves better test-time scaling than all competing approaches on AIME25, using consumed tokens as the measure of computational cost.Figure 10 compares Re2 with additional baselines.Source: §C.2 · p. 16, Figure 10 · p. 16

C.3 ANALYSIS OF THE RESOLVE REWARD ESTIMATOR

The resolve reward estimator improves as its sample dimensions increase, while Re2’s training analyses describe changes in degeneration groups and reasoning lengths. The reported patterns include more mature redo decisions later in training.

C.3 ANALYSIS OF THE RESOLVE REWARD ESTIMATOR

  • The Re2 reward estimator becomes more accurate as either n or m increases, with lower bias and variance, and outperforms the EMA baseline across settings.Its evaluation uses 1024 independent samples per training question and varies n or m while fixing the other at 8.Source: §C.3 · pp. 16–17, Figure 11 · p. 17

C.4 ANALYSIS OF THE DEGENERATION GROUP RATE

  • Re2’s overall degeneration-group rate is about 10% higher than DAPO’s, while later-training all-wrong groups gradually become all-redo groups without additional training cost from this effect.Figure 12 reports degeneration group rates for DAPO and Re2.Source: §C.4 · p. 17, Figure 12 · p. 17

C.5 ANALYSIS OF THE REASONING LENGTH

  • DAPO’s CoT lengths grow during training, with wrong CoTs longer than correct ones, whereas Re2’s lengths stabilize and show no substantial differences across correct, wrong, and redo CoTs.The authors associate Re2’s pattern with avoiding strained incorrect chains and over-commitment to unpromising trajectories.Source: §C.5 · p. 17, Figure 13 · p. 18
  • Re2’s redo-CoTs are longer early in training but gradually approach final-answer CoT lengths as training progresses.Source: §C.5 · p. 17
  • Early redo decisions track relatively shallow patterns, but later decisions relate more closely to CoT quality, including approach effectiveness and assumption soundness.Early cues include excessive length, switching between reasoning threads, and brute-force enumeration where pattern summarization is needed.Source: §C.5 · p. 18

D TRAINING COST ANALYSIS OF RE2

Re2 outperforms DAPO under the same training-time budget, with an 11% increase in rollout time. Its two-stage prefix-and-suffix generation accounts for the additional rollout cost.

D TRAINING COST ANALYSIS OF RE2

E CASE ANALYSIS OF COTS WITH DIFFERENT LENGTHS

The case analysis links shorter, smoother reasoning with higher accuracy and finds that long chains following misguided initial reasoning may still fail. Example traces show both a short correct solution and Re2 restarting after an unproductive approach.

E CASE ANALYSIS OF COTS WITH DIFFERENT LENGTHS

  • For a given problem, shorter responses following smoother reasoning tend to be more accurate, while longer responses often accompany lower accuracy.When the initial reasoning direction is misguided, additional tokens may not lead to the correct solution.Source: E CASE ANALYSIS OF COTS WITH DIFFERENT… · p. 18
  • In the perfect-squares example, the shorter response derives the correct answer, 8, while the longer response begins a different solution approach.Source: E CASE ANALYSIS OF COTS WITH DIFFERENT… · p. 22

F RE2 EXAMPLES

  • In the ordered-pairs example, the trace rejects the a = b approach after finding no real solutions, then says it is better to redo the question.Source: Re2 Examples 1 · p. 23

G LIMITATIONS AND FUTURE WORK

The section identifies limits in inference control and study scope, reports Re2 results on an edge-oriented model, and discloses the manuscript-only use of LLMs.

G LIMITATIONS AND FUTURE WORK

  • Re2 may need multiple re-solving rounds; inference lacks explicit control over re-solve probability, and multimodal, tool-use, and search-intensive tasks remain unexplored.The study primarily focuses on large language models and reasoning benchmarks.Source: G LIMITATIONS AND FUTURE WORK · p. 23

H RE2 APPLY IN PANGU

  • 16.6% accuracy improvement on GSM8K; Re2's accuracy is significantly better than the original model on all four benchmarks.The evaluated model, OpenPangu-Embedded-1B, was trained from scratch on Ascend NPUs and designed for efficient edge-device execution.Source: H RE2 APPLY IN PANGU · p. 23, Table 3 · p. 24

I USE OF LLMS

Related papers

Loading 2603.07197v1…