PaperlayerView exact revision on arXiv

Source-linked AI summary

Satori: Reinforcement Learning with Chain-of-Action-Thought Enhances LLM Reasoning via Autoregressive Search

Maohao Shen, Guangtao Zeng, Zhenting Qi, Zhang-Wei Hong, Zhenfang Chen, Wei Lu, Gregory Wornell, Subhro Das, David Cox, Chuang Gan

arXiv:2502.02508v3cs.CLcs.AI

TL;DR

Existing test-time search often relies on external feedback and does not internalize search in a single LLM. Satori trains a 7B model with COAT and two-stage tuning plus reinforcement learning, reporting strong in-domain math performance and transfer to out-of-domain tasks.

  • Problem

    Existing test-time scaling often relies on external feedback and two-player frameworks, which do not internalize search capabilities in a single LLM and incur more deployment costs.

    Source: §1 · p. 1

  • Method

    Satori introduces COAT for self-reflection and exploration through meta-actions, then uses small-scale format tuning followed by large-scale reinforcement learning with Restart and Explore.

    Source: §1 · pp. 1–2

  • Results

    Satori outperforms small-scale baselines on math benchmarks, including the instruct model built on the same base, and shows strong transfer to out-of-domain tasks.

    Source: §1 · p. 2, §5.1 · p. 6

  • Takeaways & Limitations

    The authors conclude that format tuning can warm up a policy while large-scale reinforcement learning incentivizes intrinsic reasoning, motivating further research on autoregressive search.

    Source: §7 · p. 9

    After format tuning, the policy struggled to generalize, particularly in meta-action-based self-reflection, because demonstrations were scarce.

    Source: §4.2 · p. 4

Abstract

from arXiv · show

Large language models (LLMs) have demonstrated remarkable reasoning capabilities across diverse domains. Recent studies have shown that increasing test-time computation enhances LLMs' reasoning capabilities. This typically involves extensive sampling at inference time guided by an external LLM verifier, resulting in a two-player system. Despite external guidance, the effectiveness of this system demonstrates the potential of a single LLM to tackle complex tasks. Thus, we pose a new research problem: Can we internalize the searching capabilities to fundamentally enhance the reasoning abilities of a single LLM? This work explores an orthogonal direction focusing on post-training LLMs for autoregressive searching (i.e., an extended reasoning process with self-reflection and self-exploration of new strategies). To achieve this, we propose the Chain-of-Action-Thought (COAT) reasoning and a two-stage training paradigm: 1) a small-scale format tuning stage to internalize the COAT reasoning format and 2) a large-scale self-improvement stage leveraging reinforcement learning. Our approach results in Satori, a 7B LLM trained on open-source models and data. Extensive empirical evaluations demonstrate that Satori achieves state-of-the-art performance on mathematical reasoning benchmarks while exhibits strong generalization to out-of-domain tasks. Code, data, and models are fully open-sourced.

1. Introduction

Satori aims to internalize autoregressive search in a single LLM through Chain-of-Action-Thought (COAT) and a two-stage training paradigm. The introduction presents it as effective for mathematical reasoning and transferable to out-of-domain tasks.

1. Introduction

  • Test-time search methods often rely on external feedback, adding deployment costs without internalizing search capabilities in a single LLM.They use extensive sampling of complete solutions or intermediate reasoning steps, guided by an auxiliary reward model (Wang et al., 2023; Yao et al., 2024; Wan et al., 2024; Sun et al., 2024; Wang et al., 2024a).Source: §1 · p. 1
  • COAT enables autoregressive search by letting an LLM take meta-actions for self-reflection and exploration of new strategies during problem solving.Source: §1 · p. 1
  • Satori uses small-scale format tuning followed by large-scale reinforcement-learning self-improvement with “Restart and Explore” (RAE).The format-tuning stage internalizes COAT before the self-improvement stage.Source: §1 · p. 2
  • Satori is a 7B LLM trained on open-source base models and mathematics data, with reported superior performance on in-domain and out-of-domain tasks.Source: §1 · p. 2
  • Satori performs autoregressive search as a single LLM without external guidance, using minimal supervision and large-scale self-improvement.Source: §1 · p. 2

2. Related Work

The related work contrasts Satori’s reinforcement-learning approach with teacher-model distillation and situates it among post-training and search methods. It highlights differences from DeepSeek-R1 and limitations of prior search approaches.

Concurrent Work

  • Satori uses RL with only 10K format-tuning samples; like DeepSeek-R1 (Guo et al., 2025), it follows cold-start SFT with RL, but differs in methods and scale.DeepSeek-R1 trains a 671B model, whereas Satori is 7B; the paper also presents Satori as a transparent study with open-source training data and recipes.Source: §2 · p. 2

Post-training LLMs for Reasoning

  • Prior reasoning post-training uses instruction-tuning data, self-generated training data, or PPO-style RL, often with reward models; instruction-data construction can be expensive.Instruction-tuning datasets are discussed by Hendrycks et al. (2021a), Yue et al. (2024), Yu et al. (2024), Toshniwal et al. (2024a), and Ding et al. (2024); self-improvement work includes Zelikman et al. (2022; 2024), Singh et al. (2024), and Zhang et al. (2024a). PPO and reward-model approaches include Schulman et al. (2017a), Ouyang et al. (2022), Sun et al. (2024), Wang et al. (2024a), and Yuan et al. (2024).Source: §2 · p. 2

Enabling LLMs with Searching Abilities

  • Prior search methods use prompting or train models for trajectory- or step-level self-exploration, while SoS trains one LLM to perform tree search as a flattened string.Prompting approaches use error correction and alternative paths (Yao et al., 2024; Shinn et al., 2024; Hao et al., 2023; Qi et al., 2024a), but cannot fundamentally enhance reasoning; self-correction difficulties are noted by Zhang et al. (2024b) and Kamoi et al. (2024). Trajectory-level methods include Saunders et al. (2022a), Kumar et al. (2024), Qu et al. (2024), and Havrilla et al. (2024); step-level feedback from another model is explored by Xi et al. (2024), Setlur et al. (2024), Zhang et al. (2024c), Guan et al. (2025), and Zhang et al. (2024d), with high deployment costs. SoS has been shown primarily on simple symbolic tasks, leaving generalization to complex problems open.Source: §2 · p. 2

3. Preliminaries

The section formulates mathematical problem-solving as generating a solution that matches the ground truth for a given prompt. It also introduces reinforcement learning as sequential decision-making that seeks to maximize cumulative reward.

3. Preliminaries

  • The model πθ is trained to generate a solution ỹ matching ground truth y* given a problem prompt x.The prompt, generated solution, and ground truth are token sequences from a predefined dictionary.Source: §3 · p. 2
  • RL models an agent making sequential decisions to maximize expected cumulative rewards through environment interactions.At each step, the agent observes a state, receives a reward, chooses an action, and transitions to the next state; the interaction forms a trajectory and ends at a terminal state.Source: §3 · pp. 2–3

4. Method

The method equips reasoning with COAT meta-actions for continuation, reflection, and exploration, then trains models through format tuning followed by reinforcement-learning self-improvement. Restarting from partial trajectories and adding reward signals address the challenges of long-horizon reasoning and sparse feedback.

4. Method

  • COAT adds meta-actions for continuing, reflecting, and exploring alternatives; conventional CoT is its continuation-only special case.Each reasoning step may begin with a meta-action token that guides the model to proceed, verify prior steps, or seek a new solution.Source: §4 · p. 3
  • Reasoning is modeled as sequential state/action generation with terminal answer rewards, but unfamiliar meta-actions and long-horizon sparse feedback motivate format tuning and restart-and-explore RL.The proposed training sequence first familiarizes the model with COAT, then uses restarts from intermediate steps and exploration bonuses to support correction rather than restarting from scratch.Source: §4 · pp. 3–4

4.1. Format Tuning Through Imitation Learning

  • Format tuning uses generator, critic, and reward-model LLMs to construct COAT demonstrations, then behavior-clones them with supervised fine-tuning; 10K trajectories suffice to learn the format.The generator samples CoT paths, the critic evaluates and refines them, and the reward model scores and selects a final demonstration trajectory.Source: §4.1 · p. 4

4.2. Self-improvement via Reinforcement Learning

  • Restart and Explore trains the format-tuned policy from partial correct and incorrect trajectories, backtracking steps and appending a reflect token before new rollouts.Separate positive and negative restart buffers, combined with the original problem dataset, broaden the initial-state distribution and reduce redundant recomputation.Source: §4.2 · pp. 4–5
  • Reflection bonuses reward solving from an incorrect trajectory and penalize failing after a correct one, while rule-based rewards assess final-answer correctness.The reflection bonus is scaled by hyperparameter β.Source: §4.2 · p. 5
  • To mitigate sparse correct-answer rewards, an Outcome Reward Model trained with Bradley-Terry preferences scores trajectories and supplies finer-grained reward signals.The preference data pairs correct and incorrect outputs, training the model to assign higher values to preferred correct trajectories.Source: §4.2 · p. 5
  • Iterative self-improvement alternates RL with supervised distillation into the base model, using each round’s fine-tuning as a reset intended to help escape local optima.Training starts another RL round from the newly fine-tuned model.Source: §4.2 · p. 5

5. Experiment

Satori-Qwen-7B outperforms small-scale baselines on math benchmarks and transfers to out-of-domain reasoning tasks; its second training round improves performance across most in-domain and out-of-domain benchmarks.

5. Experiment

  • Evaluation uses greedy decoding without tools and reports zero-shot pass@1 across math, logical, code, commonsense, tabular, and domain-specific reasoning benchmarks.Math benchmarks include GSM8K, MATH500, AMC2023, AIME2024, and OlympiadBench; comparisons include the same-base Qwen-2.5-Math-7B-Instruct and larger reasoning models as upper bounds.Source: §5 · p. 6

5.1. Main Results on Math Domain

  • Satori-Qwen-7B outperforms all small-scale baseline models on math benchmarks, including the same-base Qwen-2.5-Math-7B-Instruct.It achieves this despite using less SFT data and relying more on RL data than Qwen-2.5-Math-7B-Instruct.Source: §5.1 · p. 6, Table 1 · p. 6

5.2. Out-of-Domain Transferability

  • Despite math-only training, Satori-Qwen-7B transfers to diverse out-of-domain reasoning benchmarks and surpasses all same-scale baselines on BoardgameQA.The results suggest Satori acquired general reasoning capabilities rather than only math problem-solving skills.Source: Table 2 · p. 6, §5.2 · p. 7

5.3. Results on Iterative Self-improvement

  • Satori-Qwen-7B (Round 2) gains performance over Satori-Qwen-7B across most in-domain and out-of-domain benchmarks.The authors interpret these gains as evidence of the potential of iterative self-improvement to further improve LLM reasoning.Source: Table 1 · p. 6, Table 2 · p. 6, §5.3 · p. 7

6. Analysis

The analysis finds that COAT and RL support Satori’s self-correction and scaling behavior, and that RL outperforms large-scale format tuning on the reported benchmarks. It also reports that distilling Satori-Qwen-7B improves weaker models over format tuning.

COAT Reasoning v.s. CoT Reasoning

  • Removing reflect and explore while holding training data and hyperparameters constant yields a CoT model that underperforms Satori-Qwen-7B and fails to surpass Qwen-2.5-Math-7B-Instruct.The comparison tests COAT reasoning against classical CoT reasoning.Source: Table 3 · p. 7, §6 · p. 7

Satori Exhibits Self-correction Capability

  • RL improves self-correction over Satori-Qwen-FT on MATH500 and Olympiad, and the capability extends to out-of-domain MMLUProSTEM.The evaluation measures cases where the answer changes after reflection, classifying changes from incorrect to correct as positive and correct to incorrect as negative.Source: Table 4 · p. 7, §6 · p. 7

RL Enables Satori with Test-time Scaling Behavior

  • With more RL training compute, Satori’s policy accuracy and generated-token length rise overall, although response length first falls through 200 steps before increasing.The early decrease is attributed to short-cut solutions before the model learns reflection; later, reflection helps it self-correct and find better solutions.Source: §6 · pp. 7–8, Figure 3 · p. 8
  • On MATH datasets, Satori-Qwen allocates more test-time compute to harder problems and consistently improves performance over the format-tuned Satori-Qwen-FT model.The evaluation compares response length and test accuracy across problem difficulty levels.Source: Figure 4 · p. 8, §6 · p. 8

Large-scale FT v.s. Large-scale RL

  • Satori-Qwen, trained with 10K FT data and 300K RL data, outperforms Qwen-2.5-Math-7B trained with 300K FT data across math and out-of-domain benchmarks.The large-scale FT model fails to match the RL model on math benchmarks and performs significantly worse on out-of-domain tasks.Source: Table 5 · p. 8, §6 · p. 8

Distillation Enables Weak-to-Strong Generalization

  • With 240K synthetic examples from Satori-Qwen-7B, Llama-3.1-8B and Granite-3.1-8B outperform their format-tuned counterparts on average math-benchmark accuracy.The proposed approach first trains a strong model with small-scale FT and large-scale RL, then distills its reasoning into weaker models; RL requires only answer labels, reducing data-synthesis costs.Source: Figure 5 · p. 8, §6 · p. 8

7. Concluding Remarks · Appendix

Satori’s two-stage training framework uses format tuning to internalize a reasoning format and reinforcement learning to incentivize intrinsic reasoning; the authors propose several directions for future autoregressive-search research.

7. Concluding Remarks

  • Satori’s framework combines format tuning with large-scale reinforcement learning to enhance LLM reasoning capabilities.The authors describe the framework as having significant potential.Source: §7 · p. 9
  • Small-scale format tuning acts as a warm-up phase that helps the LLM policy internalize a specific reasoning format.Source: §7 · p. 9
  • Large-scale reinforcement learning plays a crucial role in incentivizing intrinsic reasoning abilities.Source: §7 · p. 9
  • One proposed direction is to develop reasoning formats with a broader range of meta-actions.Source: §7 · p. 9
  • The authors also propose designing more advanced reinforcement-learning algorithms for autoregressive search.Source: §7 · p. 9
  • Another proposed direction is extending the approach to general-domain tasks.Source: §7 · p. 9

A. Satori’s Demo Examples · B. Additional Related Work

Satori’s demo examples show it checking its reasoning, identifying errors in prior solutions, and trying alternative or simpler approaches in mathematical and out-of-domain tasks. A tabular-reasoning example also works through a mountain-peak prominence question.

A. Satori’s Demo Examples

  • Satori verifies intermediate steps in a math example and proceeds to the next reasoning step.Source: Figure 6 · p. 15
  • In math examples, Satori identifies an erroneous prior solution and proposes an alternative correct solution.One example also shows it verifying intermediate steps before recognizing the error.Source: Figure 7 · p. 16, Figure 10 · p. 18, A. Satori’s Demo Examples · p. 19
  • After verifying a previous math solution, Satori initiates a different solution.Source: Figure 8 · p. 17
  • After verifying a previous math solution, Satori further explores a simpler solution.Source: Figure 9 · p. 17
  • Out-of-domain examples show Satori noticing potential errors in intermediate steps, reflecting on its reasoning, and proposing alternative correct solutions.Some examples show it verifying early steps before identifying an error in the prior solution.Source: Figure 12 · p. 19, Figure 11 · p. 19, Figure 13 · p. 20
  • 2606.25 is the reported average prominence for Democratic Republic of the Congo peaks with elevation of at least 3000 meters in the tabular-reasoning example.The worked example reports a prominence sum of 10425 divided by 4.Source: A. Satori’s Demo Examples · p. 23

B.1. Post-training LLMs for Reasoning · B.2. Enabling LLMs with Searching Abilities

The sections survey post-training approaches to improve LLM reasoning and methods for enabling search, while identifying limitations in data generation, self-correction, and generalization. They contrast prompting and multi-model feedback with efforts to train single models to search.

B.1. Post-training LLMs for Reasoning

  • LLMs now achieve human-level or, in some cases, superhuman performance across reasoning benchmarks in diverse domains.The benchmarks cover mathematics, programming, logic, commonsense, algorithms, semi-structured data, scientific knowledge, and world knowledge.Source: §B.1 · p. 24
  • Post-training instruction tuning uses high-quality CoT-like reasoning chains from human annotation or stronger-model distillation, but annotation is costly and distillation caps student potential at the teacher’s level.Source: §B.1 · p. 24
  • Self-training reduces reliance on external resources but risks model collapse, while PPO-based reinforcement learning with PRM or ORM rewards has improved performance over supervised fine-tuning.Source: §B.1 · p. 24

B.2. Enabling LLMs with Searching Abilities

  • CoT prompting improves reasoning, but lacks error correction; later search prompting adds exploration, backtracking, and alternate paths without fundamentally improving intrinsic reasoning.Research also reports persistent difficulties with effective LLM self-correction.Source: §B.2 · p. 24
  • Trajectory-level search trains models to find errors in complete responses and produce improved answers using human-annotated revisions or model-generated training data.Source: §B.2 · p. 24
  • Step-level search enables fine-grained correction as mistakes occur, but approaches that rely on another model for feedback incur high deployment costs.Source: §B.2 · p. 24
  • SoS trains a single LLM to conduct tree search as a flattened string, but its effectiveness is shown mainly on simple symbolic tasks, leaving generalization to math word problems unexplored.Source: §B.2 · p. 24

C. Details about Data Synthesis Framework … D.1.1. PROMPT TEMPLATES

This section describes the COAT data-synthesis pipeline and its processing settings, then presents a prompt template for reviewing and correcting a continued solution.

C. Details about Data Synthesis Framework

  • The framework samples K initial trajectories per problem, checks final-answer correctness, and processes correct and incorrect trajectories through different feedback paths.Incorrect paths receive error correction and an exploration hint; correct paths receive verification feedback, after which the generator continues. Refinement repeats for up to m iterations, and meta-action tokens are inserted before reasoning steps.Source: C. Details about Data Synthesis… · p. 25
  • Reward-model scores select the top-k demonstrations, preserving both short-cut COAT paths and more complex paths with repeated reflection and exploration.Source: C. Details about Data Synthesis… · p. 26

D.1. Data Processing

  • The training data combines OpenMathInstruct2 and NuminaMath-COT, with invalid questions, answer-inconsistent examples, and duplicates removed, yielding approximately 550K samples.QwQ relabels questions for mutual-consistency filtering.Source: §D.1 · p. 27
  • Synthesis uses Qwen-2.5-Math-7B-Instruct, Llama-3.1-70B-Instruct, and Skywork-Reward-Llama-3.1-8B-v0.2, sampling 100 initial solutions and up to 16 demonstration trajectories over two refinement iterations.Invalid solutions are discarded, and four correct plus four incorrect solutions are selected for critic feedback.Source: §D.1 · p. 27
  • The synthetic trajectories span five types, from paths without critic feedback to paths combining verification, error correction, alternative exploration, and two rounds of self-reflection.The reward model selects the top-scoring sample of each type when available.Source: §D.1 · p. 27

D.1.1. PROMPT TEMPLATES

  • Prompt templates direct generators to produce initial solutions or continue correct and incorrect partial solutions, while critics verify correctness or identify and correct the first mistake.Generator prompts require brief English steps and a boxed final answer; correction prompts request both an explanation and an alternative step.Source: §D.1.1 · pp. 28–30
  • The response must identify the first potentially incorrect step in the student’s continuation, explain their own mistake, and avoid checking or criticizing the partner’s work.The prompt defines the continuation as completion based on the partner’s partial solution.Source: §D.1.1 · p. 31
  • The student must suggest an alternative corrective step without explicitly referring to the ground-truth solution, using the prescribed two-line format.The required format is “In Step <id>:” followed by a one-sentence explanation, then “Alternatively:” followed by a one-sentence suggested step.Source: §D.1.1 · p. 31
  • The student receives the question, ground-truth solution, partner’s partial solution, and their own continued solution as review context.The prompt frames the task as continuing a partner’s work before reviewing the resulting solution.Source: §D.1.1 · p. 31

D.1.2. SYNTHETIC DATA EXAMPLES · D.2. Format Tuning · D.3. Reinforcement Learning

The synthetic examples illustrate reasoning without reflection and with verification or error correction, while format tuning and reinforcement learning train COAT actions and self-improvement. The sections describe preference-based reward modeling, restart-buffer augmentation, and a second training round using distilled challenging trajectories.

D.1.2. SYNTHETIC DATA EXAMPLES

  • In the Type I no-reflection example, five couples comprise 10 people; with boats carrying 4 people each, 10/4 = 2.5, so rounding up requires 3 boats.Source: §D.1.2 · p. 31
  • In a Type III-I example, the solution corrects A for the positive-natural-number domain, obtains A ∩ B = {1, 2, 3}, and selects answer D.The correction excludes zero from A and retains B = [1, ∞).Source: §D.1.2 · p. 33
  • The Type III-II example shows multiple corrections: it revises sin C and the triangle-area calculation, ending with area 6.The example first replaces sin C = 2/5 with 2/√5, then identifies that the area formula should use sin B = √2/2.Source: §D.1.2 · p. 34

D.2. Format Tuning

  • Format tuning applies supervised fine-tuning to Qwen-2.5-Math-7B on 10K unique synthetic COAT questions and adds the <|continue|>, <|reflect|>, and <|explore|> tokens.Training uses a maximum of two epochs, a batch size of 128, and a maximum sequence length of 4096.Source: §D.2 · p. 35

D.3. Reinforcement Learning

  • A positive target reward margin improves reward-model performance, and the ORM preference data pairs correct and incorrect trajectories from sampled solutions.The preference set contains approximately 300K questions selected from 550K; ORM training uses PairWiseLoss with τ = 2.Source: §D.3 · p. 35
  • The RL set uses questions where the format-tuned model produces both correct and incorrect answers, with restart buffers and backtracking up to T = 2 to encourage reflection.For each question, the buffers draw one correct and one incorrect trajectory from the preference data.Source: §D.3 · p. 35
  • In the second round, the first-round policy generates trajectories for 240K questions, the 180K hardest are distilled, RAE is omitted, and sampling rises to temperature 1.2 with eight samples per prompt.The new fine-tuned checkpoint uses supervised fine-tuning on the selected trajectories, which include distilled reflection actions.Source: §D.3 · p. 36

D.4. Evaluation Details. · E. Additional Results

Evaluation uses greedy zero-shot CoT decoding and pass@1 across math, reasoning, coding, tabular, and STEM benchmarks. The passages describe evaluation details but provide no additional-results findings.

D.4. Evaluation Details.

  • All models use zero-shot CoT prompts, with Satori and its variants using Prompt Template 3, temperature 0 greedy decoding, and pass@1 accuracy.The supplied evaluation-details passages do not report additional-results comparisons or scores.Source: §D.4 · p. 36
  • MATH500 samples 500 MATH test problems, while GSM8K contains 1,319 test problems requiring multi-step grade-school math reasoning (Lightman et al., 2023; Cobbe et al., 2021).MATH500’s difficulty and subject distributions are described as representative of the full MATH test set; GSM8K problems require 2–8 reasoning steps.Source: §D.4 · p. 36
  • The math evaluations also include 674 open-ended text-only OlympiadBench problems (He et al., 2024), 40 AMC2023 problems, and 30 AIME2024 problems.OlympiadBench’s full benchmark is bilingual and multimodal, but the evaluation uses its text-only math competition subset.Source: §D.4 · p. 36
  • Logical reasoning is evaluated with FOLIO’s 203 test problems, 15K BoardgameQA problems, and 2,780 StrategyQA examples.These benchmarks cover first-order-logic-annotated conclusions, defeasible reasoning with conflicting information, and multi-hop questions with implicit strategies, respectively.Source: §D.4 · p. 36
  • CRUXEval tests code reasoning, understanding, and execution on 800 Python functions, using input-output prediction tasks (Gu et al., 2024).The functions are 3–13 lines long.Source: §D.4 · p. 36
  • TableBench evaluation uses 491 problems from its fact-checking and numerical-reasoning subsets (Wu et al., 2024a).The benchmark covers 18 fields across four categories; these two subsets were selected for simpler answer validation.Source: §D.4 · p. 36
  • MMLUProSTEM contributes 5,371 problems from six STEM subsets, excluding math as an in-domain task (Wang et al., 2024b).The subsets are physics, chemistry, computer science, engineering, biology, and economics.Source: §D.4 · p. 36

E.1. Ablation on Reflection Bonus … E.4. Offline Restart Buffer v.s. Online Restart Buffer

Ablations show that the reflection bonus, Restart and Explore (RAE), and ORM-based preference bonus each contribute to performance. The restart-buffer comparison favors offline sampling because it balances intermediate states from correct and incorrect trajectories.

E.1. Ablation on Reflection Bonus

  • With rreflect = 0.5 as the default, removing the reflection bonus slightly degrades performance on AMC2023 and AIME2024.The ablation compares the default bonus with rreflect = 0.Source: Table 6 · p. 37, §E.1 · p. 37

E.2. Ablation on Restart and Explore (RAE)

  • Removing RAE from RL training degrades performance, supporting its critical role in the training approach.RAE encourages self-reflection and improves the diversity of initial states.Source: Table 7 · p. 37, §E.2 · p. 37
  • RAE complements the reflection bonus by encouraging self-reflection and diversifying initial states.The RAE ablation removes this technique during RL training.Source: §E.2 · p. 37

E.3. Ablation on Preference Bonus

  • Removing the ORM preference bonus and relying on rule-based rewards plus reflection bonus degrades performance.The results support combining sparse rule-based rewards with dense ORM-based rewards.Source: Table 8 · p. 37, §E.3 · p. 37
  • The ORM preference bonus is used to mitigate sparse rewards by providing a dense reward signal.The ablation removes ORM and relies only on rule-based reward and reflection bonus.Source: §E.3 · p. 37

E.4. Offline Restart Buffer v.s. Online Restart Buffer

  • Online buffer collection after each PPO episode is suboptimal because most sampled trajectories are correct, imbalancing the intermediate states.The approach constructs a buffer from the updated policy and collects rollouts from it iteratively during PPO training.Source: §E.4 · p. 37
  • Offline restart-buffer sampling ensures balanced inclusion of intermediate states from correct and incorrect trajectories, avoiding online-collection bias.The buffer augments initial states with diverse intermediate states from both correct and incorrect reasoning paths.Source: §E.4 · pp. 37–38

Related papers

Loading 2502.02508v3…