PaperlayerView exact revision on arXiv

Source-linked AI summary

Self-rewarding correction for mathematical reasoning

Wei Xiong, Hanning Zhang, Chenlu Ye, Lichang Chen, Nan Jiang, Tong Zhang

arXiv:2502.19613v1cs.AIcs.LG

TL;DR

The paper addresses the difficulty of intrinsic self-correction, where LLMs must evaluate and revise reasoning without external reward feedback. It proposes a two-stage self-generated-data framework combining sequential rejection sampling and rule-based reinforcement learning. Experiments show improved self-correction over intrinsic methods, while the approach has training and inference limitations.

  • Problem

    LLMs often struggle to refine responses and decide when to stop without external reward feedback, despite external reward models increasing inference cost and complexity.

  • Method

    The framework fine-tunes models on sequentially rejection-sampled CoT trajectories, then applies reinforcement learning with rule-based signals to enhance self-rewarding correction.

  • Results

    The framework significantly outperforms intrinsic self-correction in experiments, with reported analyses of correction behavior and learning dynamics.

  • Takeaways & Limitations

    A single LLM can integrate reasoning, evaluation, and correction, supporting inference-time self-correction without an external reward model.

  • Takeaways & Limitations

    The method uses more inference tokens because of additional correction steps, and some RL rule designs can produce inconsistent reward-model accuracy.

Abstract

from arXiv · show

We study self-rewarding reasoning large language models (LLMs), which can simultaneously generate step-by-step reasoning and evaluate the correctness of their outputs during the inference time-without external feedback. This integrated approach allows a single model to independently guide its reasoning process, offering computational advantages for model deployment. We particularly focus on the representative task of self-correction, where models autonomously detect errors in their responses, revise outputs, and decide when to terminate iterative refinement loops. To enable this, we propose a two-staged algorithmic framework for constructing self-rewarding reasoning models using only self-generated data. In the first stage, we employ sequential rejection sampling to synthesize long chain-of-thought trajectories that incorporate both self-rewarding and self-correction mechanisms. Fine-tuning models on these curated data allows them to learn the patterns of self-rewarding and self-correction. In the second stage, we further enhance the models' ability to assess response accuracy and refine outputs through reinforcement learning with rule-based signals. Experiments with Llama-3 and Qwen-2.5 demonstrate that our approach surpasses intrinsic self-correction capabilities and achieves performance comparable to systems that rely on external reward models.

1. Introduction

The paper targets intrinsic self-correction, where LLMs must evaluate and revise their own reasoning without external reward feedback. It introduces a single-model framework and two-stage training procedure, reporting improved self-correction performance.

  • Intrinsic self-correction remains difficult because models lack external feedback for deciding whether to revise or terminate refinement.
  • Self-rewarding reasoning unifies step-by-step generation, response evaluation, and revision within one LLM.
  • The proposed framework uses sequential rejection sampling to create long CoT trajectories encoding self-rewarding and self-correction behaviors.
  • A second training stage applies reinforcement learning with rule-based signals to strengthen correctness assessment and response refinement.
  • Self-rewarding correction significantly outperforms intrinsic self-correction in experiments, with ablations analyzing its learning dynamics.

2. Related Work

Prior work studies self-rewarding alignment, intrinsic self-correction, and rule-based reinforcement learning, but this paper combines self-rewarding with inference-time correction for reasoning-focused LLMs. It also emphasizes a sequential-sampling warm-up followed by rule-based RL.

  • Self-rewarding alignment: Self-rewarding alignment unifies generation and evaluation abilities in one LLM, often using iterative DPO-type self-labeling during training.
  • Self-correction: Intrinsic self-correction studies find that current reasoning models fail to self-correct purely through intrinsic reasoning without additional training.
  • Self-correction: This work frames self-rewarding as an inference-time mechanism that lets models selectively revise attempts and avoid unnecessary iterations.
  • Self-correction: Unlike Kumar et al.'s two-turn RLOO approach, the paper uses sequential rejection sampling for long CoT warm-up data and then rule-based RL.
  • Rule-based RL for LLMs mathematical reasoning: The paper notes that similar mathematical-reasoning improvements are not observed in Llama under the same algorithms and data, while curated SFT warm-up enables more reliable self-correction patterns.

3. Self-rewarding Reasoning Language Models

The framework models self-rewarding correction as iterative reasoning in which an LLM evaluates each response and either stops or generates a refinement. Training combines sequentially sampled demonstrations with KL-regularized reinforcement learning using correctness-based rewards.

  • Self-rewarding reasoning process: The self-rewarding process is a multi-turn MDP: the LLM generates an initial attempt, evaluates it, and continues refinement unless it judges the answer correct.
  • Self-rewarding reasoning process: A ground-truth verifier determines response correctness, using the ToRA verification script with SymPy for symbolic mathematics.
  • Two-stage training framework: The first training stage fine-tunes an initial LLM on sequential-rejection-sampled trajectories to produce a reference model with self-rewarding reasoning abilities.
  • KL-regularized reinforcement learning: The second stage refines the self-rewarding IFT model with RL methods including multi-turn DPO, using rule-based oracle rewards based primarily on final-answer correctness.
  • Self-rewarding by token prediction: Self-rewarding evaluations use reasoning plus designated tokens such as “Yes”/“No” or “[VERIFY] correct”/“[VERIFY] wrong,” with evaluation tokens sampled during inference.
  • Data collection by sequential rejection sampling: The data process samples initial responses, aligned self-evaluations, and correction completions for both wrong and correct attempts.
  • Self-rewarding instruction-following fine-tuning: Multi-task training can create stability issues and slightly reduce first-round performance, motivating additional training on correct initial attempts.

4. Experiment Results

Experiments evaluate mathematical reasoning, self-correction, and reward-model accuracy across three benchmarks. Self-rewarding models generally outperform intrinsic and single-turn baselines, while RL dynamics reveal a shift toward stronger first attempts and greater inference cost.

  • Evaluation setup: The evaluation measures first-attempt accuracy, final accuracy, accuracy improvement, and class-dependent reward-model accuracy.The benchmarks are MATH500, OlympiadBench, and Minerva Math.
  • Intrinsic self-correction: Intrinsic self-correction typically reduces final accuracy because models modify responses without reliably deciding when to refine or stop.With gold rewards and prompting alone, incorrect-to-correct improvement on MATH-500 is 1.4%.
  • Baseline comparisons: STaR/RAFT raises incorrect-to-correct transitions on MATH500 from 1.4% to 5.0%, but produces 6.2% correct-to-incorrect transitions.Adding correct-to-correct trajectories makes STaR/RAFT+ more conservative and ultimately degrades test accuracy.
  • Self-rewarding correction: Self-rewarding models consistently improve final accuracy and correction gains across tasks, while self-rewarding IFT reduces correct-to-incorrect transitions to 0.4%.The corresponding values are 15.4% for intrinsic self-correction and 3.8% for STaR/RAFT+.
  • Single-turn comparisons: Self-rewarding IFT + PPO reaches 43.4% final accuracy on OlympiadBench and 38.4% on Minerva Math, versus 39.5% and 33.1% for single-turn counterparts.With DPO, the self-rewarding models reach 78.6% on MATH500, 40.1% on OlympiadBench, and 34.6% on Minerva Math, versus 76.8%, 39.0%, and 31.6%.
  • Computational cost: Self-rewarding models use more inference tokens because of the additional correction step.The study therefore examines self-rewarding correction under scaled test-time compute budgets.
  • RL analysis: PPO reaches 43.4% final accuracy on OlympiadBench compared with 40.1% for DPO, while PPO and DPO do not consistently improve reward-model accuracy.Reward-model training explores trade-offs between recognizing correct and incorrect trajectories.
  • RL learning dynamics: RL training initially improves both first-attempt and final accuracy, but later gains mainly reflect higher first-attempt accuracy and reduced reliance on self-correction.Generation length first decreases, then increases as reflection and correction are encouraged, and later decreases as first-attempt correctness improves.

5. More Experiment Results with a Two-turn Conversation Framework and Llama Models

Llama experiments show intrinsic self-correction is unreliable, while self-rewarding models more consistently correct errors and preserve correct answers. Additional analyses examine efficiency, reward-model behavior, data-distribution effects, and the trade-off between correction and conservatism.

  • Main results: Intrinsic self-correction and STaR/RAFT-like training often modify responses regardless of correctness, reducing turn-2 accuracy for stronger models.For weaker models, these methods can help when most first attempts are incorrect.
  • Main results: Self-rewarding IFT consistently improves turn-1 accuracy by correcting errors while preserving responses that were already correct.The authors report similar trends across Llama experiments and Qwen experiments.
  • Main results: 9.7% ∆i→c(t1, t2) is achieved by self-rewarding IFT, compared with 8.8% for the baseline Llama-3-SFT under oracle self-rewarding signals.This comparison is intended to isolate correction ability from reward-signal quality.
  • Inference efficiency: Early termination reduces inference cost: the M-DPO-aligned model averages 1.65 samples per MATH trajectory and 1.25 per GSM8K trajectory.The models terminate after the first response when sufficiently confident in its correctness.
  • Reward-model comparison: Self-rewarding IFT outperforms self-correction with an external ORM in both turn-2 accuracy and ∆(t1, t2).The comparison uses an external ORM trained on the same dataset.
  • Data distribution: Increasing correct trajectories biases the ORM toward correct predictions, improving correct-trajectory recognition while reducing incorrect-trajectory recognition from 72.1%/75.3% to 63.6%/82.4%.The results expose a trade-off between class-dependent reward-model accuracies.

6. Conclusion and Future Research Direction

The framework combines self-rewarding instruction fine-tuning and reinforcement learning to improve self-correction and computational efficiency, while leaving several directions for further improvement.

  • Conclusion: The self-rewarding reasoning framework integrates self-rewarding IFT and reinforcement learning to help LLMs detect reasoning errors and refine responses.The method uses historical attempts and self-rewarding signals during correction.
  • Conclusion: The framework significantly outperforms intrinsic self-correction, supporting its effectiveness as an approach for enhancing LLM reasoning.
  • Future Research Direction: Current models have lower reward-model accuracy than external ORMs, likely because of distribution shifts and model-capacity limitations.The authors suggest model merging as one possible way to address these issues.
  • Future Research Direction: Extending correction from turn-based to step-wise feedback may provide greater advantages and a more scalable, dynamic reasoning approach.The proposed direction is related to outcome-supervised and process-supervised rewards.

Impact Statement

The paper presents self-rewarding reasoning as a way to integrate reward modeling and generation within one LLM for mathematical reasoning and complex decision-making problems.

  • Impact Statement: The proposed framework integrates the reward model and generator into a single LLM for mathematical reasoning.
  • Impact Statement: The framework is intended to help build stronger LLMs for complex decision-making problems and contribute to broader societal welfare.

A. Extended Related Works

Prior work develops mathematical reasoning through chain-of-thought prompting, external tools, reward-ranked fine-tuning, and reinforcement learning methods such as PPO and GRPO.

  • LLMs for Mathematical Problem Solving: Chain-of-thought prompting is used to elicit step-by-step reasoning in mathematical problem-solving tasks.Related approaches also equip LLMs with external tools such as calculators.
  • RLHF for Mathematical Problem Solving: Reward-ranked or rejection-sampling fine-tuning generates synthetic trajectories and selects samples with reward models for further fine-tuning.These methods may use on-policy self-improvement or off-policy knowledge distillation.
  • RLHF for Mathematical Problem Solving: Deep-reinforcement-learning approaches include PPO and Reinforce variants for training mathematical reasoning models.
  • RLHF for Mathematical Problem Solving: GRPO is described as improving multi-turn mathematical problem solving in the chain-of-thought format.

B. Missing Experimental Details

The appendix documents the experiment prompt templates and software environment, including distinct prompts for self-rewarding, intrinsic self-correction, and gold-test conditions.

  • Prompt Template: The experiments use a prompt template that mainly follows Kumar et al. (2024) with slight modifications.
  • Prompt Template: The self-rewarding prompt tells the model its initial response was self-evaluated as incorrect before requesting correction and rewriting.
  • Prompt Template: The intrinsic self-correction prompt requests correction and rewriting without stating that an external or self-rewarding evaluation marked the response incorrect.
  • Prompt Template: The gold-test prompt explicitly states that the initial response was evaluated as incorrect before requesting a revised solution.
  • Python Experiment Environment: The appendix reports software versions because package and virtual-machine choices can influence evaluation results without affecting the overall trend.For Llama IFT and M-DPO training, it lists transformers 4.44.1 and torch 2.1.2, along with axolotl 0.4.1 for IFT.

C. Additional Experimental Results

This section adds ablation studies and evaluation results for the self-rewarding reasoning framework. Table 10 reports main results on MATH at a test temperature of 0.7.

  • Additional ablation studies and evaluation results provide a more comprehensive understanding of the framework and proposed algorithms.
  • Table 10 reports main results for different methods on the MATH test set at a test temperature of 0.7.

D. Examples

The examples illustrate self-correction across mathematical tasks: models identify incorrect answers, revise them, and verify corrected results. They also include cases where an initially correct answer is verified correctly.

  • Examples: The examples include prompts, model responses, intermediate calculations, final answers, and self-evaluation labels.The examples span MATH500, Minerva Math, and OlympiadBench cases.
  • MATH500: 90° is accepted after recognizing that a zero dot product makes the direction vectors orthogonal.The self-evaluation changes from wrong to correct after the geometric reasoning is rechecked.
  • OlympiadBench: 0 is confirmed as the difference between counts of positive integers with even and odd binary weight.The reasoning and Python confirmation both support the final answer, which is marked correct by self-evaluation.
Loading 2502.19613v1…