PaperlayerView exact revision on arXiv

Source-linked AI summary

Self-rewarding correction for mathematical reasoning

Wei Xiong, Hanning Zhang, Chenlu Ye, Lichang Chen, Nan Jiang, Tong Zhang

arXiv:2502.19613v1cs.AIcs.LG

TL;DR

The paper addresses the difficulty of intrinsic self-correction, where LLMs must evaluate and revise reasoning without external reward feedback. It proposes a two-stage self-generated-data framework combining sequential rejection sampling and rule-based reinforcement learning. Experiments show improved self-correction over intrinsic methods, while the approach has training and inference limitations.

  • Problem

    LLMs often struggle to refine responses and decide when to stop without external reward feedback, despite external reward models increasing inference cost and complexity.

    Source: §1 · p. 1

  • Method

    The framework fine-tunes models on sequentially rejection-sampled CoT trajectories, then applies reinforcement learning with rule-based signals to enhance self-rewarding correction.

    Source: §1 · p. 2, §3.1 · p. 3

  • Results

    The framework significantly outperforms intrinsic self-correction in experiments, with reported analyses of correction behavior and learning dynamics.

    Source: §1 · p. 2, §4.1 · p. 8

  • Takeaways & Limitations

    A single LLM can integrate reasoning, evaluation, and correction, supporting inference-time self-correction without an external reward model.

    Source: §1 · p. 2, §6 · p. 15

    The method uses more inference tokens because of additional correction steps, and some RL rule designs can produce inconsistent reward-model accuracy.

    Source: §4.1 · p. 9, §5.6 · p. 15

Abstract

from arXiv · show

We study self-rewarding reasoning large language models (LLMs), which can simultaneously generate step-by-step reasoning and evaluate the correctness of their outputs during the inference time-without external feedback. This integrated approach allows a single model to independently guide its reasoning process, offering computational advantages for model deployment. We particularly focus on the representative task of self-correction, where models autonomously detect errors in their responses, revise outputs, and decide when to terminate iterative refinement loops. To enable this, we propose a two-staged algorithmic framework for constructing self-rewarding reasoning models using only self-generated data. In the first stage, we employ sequential rejection sampling to synthesize long chain-of-thought trajectories that incorporate both self-rewarding and self-correction mechanisms. Fine-tuning models on these curated data allows them to learn the patterns of self-rewarding and self-correction. In the second stage, we further enhance the models' ability to assess response accuracy and refine outputs through reinforcement learning with rule-based signals. Experiments with Llama-3 and Qwen-2.5 demonstrate that our approach surpasses intrinsic self-correction capabilities and achieves performance comparable to systems that rely on external reward models.

1. Introduction

The paper targets intrinsic self-correction, where LLMs must evaluate and revise their own reasoning without external reward feedback. It introduces a single-model framework and two-stage training procedure, reporting improved self-correction performance.

  • Intrinsic self-correction remains difficult because models lack external feedback for deciding whether to revise or terminate refinement.Source: §1 · p. 1
  • Self-rewarding reasoning unifies step-by-step generation, response evaluation, and revision within one LLM.Source: §1 · p. 2
  • The proposed framework uses sequential rejection sampling to create long CoT trajectories encoding self-rewarding and self-correction behaviors.Source: §1 · p. 2
  • A second training stage applies reinforcement learning with rule-based signals to strengthen correctness assessment and response refinement.Source: §1 · p. 2
  • Self-rewarding correction significantly outperforms intrinsic self-correction in experiments, with ablations analyzing its learning dynamics.Source: §1 · p. 2

2. Related Work

Prior work studies self-rewarding alignment, intrinsic self-correction, and rule-based reinforcement learning, but this paper combines self-rewarding with inference-time correction for reasoning-focused LLMs. It also emphasizes a sequential-sampling warm-up followed by rule-based RL.

Self-rewarding alignment

  • Self-rewarding alignment unifies generation and evaluation abilities in one LLM, often using iterative DPO-type self-labeling during training.Source: §2 · p. 2

Self-correction

  • Intrinsic self-correction studies find that current reasoning models fail to self-correct purely through intrinsic reasoning without additional training.Source: §2 · p. 2
  • This work frames self-rewarding as an inference-time mechanism that lets models selectively revise attempts and avoid unnecessary iterations.Source: §2 · p. 2
  • Unlike Kumar et al.'s two-turn RLOO approach, the paper uses sequential rejection sampling for long CoT warm-up data and then rule-based RL.Source: §2 · pp. 2–3

Rule-based RL for LLMs mathematical reasoning

  • The paper notes that similar mathematical-reasoning improvements are not observed in Llama under the same algorithms and data, while curated SFT warm-up enables more reliable self-correction patterns.Source: §2 · p. 3

3. Self-rewarding Reasoning Language Models

The framework models self-rewarding correction as iterative reasoning in which an LLM evaluates each response and either stops or generates a refinement. Training combines sequentially sampled demonstrations with KL-regularized reinforcement learning using correctness-based rewards.

Self-rewarding reasoning process

  • The self-rewarding process is a multi-turn MDP: the LLM generates an initial attempt, evaluates it, and continues refinement unless it judges the answer correct.Source: §3 · p. 3
  • A ground-truth verifier determines response correctness, using the ToRA verification script with SymPy for symbolic mathematics.Source: §3 · p. 3

Two-stage training framework

  • The first training stage fine-tunes an initial LLM on sequential-rejection-sampled trajectories to produce a reference model with self-rewarding reasoning abilities.Source: §3 · p. 3, §3.1 · p. 4

KL-regularized reinforcement learning

  • The second stage refines the self-rewarding IFT model with RL methods including multi-turn DPO, using rule-based oracle rewards based primarily on final-answer correctness.Source: §3 · p. 3, §3.2 · pp. 5–6

Self-rewarding by token prediction

  • Self-rewarding evaluations use reasoning plus designated tokens such as “Yes”/“No” or “[VERIFY] correct”/“[VERIFY] wrong,” with evaluation tokens sampled during inference.Source: §3.1 · pp. 3–4

Data collection by sequential rejection sampling

  • The data process samples initial responses, aligned self-evaluations, and correction completions for both wrong and correct attempts.Source: §3.1 · p. 4

Self-rewarding instruction-following fine-tuning

  • Multi-task training can create stability issues and slightly reduce first-round performance, motivating additional training on correct initial attempts.Source: §3.1 · p. 5

4. Experiment Results

Experiments evaluate mathematical reasoning, self-correction, and reward-model accuracy across three benchmarks. Self-rewarding models generally outperform intrinsic and single-turn baselines, while RL dynamics reveal a shift toward stronger first attempts and greater inference cost.

Evaluation setup

  • The evaluation measures first-attempt accuracy, final accuracy, accuracy improvement, and class-dependent reward-model accuracy.The benchmarks are MATH500, OlympiadBench, and Minerva Math.Source: §4 · p. 6

Intrinsic self-correction

  • Intrinsic self-correction typically reduces final accuracy because models modify responses without reliably deciding when to refine or stop.With gold rewards and prompting alone, incorrect-to-correct improvement on MATH-500 is 1.4%.Source: §4.1 · p. 8

Baseline comparisons

  • STaR/RAFT raises incorrect-to-correct transitions on MATH500 from 1.4% to 5.0%, but produces 6.2% correct-to-incorrect transitions.Adding correct-to-correct trajectories makes STaR/RAFT+ more conservative and ultimately degrades test accuracy.Source: §4.1 · p. 8

Self-rewarding correction

  • Self-rewarding models consistently improve final accuracy and correction gains across tasks, while self-rewarding IFT reduces correct-to-incorrect transitions to 0.4%.The corresponding values are 15.4% for intrinsic self-correction and 3.8% for STaR/RAFT+.Source: §4.1 · p. 8

Single-turn comparisons

  • Self-rewarding IFT + PPO reaches 43.4% final accuracy on OlympiadBench and 38.4% on Minerva Math, versus 39.5% and 33.1% for single-turn counterparts.With DPO, the self-rewarding models reach 78.6% on MATH500, 40.1% on OlympiadBench, and 34.6% on Minerva Math, versus 76.8%, 39.0%, and 31.6%.Source: §4.1 · pp. 8–9

Computational cost

  • Self-rewarding models use more inference tokens because of the additional correction step.The study therefore examines self-rewarding correction under scaled test-time compute budgets.Source: §4.1 · p. 9

RL analysis

  • PPO reaches 43.4% final accuracy on OlympiadBench compared with 40.1% for DPO, while PPO and DPO do not consistently improve reward-model accuracy.Reward-model training explores trade-offs between recognizing correct and incorrect trajectories.Source: §4.1 · p. 9

RL learning dynamics

  • RL training initially improves both first-attempt and final accuracy, but later gains mainly reflect higher first-attempt accuracy and reduced reliance on self-correction.Generation length first decreases, then increases as reflection and correction are encouraged, and later decreases as first-attempt correctness improves.Source: §4.1 · pp. 9–10

5. More Experiment Results with a Two-turn Conversation Framework and Llama Models

Llama experiments show intrinsic self-correction is unreliable, while self-rewarding models more consistently correct errors and preserve correct answers. Additional analyses examine efficiency, reward-model behavior, data-distribution effects, and the trade-off between correction and conservatism.

Main results

  • Intrinsic self-correction and STaR/RAFT-like training often modify responses regardless of correctness, reducing turn-2 accuracy for stronger models.For weaker models, these methods can help when most first attempts are incorrect.Source: §5.3 · p. 11
  • Self-rewarding IFT consistently improves turn-1 accuracy by correcting errors while preserving responses that were already correct.The authors report similar trends across Llama experiments and Qwen experiments.Source: §5.3 · p. 11
  • 9.7% ∆i→c(t1, t2) is achieved by self-rewarding IFT, compared with 8.8% for the baseline Llama-3-SFT under oracle self-rewarding signals.This comparison is intended to isolate correction ability from reward-signal quality.Source: §5.3 · p. 11

Inference efficiency

  • Early termination reduces inference cost: the M-DPO-aligned model averages 1.65 samples per MATH trajectory and 1.25 per GSM8K trajectory.The models terminate after the first response when sufficiently confident in its correctness.Source: §5.4 · p. 12

Reward-model comparison

  • Self-rewarding IFT outperforms self-correction with an external ORM in both turn-2 accuracy and ∆(t1, t2).The comparison uses an external ORM trained on the same dataset.Source: §5.5 · p. 12

Data distribution

  • Increasing correct trajectories biases the ORM toward correct predictions, improving correct-trajectory recognition while reducing incorrect-trajectory recognition from 72.1%/75.3% to 63.6%/82.4%.The results expose a trade-off between class-dependent reward-model accuracies.Source: §5.5 · p. 13

6. Conclusion and Future Research Direction

The framework combines self-rewarding instruction fine-tuning and reinforcement learning to improve self-correction and computational efficiency, while leaving several directions for further improvement.

Conclusion

  • The self-rewarding reasoning framework integrates self-rewarding IFT and reinforcement learning to help LLMs detect reasoning errors and refine responses.The method uses historical attempts and self-rewarding signals during correction.Source: §6 · p. 15
  • The framework significantly outperforms intrinsic self-correction, supporting its effectiveness as an approach for enhancing LLM reasoning.Source: §6 · p. 15

Future Research Direction

  • Current models have lower reward-model accuracy than external ORMs, likely because of distribution shifts and model-capacity limitations.The authors suggest model merging as one possible way to address these issues.Source: §6 · p. 15
  • Extending correction from turn-based to step-wise feedback may provide greater advantages and a more scalable, dynamic reasoning approach.The proposed direction is related to outcome-supervised and process-supervised rewards.Source: §6 · p. 16

Impact Statement

The paper presents self-rewarding reasoning as a way to integrate reward modeling and generation within one LLM for mathematical reasoning and complex decision-making problems.

Impact Statement

  • The proposed framework integrates the reward model and generator into a single LLM for mathematical reasoning.Source: Impact Statement · p. 16
  • The framework is intended to help build stronger LLMs for complex decision-making problems and contribute to broader societal welfare.Source: Impact Statement · p. 16

A. Extended Related Works

Prior work develops mathematical reasoning through chain-of-thought prompting, external tools, reward-ranked fine-tuning, and reinforcement learning methods such as PPO and GRPO.

LLMs for Mathematical Problem Solving

  • Chain-of-thought prompting is used to elicit step-by-step reasoning in mathematical problem-solving tasks.Related approaches also equip LLMs with external tools such as calculators.Source: A. Extended Related Works · p. 21

RLHF for Mathematical Problem Solving

  • Reward-ranked or rejection-sampling fine-tuning generates synthetic trajectories and selects samples with reward models for further fine-tuning.These methods may use on-policy self-improvement or off-policy knowledge distillation.Source: A. Extended Related Works · p. 21
  • Deep-reinforcement-learning approaches include PPO and Reinforce variants for training mathematical reasoning models.Source: A. Extended Related Works · p. 21
  • GRPO is described as improving multi-turn mathematical problem solving in the chain-of-thought format.Source: A. Extended Related Works · p. 21

B. Missing Experimental Details

The appendix documents the experiment prompt templates and software environment, including distinct prompts for self-rewarding, intrinsic self-correction, and gold-test conditions.

Prompt Template

Python Experiment Environment

  • The appendix reports software versions because package and virtual-machine choices can influence evaluation results without affecting the overall trend.For Llama IFT and M-DPO training, it lists transformers 4.44.1 and torch 2.1.2, along with axolotl 0.4.1 for IFT.Source: B. Missing Experimental Details · p. 21

C. Additional Experimental Results

This section adds ablation studies and evaluation results for the self-rewarding reasoning framework. Table 10 reports main results on MATH at a test temperature of 0.7.

  • Additional ablation studies and evaluation results provide a more comprehensive understanding of the framework and proposed algorithms.Source: C. Additional Experimental Results · p. 22
  • Table 10 reports main results for different methods on the MATH test set at a test temperature of 0.7.Source: Table 10 · p. 22

D. Examples

The examples illustrate self-correction across mathematical tasks: models identify incorrect answers, revise them, and verify corrected results. They also include cases where an initially correct answer is verified correctly.

Examples

MATH500

  • 90° is accepted after recognizing that a zero dot product makes the direction vectors orthogonal.The self-evaluation changes from wrong to correct after the geometric reasoning is rechecked.Source: D. Examples · p. 24

OlympiadBench

  • 0 is confirmed as the difference between counts of positive integers with even and odd binary weight.The reasoning and Python confirmation both support the final answer, which is marked correct by self-evaluation.Source: D. Examples · p. 26

Related papers

Loading 2502.19613v1…