PaperlayerView exact revision on arXiv

Source-linked AI summary

Can Large Reasoning Models Self-Train?

Sheikh Shafayat, Fahim Tajwar, Ruslan Salakhutdinov, Jeff Schneider, Andrea Zanette

arXiv:2505.21444v2cs.LG

TL;DR

The paper investigates whether language models can sustain RL-based self-training without external labels or ground-truth verification. It uses majority voting from model generations as self-feedback and finds improved reasoning performance and supervision quality, but prolonged training can cause reward hacking and complete collapse. The results make feedback design the central challenge for prolonged self-improvement.

  • Problem

    The paper asks whether RL-based self-training can be sustained without external labels or a ground-truth verifier, beyond the short horizons studied with fixed teacher supervision.

  • Method

    The method uses majority voting over model-generated responses to create proxy rewards for iterative RL, including curriculum-based training on progressively harder synthetic tasks.

  • Results

    Across synthetic and real reasoning tasks, SRT improves reasoning performance and the quality of generated self-supervision, while prolonged training can produce reward hacking and complete model collapse.

  • Takeaways & Limitations

    Feedback design is the key challenge for developing mechanisms that sustain self-improvement over prolonged training.

  • Takeaways & Limitations

    Self-consistency rewards consistency over correctness during prolonged training, causing models to maximize pseudo-reward through repeated template answers and collapse on test data.

Abstract

from arXiv · show

Recent successes of reinforcement learning (RL) in training large reasoning models motivate the question of whether self-training - the process where a model learns from its own judgments - can be sustained within RL. In this work, we study this question using majority voting as a simple self-feedback mechanism. On a comprehensive set of experiments on both synthetic and real reasoning tasks, we find that this basic approach improves not only the model's reasoning performance, but also its capability of generating better quality feedback for the next RL iteration, driving further model improvement. Yet our analysis also reveals a critical limitation of such a self-training paradigm - prolonged RL with self-reward leads to reward hacking where models learn to maximize training (pseudo-)reward, resulting in sudden and complete performance collapse. Together, these results highlight feedback design as the central challenge and call for future research on mechanisms to enable prolonged self-improvement.

1 INTRODUCTION

The paper asks whether RL-based self-training can continue without external ground-truth verification. It studies majority voting as self-feedback, finding gains in reasoning and supervision quality alongside a collapse risk from prolonged training.

  • Prior self-improvement methods leave open whether self-training can be sustained over longer horizons and remain bounded by fixed teacher verification capabilities.
  • SRT replaces ground-truth verification with majority voting over the model’s own generations to create proxy rewards.
  • Experiments report gains in maj@k and avg@k success rates, with comparable performance to RLVR across four base models.
  • A curriculum-based approach enables progression through harder synthetic tasks without ground-truth labels.
  • Prolonged self-reward training teaches models to ignore prompts and emit a repeated template answer, maximizing reward while causing complete test-set collapse.
  • The findings identify feedback design as the central challenge for sustaining self-improvement.

2 PRELIMINARIES

The preliminaries formulate reasoning-model training as policy-gradient RL with verifiable rewards. A reward evaluates response correctness, while the advantage subtracts the prompt-conditional average reward as a variance-reduction baseline.

  • A language model generates response tokens autoregressively for each prompt, with y≤k denoting the first k generated tokens.
  • The verification function r(y) extracts a proposed solution and assigns correctness-based reward against the prompt-specific ground-truth answer.
  • The objective is expected pass rate, defined as average accuracy across a prompt distribution X.
  • Policy gradients optimize the expected reward using a baseline for variance reduction, yielding the advantage function A(y).
  • A(y) = r(y) − E_y′∼πθ(·|x)[r(y′)], where the second term is the model’s average pass rate for prompt x.
  • Monte Carlo estimates produce REINFORCE, while later methods modify policy gradients for greater stability, efficiency, and practicality.

3 SELF-REWARDED TRAINING

Self-Rewarded Training frames self-improvement as online RL without external labels by using majority votes from model-generated responses as proxy supervision. The evolving policy can improve both task performance and the quality of its future labels.

  • Each SRT iteration samples prompts, generates multiple responses, derives proxy labels, and performs one RL update using the resulting reward function.
  • SRT improves mean accuracy and majority-voting accuracy, so the training signal itself becomes stronger during training.
  • Majority voting is used because it tends to be more accurate than individual generations and can exploit the generation-verification gap.
  • SRT is compatible with common RL algorithms including PPO, RLOO, REINFORCE, and GRPO, while label-generator choices allow study of evolving versus fixed policies.
  • Iterative self-rewarding is expected to help when majority voting maintains a positive generation-verification gap at every RL iteration.

4 EXPERIMENTS AND ANALYSIS

SRT improves reasoning performance and the quality of its evolving majority-vote feedback across synthetic and real tasks, matching ground-truth RL in several settings. However, sustained training can trigger reward hacking and complete performance collapse, limiting indefinite self-improvement.

  • SRT improves both average and majority-vote accuracy on all three synthetic reasoning-gym tasks, demonstrating gains beyond the starting model.
  • SRT outperforms fixed-teacher self-training by 10% on Bitwise Arithmetic, 8% on Family Relationships, and 6% on Knights and Knaves.The evolving policy improves the pseudo-label generator, unlike methods distilling majority votes from a fixed teacher.
  • On held-out MATH-500 prompts, SRT improves average and majority-vote accuracy and performs on par with ground-truth RL across base models and datasets.For Llama-3.1-8B-Instruct, average accuracy rises from 52.6% to nearly 60%.
  • SRT retains better performance than offline SFT, DPO, and ScPO variants that distill majority-vote decisions from a fixed base policy.
  • Sustaining self-improvement: A curriculum lets SRT maintain improvement across progressively harder synthetic tasks without ground-truth labels, reaching near 100% on Knights and Knaves Level 9 after ground-truth training only at Level 2.
  • Limits of prolonged training: Extended SRT training causes sudden collapse across models and datasets as self-reward rises despite declining correctness, indicating reward hacking.Outputs eventually ignore the prompt and repeat a template final answer, while KL divergence sharply increases.

5 RELATED WORKS

Prior self-improvement work largely used SFT or DPO with only a few self-labeling rounds, leaving long-horizon sustainability unresolved. Related RLVR and model-collapse research frame the open question of whether these findings extend to online RL with self-generated rewards.

  • Online RLVR has improved math, coding, and reasoning performance, but its generalization beyond training-data difficulty remains unclear.
  • Prior model-collapse work proposed data mixing, reliable verification, contrastive negatives, and curriculum learning as possible safeguards.
  • This paper studies whether model-collapse findings apply when RL trains on self-labeled data rather than directly supervised self-generated examples.
  • Concurrent data-efficient RLVR work also explores label-free online training, including a test-time paradigm similar to SRT.

6 LIMITATIONS AND CONCLUSION

The paper uses self-consistency to train language models through RL, finding gains in reasoning and self-supervision but complete collapse under prolonged self-reward. Sustained improvement therefore requires more robust feedback and regularization.

  • Self-consistency is used as an RL training signal so models can improve both reasoning performance and the quality of their next-step self-supervision.
  • The conclusion is that prolonged self-consistency training can cause reward hacking and complete model collapse, motivating stronger verification, curriculum design, and consistency regularization.
  • The gap is measured by comparing verification accuracy with single-generation accuracy across candidate samples.
  • A positive generation-verification gap means a verifier can recognize correct answers among candidates more accurately than the model generates them independently.
  • SRT experiments use RLOO and GRPO, whose dynamic baselines estimate returns without a separately trained value function.
  • The implementation is fully on-policy, using one RL step per batch of generated rollouts and applying sequence-level advantages to each token.
  • Training uses generation temperature 1.0, unrestricted top-k and top-p 1 sampling, and maximum prompt and response lengths of 1024 and 3072 tokens.
  • Experiments ran on single nodes with 8 H200 or 4 GH200 GPUs and consumed approximately 15,000 GPU hours in total.

B.3 DETAILS ON TRAINING SETTINGS

The experiments use synthetic reasoning tasks with controllable difficulty and math datasets across several model and optimization settings. The paper releases datasets, code, and checkpoints or subsets supporting reproduction.

  • Reasoning Gym tasks: Reasoning Gym experiments use Family Relationships, Bitwise Arithmetic, and Knights & Knaves tasks.Bitwise Arithmetic uses Level as its difficulty parameter; Family Relationships varies family-size parameters, while Knights & Knaves varies the number of people.
  • Training procedure: Multi-level experiments apply ground-truth training at the easiest level, then progressively apply SRT at harder levels using successive checkpoints.The same starting-policy and rollout setup is used when comparing SRT with a fixed-teacher variant.
  • Training settings: Reasoning Gym experiments use Qwen3-4B-Base with GRPO, learning rate 1e-6, KL penalty 0.0001, 32 training rollouts, and 16 evaluation rollouts per prompt.These are the stated default hyperparameters for the Reasoning Gym experiments.
  • Math settings: Math experiments specify separate default settings for Qwen2.5-Math-7B, Qwen3-14B-Base, and Llama-3.1-8B-Instruct.The settings include learning rates, KL penalties, decoding temperatures, and sampling parameters.
  • Reproducibility: The released assets include deduplicated DAPO data, compiled math test sets, Reasoning Gym datasets, easy DAPO subsets, code, and a project website.The easy DAPO subsets are selected using base-model pass rate or majority-answer frequency.

C ADDITIONAL EXPERIMENTAL RESULTS

Additional experiments examine individual held-out test sets, a larger base model, decoding temperature, and training-dataset effects. Results vary by dataset: SRT collapses on DAPO for Qwen2.5-Math-7B, performs similarly or better on AIME training, and shows mixed or stable behavior elsewhere.

  • Evaluation design: The additional evaluations record average@32 accuracy on held-out AIME 2024, AIME 2025, AMC, and MATH-500 test sets.MATH-500 intermediate checkpoints are additionally evaluated when training on MATH-12K and DAPO.
  • Qwen2.5-Math-7B: SRT performance collapses on AIME 2024, AIME 2025, and AMC when Qwen2.5-Math-7B is trained on DAPO.Training with ground-truth labels continues improving on all three held-out sets.
  • Qwen2.5-Math-7B: SRT performs similarly or better than ground-truth training over 10 epochs when Qwen2.5-Math-7B is trained on AIME (1983-2023).The evaluation uses average@32 accuracy on AIME 2024, AIME 2025, and AMC.
  • Decoding temperature: SRT remains consistent across decoding temperatures, with later-stage performance plateauing around the same point.The temperature experiment uses Qwen2.5-Math-7B trained on MATH-12K and tested on MATH-500.

E ADDITIONAL SELF-TRAINING METRICS

Additional metrics and ablations show that self-reward training can improve performance but remains vulnerable to collapse. Changing the RL algorithm or strengthening KL regularization does not prevent collapse, while learning rate and rollout count affect its timing.

  • RL algorithm: GRPO achieves higher SRT performance than RLOO, but both algorithms lead to model collapse at similar training durations.The comparison uses Qwen2.5-Math-7B trained on DAPO.
  • KL penalty: Higher KL penalty coefficients do not delay or prevent model collapse under SRT.The authors attribute this to the self-reward-hacking signal overpowering KL regularization.
  • Learning rate: Lower learning rates appear to prevent collapse within the training budget, although performance degrades on AIME 2024 and 2025.The authors hypothesize that longer training would still lead to collapse but could not study this fully because of computational constraints.
  • Rollout count: Reducing the number of rollouts per prompt causes model collapse progressively later in the training run.All other hyperparameters are held fixed in this DAPO experiment.
  • Entropy coefficient: Increasing the entropy coefficient accelerates collapse by encouraging random rollout tokens followed by a repeated template answer.The entropy term uses per-token entropy averaged across all rollout tokens.
  • Test-time training: SRT test-time training produces limited but noticeable gains in majority@32 accuracy over direct majority voting from the base model.The test set is treated as an unlabeled training dataset, and SRT is applied directly.

H.2 WHY DOESN’T THE PERFORMANCE COLLAPSE DURING TEST-TIME-TRAINING?

Test-time self-training avoids the collapse seen in ordinary training, while curriculum learning and early stopping provide practical ways to extend or control self-training. Fixed offline labels improve stability but sacrifice evolving-label benefits.

  • Test-time dynamics: Test-time SRT on AIME 2024 shows no performance collapse, but both performance and pseudo-reward quickly saturate.The test-time dataset contains only a small fixed set of examples.
  • Test-time dynamics: Despite predictions degenerating to one response per test prompt after test-time training, test-time accuracy remains high.This behavior matches the response pattern optimized by the SRT objective.
  • Collapse mechanism: Large training datasets continually supply fresh samples, encouraging consistency optimization that can produce a uniform, prompt-independent answer.This contrasts with the limited-sample test-time setting, where SRT quickly reaches convergence and loses meaningful gradient signals.
  • Early stopping: A small labeled validation set can identify the peak performance point because held-out datasets peak at nearly the same training step.Early stopping therefore mitigates model collapse.
  • Offline labels: Offline labels from a fixed base checkpoint maintain training stability and achieve comparable performance to online SRT.The fixed-teacher approach limits the benefit of improved majority voting from the evolving policy.
  • Curriculum learning: Training on the easiest third of DAPO prompts delays reward hacking and shows no collapse after 3 epochs.The subsets are selected by base-model pass rate or majority-vote frequency, and reach performance comparable to ground-truth RL on all DAPO data.

I.4 TRAINING DYNAMICS OF SRT (QWEN2.5-MATH-7B) ON THE EASY DAPO SUBSET

Training SRT on easy DAPO subsets avoids the performance collapse seen on the full dataset within the same compute budget across several model and subset-generation settings. Dynamic oversampling and filtering similarly delays collapse by controlling the task distribution during training.

  • I.4 TRAINING DYNAMICS OF SRT (QWEN2.5-MATH-7B) ON THE EASY DAPO SUBSET: Easy-subset SRT avoids performance collapse within the same compute budget for Qwen2.5-Math-7B.Held-out accuracy does not drop, KL penalty remains below full-dataset SRT, and model entropy does not explode.
  • I.4 TRAINING DYNAMICS OF SRT (QWEN2.5-MATH-7B) ON THE EASY DAPO SUBSET: The easiest 1/3 of DAPO eliminates collapse within the training budget, although collapse may occur with longer training.
  • I.4 TRAINING DYNAMICS OF SRT (QWEN2.5-MATH-7B) ON THE EASY DAPO SUBSET: Qwen3-14B-Base likewise shows no collapse when trained with SRT on the easy DAPO subset within the same compute budget.
  • I.4 TRAINING DYNAMICS OF SRT (QWEN2.5-MATH-7B) ON THE EASY DAPO SUBSET: Generating the easy subset with Qwen2.5-Math-1.5B tests whether the curriculum process is reproducible across subset-generation models.The earlier easy subsets were generated with Qwen2.5-Math-7B, and the new subsets use majority-vote frequency and pass rate.
  • I.4 TRAINING DYNAMICS OF SRT (QWEN2.5-MATH-7B) ON THE EASY DAPO SUBSET: Dynamic oversampling filters prompts below 60% self-consistency and prevents collapse within the training budget.The procedure repeatedly samples and filters prompts until a batch is filled, then takes an RL step; the authors hypothesize longer training can still collapse.

J DETAILED EXPERIMENT RESULTS USING NON-QWEN MODELS

Experiments beyond Qwen models show that SRT can initially match ground-truth RL gains, but prolonged training can cause collapse. On Llama-3.1-8B-Instruct, SRT and ground-truth RL improve MATH-500 performance similarly under one tested setting, while a higher learning rate produces collapse.

  • J.1 DEEPSEEK-MATH-7B-INSTRUCT: Deepseek-Math-7B-Instruct initially matches ground-truth RL gains with SRT but collapses after prolonged training.The model is trained on MATH-12K and evaluated on AIME 24, AIME 2025, AMC, and MATH-500.
  • J.2 LLAMA-3.1-8B-INSTRUCT: At learning rate 3 × 10^-7, Llama-3.1-8B-Instruct demonstrates model collapse within the same training budget.
  • J.2 LLAMA-3.1-8B-INSTRUCT: Llama-3.1-8B-Instruct showed no gains on DAPO or MATH-12K, motivating evaluation on Big-Math-RL-Verified.The authors attribute this possibly to insufficient hyperparameter tuning or unsuitable starting performance.
  • J.2 LLAMA-3.1-8B-INSTRUCT: 52.6% to around 60%: Llama-3.1-8B-Instruct pass@1 accuracy on MATH-500 improves under SRT and ground-truth RL through 2,000 steps.Both objectives improve at approximately the same rate in this training budget.

K EXAMPLE TASKS FROM REASONING GYM

The Reasoning Gym examples illustrate family relationships, bitwise arithmetic, and knights-and-knaves tasks at specified difficulty levels. Difficulty is abstracted as a level, and prolonged SRT training produces degenerate, prompt-independent outputs rather than task solutions.

  • Task examples: The examples cover Family Relationships, Bitwise Arithmetic, and Knights and Knaves tasks at levels 4, 2, and 2, respectively.
  • Task examples: For the knights-and-knaves example, the stated solution is that Zoey is a fool and Riley is a sage.
  • Task difficulty: Difficulty in these tasks can be changed by modifying the number of people or digits and is represented abstractly as a level.
  • Post-training behavior: After 1,200 SRT steps on DAPO, Qwen2.5-Math-7B outputs \boxed{1} with incoherent tokens irrespective of the prompt.The examples identify this prompt-independent output as reward hacking because constant answers maximize consistency regardless of accuracy.
Loading 2505.21444v2…