PaperlayerView exact revision on arXiv

Source-linked AI summary

Reinforcement Learning for Reasoning in Large Language Models with One Training Example

Yiping Wang, Qing Yang, Zhiyuan Zeng, Liliang Ren, Liyuan Liu, Baolin Peng, Hao Cheng, Xuehai He, Kuan Wang, Jianfeng Gao, Weizhu Chen, Shuohang Wang, Simon Shaolei Du, Yelong Shen

arXiv:2504.20571v3cs.LGcs.AIcs.CL

TL;DR

The paper investigates whether RLVR can achieve strong reasoning improvements with far less training data than usual. It applies verifiable-reward reinforcement learning to one or a few selected examples and finds that one example can match thousands-example training while policy-gradient-driven exploration supports the gains.

  • Problem

    The paper asks how much RLVR training data is necessary to maintain performance comparable to using the full dataset.

  • Method

    The authors apply verifiable-reward reinforcement learning to one or a few selected mathematical examples across multiple models, algorithms, tasks, and ablations.

  • Results

    One example raises Qwen2.5-Math-1.5B from 36.0% to 73.6% on MATH500 and from 17.6% to 35.7% averaged across six mathematical benchmarks.

  • Takeaways & Limitations

    Tiny RLVR datasets can trigger broad reasoning improvements, including cross-category generalization, self-reflection, and post-saturation test gains.

  • Takeaways & Limitations

    Historical-variance selection is not necessarily optimal for choosing individual one-shot training examples.

Abstract

from arXiv · show

We show that reinforcement learning with verifiable reward using one training example (1-shot RLVR) is effective in incentivizing the math reasoning capabilities of large language models (LLMs). Applying RLVR to the base model Qwen2.5-Math-1.5B, we identify a single example that elevates model performance on MATH500 from 36.0% to 73.6% (8.6% improvement beyond format correction), and improves the average performance across six common mathematical reasoning benchmarks from 17.6% to 35.7% (7.0% non-format gain). This result matches the performance obtained using the 1.2k DeepScaleR subset (MATH500: 73.6%, average: 35.9%), which contains the aforementioned example. Furthermore, RLVR with only two examples even slightly exceeds these results (MATH500: 74.8%, average: 36.6%). Similar substantial improvements are observed across various models (Qwen2.5-Math-7B, Llama3.2-3B-Instruct, DeepSeek-R1-Distill-Qwen-1.5B), RL algorithms (GRPO and PPO), and different math examples. In addition, we identify some interesting phenomena during 1-shot RLVR, including cross-category generalization, increased frequency of self-reflection, and sustained test performance improvement even after the training accuracy has saturated, a phenomenon we term post-saturation generalization. Moreover, we verify that the effectiveness of 1-shot RLVR primarily arises from the policy gradient loss, distinguishing it from the "grokking" phenomenon. We also show the critical role of promoting exploration (e.g., by incorporating entropy loss with an appropriate coefficient) in 1-shot RLVR training. We also further discuss related observations about format correction, label robustness and prompt modification. These findings can inspire future work on RLVR efficiency and encourage a re-examination of recent progress and the underlying mechanisms in RLVR. All resources are open source at https://github.com/ypwang61/One-Shot-RLVR.

1 Introduction

The paper asks how far RLVR training data can be reduced and shows that one carefully selected example can substantially improve mathematical reasoning, matching much larger datasets. It also reports generalization and training dynamics that clarify how 1-shot RLVR works.

  • Main findings: 35.7% average performance from one example approaches the 35.9% obtained with 1.2k DeepScaleR examples, while two examples reach 36.6%.The two-example result is also comparable to using the 7.5k MATH training dataset at 36.7%.
  • Motivation: One training example can substantially enhance a base model’s mathematical performance, motivating the 1-shot RLVR setting.The authors frame this as reducing the RLVR training dataset to as little as one example.
  • Main findings: 36.0% to 73.6% on MATH500 and 17.6% to 35.7% averaged across six mathematical reasoning benchmarks, with gains beyond format correction.The selected example achieves performance comparable to the 1.2k DeepScaleR subset containing it.
  • Generalization: 1-shot RLVR transfers beyond mathematics, improves performance across reasoning categories, and increases self-reflective behavior during training.The paper reports improvements on non-mathematical reasoning tasks and cross-category generalization from a single example.
  • Training dynamics: Training accuracy can saturate while test performance continues improving, a phenomenon the authors call post-saturation generalization.The paper also observes that reasoning outputs can remain human-interpretable on test examples after training outputs degrade.
  • Mechanism: Policy gradient loss primarily drives 1-shot RLVR improvements, distinguishing the effect from grokking, while entropy loss supports exploration.The paper notes that an excessively large entropy coefficient can destabilize training.

2 Preliminary

The paper defines the RLVR components and a historical-variance approach for selecting training examples, then evaluates whether one-shot math training transfers to non-mathematical reasoning tasks.

  • Transfer evaluation: 1-shot RLVR with math examples π1/π13 improves ARC-Easy and ARC-Challenge performance, even compared with full-set RLVR.The reported checkpoints are selected using the best average across six mathematical benchmarks.
  • RL loss function: RLVR uses a binary outcome reward that assigns 1 only when the model’s mathematical answer matches the ground truth.The format reward is excluded from outcome-reward RLVR and used separately as a baseline for Qwen models.
  • RL loss function: Policy gradient loss reinforces better-than-average responses through group-normalized advantages and penalizes inferior responses.This is the principal optimization signal described for the adopted GRPO setup.
  • RL loss function: KL loss measures divergence from a reference model to help maintain general language quality.
  • RL loss function: Entropy loss uses a negative coefficient to encourage higher per-token entropy, exploration, and more diverse reasoning paths.It is included by default in the verl implementation used in the experiments but is not strictly necessary for GRPO.
  • Data selection: Historical variance ranks examples by the variance of their average training accuracy across epochs.The resulting ordering is used to select candidate examples for one-shot RLVR.

3 Experiments

Experiments evaluate 1(few)-shot RLVR across models, benchmarks, examples, and training dynamics. They find strong performance gains, cross-category generalization, post-saturation improvement, and increased self-reflection, while revealing important selection and scope caveats.

  • Experimental setup: 1-shot RLVR duplicates selected examples to reach the 128-sample training batch and evaluates six mathematical reasoning benchmarks plus ARC tasks.The default experiments use Qwen2.5-Math-1.5B and also test other models and algorithms.
  • 3.2.2 Post-saturation Generalization: Generalization After Training Accuracy Saturation: 3.4% average improvement follows π1 training-accuracy saturation, and π13 yields 9.9% improvement after saturation, unlike full-set RLVR.With π1 and π13, test performance keeps improving after training accuracy saturates; DSR-sub test performance instead stops improving after step 1000.
  • 3.2.2 Post-saturation Generalization: Generalization After Training Accuracy Saturation: After overfitting π1, training outputs become unintelligible multilingual mixtures, but test responses remain normal and accurate.The model can also use a different strategy, such as the Rational Root Theorem, on a test problem.
  • 3.3 1/Few-shot RLVR on Other Models/Algorithms: 1(few)-shot RLVR remains viable across Qwen2.5-Math-7B, Llama-3.2-3B-Instruct, DeepSeek-R1-Distill-Qwen-1.5B, and PPO, though gains and stability vary.For Qwen2.5-Math-7B, 1-shot RLVR improves average performance by 17.8%, while other models show different gaps from full-set RLVR.

4 Analysis

The analysis finds that 1-shot RLVR’s gains primarily come from policy gradient loss, while entropy-based exploration strengthens post-saturation generalization. Label accuracy and loss-component ablations further characterize when these improvements occur.

  • Comparison with Grokking: 1-shot RLVR’s post-saturation generalization resembles grokking, but the ablation tests whether weight decay or policy optimization explains the gain.The comparison is motivated by grokking’s sensitivity to regularization, especially weight decay.
  • Policy Gradient Loss: Policy gradient loss produces results close to full-loss training, whereas weight decay and KL divergence add little performance.Using only policy gradient loss nearly matches full GRPO; policy gradient plus entropy behaves almost like the full loss.
  • Entropy Loss: Entropy loss adds 4.0% on MATH500 and 2.5% on AIME24, but an excessive coefficient can destabilize training.The default entropy coefficient is beneficial, while the larger -0.003 coefficient makes training more unstable.
  • Post-Saturation Generalization: Entropy loss yields an average 2.3% improvement beyond training-accuracy saturation, and temperature t = 1.0 adds another 0.8%.Without entropy, performance improves little beyond step 150; increasing rollout temperature further improves results.
  • Label Robustness: Slightly inaccurate labels do not necessarily impair 1-shot RLVR, with performance rankings (1) ≈ (4) > (3) > (2).The tested labels were the correct label, an overfittable incorrect label, a non-overfittable incorrect label, and the original label.

5 Conclusion

The conclusion argues that 1-shot RLVR can substantially improve reasoning performance while matching RLVR trained on thousands of examples. It attributes this effectiveness to exploiting existing base-model reasoning through exploration on very small data.

  • Main Findings: 1-shot RLVR substantially improves reasoning tasks and can match RLVR trained with thousands of examples.The conclusion also reports post-saturation generalization, cross-category generalization, and more frequent self-reflection.
  • Interpretation: The findings suggest that base models already contain reasoning capability that small-data exploration can activate through useful RL training signals.The paper frames this as an interpretation of the observed 1-shot RLVR results.
  • Training Behavior: The authors report that zero-mean advantage gives RLVR an anti-overfitting property, allowing a single example to be trained millions of times without performance degradation.This claim is presented as an observed property of the RLVR algorithm in the conclusion.
  • Implications: The work emphasizes better data selection and collection for RLVR and identifies future directions alongside limitations discussed in the appendices.The conclusion points readers to Appendix D.4 for future work and Appendix D.1 for limitations.

6 Acknoledgements

This supplied section combines acknowledgements with background, training-loss details, dataset descriptions, and evaluation procedures. It documents the study’s experimental and methodological context alongside contributor and funding acknowledgements.

  • Acknowledgements: The authors acknowledge constructive discussions with numerous researchers and report fellowship or grant support from Amazon AI, NSF, and the Sloan Research Fellowship.The acknowledgements name individual contributors and list the supporting fellowships and grants.
  • RLVR Background: RLVR uses rule-based verification, commonly assigning a binary reward when a mathematical answer matches the gold reference.This describes the standard answer-matching reward design used in mathematical reasoning RLVR.
  • Loss Function: The GRPO pipeline samples groups of outputs from an old policy and optimizes the policy with policy-gradient, KL-divergence, and entropy components.The loss coefficients include β for KL divergence and α for entropy, while group-normalized advantages weight policy updates.
  • Loss Function: For mathematical questions, the reward is 1 only when an output is correct, while entropy loss encourages more diverse responses through α < 0.Entropy is computed as average per-token entropy over response tokens selected by a response mask.
  • Training Data: The study uses a processed DeepScaleR dataset built from approximately 40,000 mathematics problem-answer pairs, with answer extraction, deduplication, and SymPy-based filtering.The source problems include AIME, AMC, Omni-MATH, and Still.
  • Evaluation Datasets: OlympiadBench is evaluated through an English, text-only subset of 675 open-ended mathematics competition problems from a larger bilingual multimodal benchmark.The full benchmark contains 8,476 Olympiad-level mathematics and physics problems with expert-annotated solutions.

B.5 More Evaluation Details

These appendices document evaluation settings, checkpoint and precision considerations, and detailed comparisons across models, examples, and RLVR configurations.

  • Evaluation settings: Evaluation uses a maximum of 3072 generated tokens, model-specific prompt templates, seed 0, and top_p 1 by default.DeepSeek-R1-Distill-Qwen-1.5B uses separate temperature, sampling, averaging, and evaluation-length settings.
  • Checkpoint precision: Changing saved precision metadata from float32 to bfloat16 resolves the numerical inconsistency between downloaded models and RL pipeline checkpoints.The discrepancy arises because the pipeline saves float32 checkpoints while Hugging Face base models use bfloat16.
  • Detailed results: Detailed tables report best-average and best-per-benchmark results for 1-, 2-, and 4-shot RLVR across Qwen, Llama, and additional model configurations.The tables distinguish the checkpoint maximizing overall average from independently best checkpoints on each benchmark.
  • Additional model evaluations: For Qwen2.5-1.5B, 16-shot RLVR approaches full-set RLVR, whereas for Qwen2.5-Math-1.5B-Instruct, 1-shot RLVR is more stable than full-set RLVR.The base model still improves substantially under 1-shot RLVR despite a larger gap from full-set training.

C.1.2 Detailed Performance on More Models and Training Examples.

Additional evaluations show that 1(few)-shot RLVR remains effective across model variants, while performance depends strongly on the selected training example.

  • Base and instruction models: 3.2% to 43.6%: 1-shot RLVR with π1 substantially improves Qwen2.5-1.5B on MATH500, while 16-shot RLVR closely approaches full-set RLVR.For Qwen2.5-Math-1.5B-Instruct, both full-set and 1-shot RLVR provide limited improvement because the initial model already performs well.
  • Training-example selection: π605 and π1209 perform worse than π1 on Qwen2.5-Math-7B, while 16-shot RLVR more consistently closes the gap with full-set RLVR.The results indicate that single-example effectiveness is sensitive to example selection.

C.1.3 Detailed performance with best per-benchmark results

Detailed analyses examine best per-benchmark checkpoints, training dynamics, loss ablations, format correction, in-context learning, and label robustness in 1-shot RLVR.

  • Best per-benchmark results: Best results often occur at later checkpoints after training accuracy saturates, consistent with post-saturation generalization.The tables report each benchmark’s best checkpoint independently, so their average can exceed the best overall average.
  • Training dynamics: Llama3.2-3B-Instruct begins degrading before 200 steps in nearly all setups, indicating greater training instability than the other evaluated configurations.The authors suggest testing more stable GRPO variants to address this phenomenon.
  • Loss ablations: Policy gradient loss is the main contributor to 1-shot RLVR, while entropy-only training improves most math benchmarks briefly but underperforms the format-reward baseline.Excessive entropy can make outputs random, and the apparent entropy-only benefit may partly reflect format fixing.
  • Format and reasoning effects: A high ratio of boxed answers can coexist with continued test-accuracy gains on several benchmarks, suggesting reasoning improvement beyond formatting.The rule-based and LLM-judged accuracies are close, and higher boxed-answer ratios correlate with higher test accuracy.
  • Format correction and prompting: 1-shot RLVR with outcome reward outperforms in-context learning, while retaining gains beyond those readily obtained from format reward or prompting.On Qwen2.5-Math-7B, in-context learning with π1 raises MATH500 from 51.0% to 75.4% and average performance from 22.4% to 37.4%.
  • Label robustness: 60% randomly corrupted labels still yield good RLVR results, but 90% corruption performs worse than 1-shot RLVR with one correct label.The results support partial robustness to label noise, with excessive wrong labels harming improvements from correct data.
  • Example-dependent dynamics: The selected example affects post-saturation exploration because difficult intermediate steps can generate more varied chains of thought.The discussion uses π1’s cube-root and algebraic steps as an example of this mechanism.

C.4 Pass@8 Results

Pass@8 evaluation compares 1-shot and full-set RLVR on three mathematical tasks, with format-reward RLVR included as a stronger baseline.

  • Pass@8 comparison: 51.7%: 1-shot RLVR achieves comparable or slightly better pass@8 performance, while full-set RLVR declines noticeably after 200 steps.The observed decline is consistent with reports that RLVR can degrade pass@n performance.

D.1 Limitations of Our Work

The work is limited by its computational and domain scope, and by uncertainty about how broadly its selected one-example setup transfers. It also notes that few-shot training can be more stable and that 1-shot RLVR may not reduce computation.

  • The experiments do not include larger models such as Qwen2.5-32B because of computational-resource limits.
  • The study focuses mainly on mathematical reasoning and does not evaluate 1(few)-shot RLVR on other verifiable domains such as coding.
  • 1-shot RLVR does not save, and may require more, reinforcement-learning computation.
  • The example π1 is not necessarily optimal for other models because it was selected using Qwen2.5-Math-1.5B’s historical variance score.
  • RLVR with 16 examples can be as effective as the 1.2k-example subset and outperform 1-shot RLVR on several tested models.

D.4 Future Works

The paper identifies data selection, theoretical understanding, exploration, and broader applications as priorities for future work. It emphasizes that example value varies and that the mechanisms behind post-saturation generalization remain unresolved.

  • Data Selection and Curation: 1-shot RLVR could support data selection by evaluating examples individually and identifying which data most stimulates reasoning capability.
  • Data Selection and Curation: Scaling RLVR datasets is not rendered useless, but better selection and collection of training data are emphasized.
  • Understanding 1-shot RLVR and Post-saturation Generalization: A rigorous explanation of 1-shot RLVR feasibility and post-saturation generalization remains open.
  • Importance of Exploration: Further work should clarify how entropy loss and exploration enhance performance in 1-shot RLVR.
  • Other Applications: The effectiveness of 1-shot RLVR should be tested in code generation and other domains, including settings without verifiable rewards.

E Example Details

The supplementary material provides detailed information for the mathematical examples used in the experiments, including prompts and ground-truth labels. It also gives additional details for examples such as π13 and π2–π17.

  • The supplementary tables provide detailed information for each example used in the experiments and for examples π1 through π17.
  • Each supplementary table contains an individual example’s specific prompt and corresponding ground-truth label.
  • The supplementary material also includes details for examples π605, π606, π1201, and π1207–π1209.
Loading 2504.20571v3…