Source-linked AI summary
A Minimalist Approach to LLM Reasoning: from Rejection Sampling to Reinforce
Wei Xiong, Jiarui Yao, Yuhui Xu, Bo Pang, Lei Wang, Doyen Sahoo, Junnan Li, Nan Jiang, Tong Zhang, Caiming Xiong, Hanze Dong
TL;DR
The paper asks why GRPO is effective for reward-based LLM reasoning when its algorithmic advantage remains unclear. It revisits rejection sampling and Reinforce variants, finding that selective prompt filtering—not reward normalization—accounts for much of GRPO’s benefit, while RAFT and Reinforce-Rej provide simpler alternatives.
Problem
GRPO is widely used for mathematical-reasoning post-training, but its algorithmic advantage over simpler methods remains insufficiently justified.
Method
The study analyzes RAFT, Reinforce, and GRPO through controlled ablations, then proposes Reinforce-Rej, which filters prompts with entirely correct or entirely incorrect responses.
Results
RAFT matches or outperforms more sophisticated methods such as PPO and iterative DPO, while GRPO’s main benefit comes from discarding prompts with entirely incorrect responses rather than reward normalization.
Takeaways & Limitations
RAFT and Reinforce-Rej are lightweight, interpretable baselines, and negative samples should be incorporated selectively through more principled mechanisms.
Takeaways & Limitations
Binary correctness rewards can be too coarse, making unlearning from negative samples unstable when negative signals lack fine-grained information.
Abstract
from arXiv · showhide
Reinforcement learning (RL) has become a prevailing approach for fine-tuning large language models (LLMs) on complex reasoning tasks. Among recent methods, GRPO stands out for its empirical success in training models such as DeepSeek-R1, yet the sources of its effectiveness remain poorly understood. In this work, we revisit GRPO from a reinforce-like algorithm perspective and analyze its core components. Surprisingly, we find that a simple rejection sampling baseline, RAFT, which trains only on positively rewarded samples, yields competitive performance than GRPO and PPO. Our ablation studies reveal that GRPO's main advantage arises from discarding prompts with entirely incorrect responses, rather than from its reward normalization. Motivated by this insight, we propose Reinforce-Rej, a minimal extension of policy gradient that filters both entirely incorrect and entirely correct samples. Reinforce-Rej improves KL efficiency and stability, serving as a lightweight yet effective alternative to more complex RL algorithms. We advocate RAFT as a robust and interpretable baseline, and suggest that future advances should focus on more principled designs for incorporating negative samples, rather than relying on them indiscriminately. Our findings provide guidance for future work in reward-based LLM post-training.
1 Introduction
The paper examines why GRPO works for reward-based LLM post-training and finds that sample selection, especially rejecting wholly incorrect prompts, matters more than reward normalization. It presents RAFT as a strong positive-only baseline and Reinforce-Rej as a selective policy-gradient alternative.
- PPO adds a critic network beyond Reinforce, increasing computational overhead and algorithmic complexity.
- GRPO mixes accepted and rejected responses, whereas RAFT trains only on positively rewarded samples.The authors report that RAFT-like methods can converge faster early in training.
- RAFT achieves competitive performance with GRPO and PPO despite using only positive samples.The paper identifies this result as surprisingly small performance gaps between simple rejection sampling and more complex RL methods.
- All-incorrect prompts can significantly harm on-policy training, while within-prompt reward normalization has minimal impact.Controlled experiments isolate negative-sample filtering and reward normalization as separate design choices.
- Reinforce-Rej filters prompts with either all correct or all incorrect responses and achieves performance comparable to GRPO with superior KL efficiency.
- The study emphasizes sample selection over algorithmic complexity and provides publicly available implementation code.
2 Related Works
Related work spans Reinforce-style RL, rejection-sampling fine-tuning, preference optimization, data filtering, and verifier-rewarded mathematical reasoning models.
- Preference methods such as Slic-HF and DPO optimize contrastive objectives using pairwise comparison data.
- Recent mathematical-reasoning models such as GPT-o1 and DeepSeek-R1 use RL with verifier-based rewards, contrasting with earlier supervised fine-tuning efforts.
3 Method
The method formalizes LLM post-training with binary verifier rewards and reviews RAFT, policy-gradient Reinforce, GRPO, DPO, and RAFT++ as related training procedures.
- Notation: An LLM is modeled as a policy π(a|x), with a binary verifier reward r(x, a) ∈ {−1, 1} and prompt-response dataset D.
- RAFT: RAFT samples n responses per prompt, retains responses with the highest reward, and fine-tunes the current policy on the selected positive dataset.
- RAFT: STaR differs from RAFT by retraining from the original model, using greedy single-response decoding, and supplying answers for difficult problems.
- Policy Gradient and Reinforce: Reinforce updates the policy using reward-weighted policy gradients, with importance sampling correcting distribution shift during multiple minibatch updates.
- Policy Gradient and Reinforce: Clipping is added to stabilize updates when the current and data-collection policies diverge, and the objective can be extended from responses to tokens.
- GRPO: GRPO samples multiple responses per prompt and replaces scalar rewards with token-level advantages based on within-prompt reward normalization.
- DPO: DPO trains on response pairs with a preference ordering using a contrastive loss, and later variants can generate fresh on-policy data iteratively.
- RAFT++: RAFT++ applies importance sampling and clipping to RAFT, while its indicator restricts training to highest-reward positive responses.
4 Experiment Setup
Experiments use mathematical-reasoning prompts, two model families, verifier-based evaluation across three benchmarks, and standardized training settings.
- The implementations focus on mathematical reasoning and are mainly based on the verl framework.
- Dataset and Models: Numina-Math provides approximately 860k labeled math problems spanning school exercises and mathematics olympiad problems.
- Dataset and Models: Experiments use Qwen2.5-Math-7B-base and LLaMA-3.2-3B-instruct to assess generality across model families.
- Hyper-parameters: Training samples 1024 prompts per iteration, generates n = 4 responses per prompt for RAFT and GRPO, and uses a 1×10^-6 AdamW learning rate.
- Evaluation: Evaluation covers Math500, Minerva Math, and Olympiad Bench using average@16 with temperature 1.0 and a 4096-token maximum.
- The project code is publicly available at the Minimal-RL GitHub repository.
5 Main Results
RAFT and RAFT++ achieve performance close to deep RL methods, with RAFT++ benefiting from distribution correction and clipping but eventually being surpassed by GRPO. Ablations indicate that clipping stabilizes importance sampling, while later training dynamics motivate investigating negative samples.
- 52.3% average accuracy makes vanilla RAFT competitive with iterative DPO and close to PPO, while RAFT++ reaches 56.1% versus GRPO’s 56.3%.These results are reported for Qwen2.5-Math-7B-base.
- Importance sampling and clipping improve RAFT, with RAFT++ converging faster and reaching higher final accuracy than vanilla RAFT.The methods steadily improve reasoning ability through online updates.
- Importance sampling without clipping underperforms vanilla RAFT, suggesting that unbounded updates can violate the on-policy assumption and degrade stability.The authors hypothesize this occurs when the policy ratio deviates far from 1.
- RAFT++ learns faster early but slows around iteration 100 and is eventually surpassed by GRPO in later training and final test accuracy.The authors identify the slowdown and the role of missing negative samples as targets for further ablation.
- The ablation study examines why RAFT++ is initially faster but later underperforms Reinforce and GRPO, focusing on the role of negative samples.
2. From Vanilla Reinforce to GRPO: What is the key factor behind GRPO’s superior performance?
The analysis attributes GRPO’s advantage primarily to selectively discarding harmful prompts, especially those with entirely incorrect responses, rather than to reward normalization. Filtering both fully correct and fully incorrect prompts yields a stable Reinforce variant that preserves exploration more effectively than positive-only training.
- Learning dynamics: RAFT++ converges faster initially but rapidly loses policy entropy, reducing exploration and eventually allowing GRPO to surpass it.Its KL divergence from the initial policy also rises faster early in training.
- Ablation design: The ablations isolate negative-sample filtering and reward normalization as the two main differences between GRPO and vanilla Reinforce.The study evaluates removing fully correct prompts, removing fully incorrect prompts, removing both, and applying mean or standard-deviation normalization.
- Sample filtering: Removing entirely incorrect prompts provides the largest reward improvement, whereas removing only entirely correct prompts remains unsatisfactory.The results identify incorrect samples as particularly harmful during Reinforce training.
- Sample filtering: Removing both entirely correct and incorrect prompts produces better-behaved entropy loss and slightly higher reward, helping maintain exploration.This variant balances reward, KL stability, and entropy regularization.
- Normalization: Mean-zero normalization increases KL divergence and destabilizes training, while standard-deviation normalization adds little beyond removing bad samples.These findings indicate that normalization is not the main contributor to GRPO’s performance.
- Resulting method: Reinforce-Rej is the name given to the variant that removes both fully correct and fully incorrect samples.It is presented as a simplified, competitive baseline for reward-based policy optimization.
6 Conclusion
The paper revisits LLM post-training through rejection sampling, showing that simple positive-sample training is a strong baseline and that selective rejection captures GRPO’s main benefit. It proposes RAFT++ and Reinforce-Rej as lightweight alternatives with strong performance, stability, or interpretability.
- Conclusion: RAFT relies only on positively rewarded samples yet matches or outperforms more sophisticated approaches such as PPO and iterative DPO.RAFT++ adds importance sampling and clipping while retaining a simple and stable training pipeline.
- Conclusion: GRPO’s primary benefit comes from discarding prompts with entirely correct and incorrect responses, not from reward normalization.This conclusion is based on extensive ablation studies.
- Conclusion: Reinforce-Rej filters entirely incorrect and entirely correct samples, improving KL efficiency and entropy stability.The method is a minimal policy-gradient variant motivated by the ablation findings.
- Conclusion: The utility of negative samples is nuanced, motivating selective and principled mechanisms rather than indiscriminate use.The paper advocates RAFT and Reinforce-Rej as lightweight, interpretable, and effective baselines.