Source-linked AI summary
JustRL: Scaling a 1.5B LLM with a Simple RL Recipe
Bingxiang He, Zekai Qu, Zeyuan Liu, Yinghao Chen, Yuxin Zuo, Cheng Qian, Kaiyan Zhang, Weize Chen, Chaojun Xiao, Ganqu Cui, Ning Ding, Zhiyuan Liu
TL;DR
The paper asks whether complex RL pipelines are necessary for small reasoning models. It introduces JustRL, a single-stage recipe with fixed hyperparameters, and reports competitive performance, stable training, and lower compute while identifying cases where added techniques hurt performance.
Problem
Increasingly complex RL pipelines make it unclear whether their many techniques are necessary for stable and strong performance in small language models.
Method
JustRL uses single-stage RL with fixed hyperparameters and a minimal set of standard components across two 1.5B reasoning models.
Results
JustRL matches or exceeds sophisticated approaches using 2× less compute, with the same hyperparameters transferring across both models and smooth improvement over 4,000+ steps.
Takeaways & Limitations
The findings support establishing simple, scaled-up baselines before adding complexity to address specific observed problems.
Takeaways & Limitations
The results are limited to mathematical reasoning at the 1.5B scale, and the study does not isolate which specific components are responsible for its success.
Abstract
from arXiv · showhide
Recent advances in reinforcement learning for large language models have converged on increasing complexity: multi-stage training pipelines, dynamic hyperparameter schedules, and curriculum learning strategies. This raises a fundamental question: \textbf{Is this complexity necessary?} We present \textbf{JustRL}, a minimal approach using single-stage training with fixed hyperparameters that achieves state-of-the-art performance on two 1.5B reasoning models (54.9\% and 64.3\% average accuracy across nine mathematical benchmarks) while using 2$\times$ less compute than sophisticated approaches. The same hyperparameters transfer across both models without tuning, and training exhibits smooth, monotonic improvement over 4,000+ steps without the collapses or plateaus that typically motivate interventions. Critically, ablations reveal that adding ``standard tricks'' like explicit length penalties and robust verifiers may degrade performance by collapsing exploration. These results suggest that the field may be adding complexity to solve problems that disappear with a stable, scaled-up baseline. We release our models and code to establish a simple, validated baseline for the community.
1. Introduction
The paper asks whether increasingly complex RL techniques are necessary for small reasoning models. JustRL shows that fixed-hyperparameter, single-stage training can achieve competitive performance with stable improvement and lower compute.
- RL can improve smaller models after distillation reaches a performance ceiling imposed by the teacher’s capabilities.
- The field’s growing collection of stabilization techniques makes it difficult to identify which methods genuinely drive performance.Different combinations of interventions can obscure whether they solve fundamental RL problems or compensate for complexity introduced elsewhere.
- JustRL matches or exceeds more complex approaches while using 2× less compute on two 1.5B reasoning models.It avoids multi-stage pipelines and dynamic schedules while maintaining competitive results.
- The paper argues that simple baselines should be established before additional complexity is introduced.Its goal is not to reject all techniques, but to test what is fundamentally sufficient.
2. Related Work
Recent small-model RL work increasingly combines multiple stabilization, scheduling, sampling, and curriculum techniques. This accumulated complexity motivates testing whether a simpler approach can remain competitive.
- JustRL is positioned as a simpler alternative to the prevailing trend toward combining many techniques.
- Recent methods span three distilled model families: DeepSeek-R1-Distill-Qwen-1.5B, OpenMath-Nemotron-1.5B, and Qwen3-1.7B.
- The field has progressed from hyperparameter tuning and reference resets to multi-stage training, progressive context lengthening, and curriculum learning.DeepScaleR uses three stages, FastCuRL uses five, and QuestA augments questions with partial chain-of-thought hints.
- Table 1 catalogs a growing toolkit including multi-stage training, adaptive hyperparameters, length penalties, dynamic sampling, and stabilization mechanisms.The breadth of techniques makes it difficult to isolate which components matter.
3. JustRL: Simplicity at Scale
JustRL uses a deliberately minimal RL recipe built around standard GRPO, binary verifier rewards, fixed settings, and continuous single-stage training. It applies this setup to two 1.5B models and evaluates them across nine mathematical benchmarks.
- 3. JustRL: Simplicity at Scale: The core algorithm is default GRPO with binary outcome rewards from a lightweight rule-based verifier.The verifier avoids symbolic math libraries such as SymPy to reduce computational overhead.
- 3.1. Training Setup: JustRL trains continuously in a single stage with fixed hyperparameters, standard data, basic prompting, and a 16K-token context limit.It omits curriculum switching, adaptive schedules, dynamic sampling, and explicit length penalties.
- 3. JustRL: Simplicity at Scale: The recipe retains only “clip higher” as an established stability practice for long-horizon RL training.
- 3. JustRL: Simplicity at Scale: The same fixed recipe is trained on two 1.5B reasoning models without per-model hyperparameter tuning.Each model is trained with 32 A800-80GB GPUs for approximately 15 days.
- 3.2. Evaluation Protocol: The evaluation covers nine mathematical reasoning benchmarks and reports Pass@1 accuracy using sampled responses per problem.Sampling uses N=4 for MATH-500, Minerva Math, and OlympiadBench, and N=32 for the other benchmarks.
4. Experimental Results
JustRL achieves competitive or better mathematical-reasoning performance on two 1.5B models with a fixed, single-stage recipe, using substantially less compute and requiring no hyperparameter retuning. Training remains smooth and stable, while ablations show that added penalties and verifier changes can reduce performance by collapsing exploration.
- 4.3. Training Dynamics Analysis: Training improves smoothly and monotonically for over 4,000 steps, without oscillations or plateaus that typically prompt intervention.The analysis tracks reward, entropy, and response length as stability indicators.
- 4.1. Scaling a Weaker Base: JustRL-DeepSeek-1.5B: 54.87% average across nine benchmarks outperforms ProRL-V2’s 53.08%, with JustRL-DeepSeek leading on six benchmarks while using 2× less compute.The comparison uses a single-stage recipe with fixed hyperparameters, whereas ProRL-V2 uses a nine-stage pipeline with dynamic hyperparameters.
- 4.2. Scaling a Stronger Base: JustRL-Nemotron-1.5B: The same fixed recipe works across two 1.5B backbones without hyperparameter tuning, suggesting robustness rather than optimization for one model.Both models also exhibit smooth training curves under the transferred recipe.
- 4.4. Ablation Studies: Adding an overlong penalty lowers AIME 2024 performance from 55% to 50%, while combining it with a robust verifier lowers performance to 45%.The explicit penalty also drives entropy down to 0.5-0.6 versus 1.2-1.4 for the base approach, indicating collapsed exploration.
- 4.4. Ablation Studies: The ablations show that standard interventions do not necessarily transfer: both tested modifications degraded performance and may disrupt the base recipe’s balance.The authors state that only two modifications were tested and that other techniques remain unexplored.
5. Discussion
JustRL’s stable, minimal recipe contrasts with the instabilities that motivate complex RL techniques, while negative ablations suggest added complexity can hurt. The authors frame simplicity as a baseline to test before adding methods for specific failures or settings.
- Smooth training with healthy entropy, monotonic rewards, and natural length convergence contrasts with instabilities often motivating complex techniques.The authors suggest some complexity may address symptoms introduced by other design choices rather than fundamental RL challenges.
- The authors cannot isolate whether hyperparameters, training data, verifier design, or their interaction explains why simple RL works well.
- Additional techniques may help under extreme compute constraints, specific failure modes, higher performance ceilings, or noisier reward signals.
6. Conclusion
The paper argues that adequate scale and stable fundamentals can make simple RL competitive with sophisticated approaches for small reasoning models. It advocates starting with a robust baseline and adding complexity only when that baseline demonstrably fails.
- Single-stage training with fixed hyperparameters matched or exceeded multi-stage and curriculum-based approaches while using 2× less compute.
- Smooth, stable improvement over thousands of steps occurred without interventions typically used to prevent training collapse.
- The authors recommend starting simple, scaling up, and adding complexity only when a simple robust baseline demonstrably fails.
Limitations
The study’s evidence is limited to mathematical reasoning at the 1.5B-parameter scale, and it does not isolate which design components drive success.
- Generalization beyond 1.5B mathematical reasoning, including coding and general question answering, remains unexplored.
- The study cannot definitively identify whether hyperparameters, verifier design, or training data are most critical to its success.
- The reported conclusions are therefore bounded by the evaluated task domain and model scale.