Source-linked AI summary
Light-R1: Curriculum SFT, DPO and RL for Long COT from Scratch and Beyond
Liang Wen, Yunke Cai, Fenrui Xiao, Xin He, Qi An, Zhenyu Duan, Yimin Du, Junchen Liu, Lifu Tang, Xiaowei Lv, Haosheng Zou, Yongchao Deng, Shousheng Jia, Xiangzheng Zhang
TL;DR
Large long-COT models are costly to train and deploy, motivating compact reasoning systems built from public data and models. Light-R1 uses difficulty-progressive curriculum training with multi-stage post-training and RL, producing strong mathematical reasoning and cross-domain performance across compact model sizes.
Problem
Large R1-level models require prohibitive computation, creating a need for compact long-COT models suitable for resource-constrained applications.
Method
Light-R1 combines public-data difficulty filtering, progressive curriculum SFT, DPO, and RL to train long-COT models from scratch and refine distilled models.
Results
Light-R1 establishes strong 7B, 14B, and 32B results, including about 2% absolute improvement from RL on Light-R1-14B-DS.
Takeaways & Limitations
The released models, datasets, and code support reproducible development of compact long-reasoning systems under resource constraints.
Abstract
from arXiv · showhide
This paper introduces Light-R1, an open-source suite for training long reasoning models using reproducible and cost-effective methodology. Given the proprietary nature of data used in the DeepSeek-R1 series, we develop an alternative approach leveraging exclusively public data and models. Our curriculum training progressively increases data difficulty, combined with multi-staged post-training. Our Light-R1-32B model, trained from Qwen2.5-32B-Instruct, outperforms DeepSeek-R1-Distill-Qwen-32B in math reasoning. Experimental results show that this curriculum approach becomes more effective when distinct, diverse datasets are available for different training stages: fine-tuning DeepSeek-R1-Distilled models (pre-tuned by DeepSeek team on proprietary data) with 3,000 challenging examples from our curriculum dataset yielded state-of-the-art 7B and 14B models, while the 32B model, Light-R1-32B-DS performed comparably to QwQ-32B and DeepSeek-R1. Furthermore, we extend our work by applying GRPO on long reasoning models. Our final Light-R1-14B-DS achieves SOTA performance among 14B models in math, with AIME24 & 25 scores of 74.0 and 60.2 respectively, surpassing many 32B models and DeepSeek-R1-Distill-Llama-70B. Despite math-focused training, Light-R1-14B-DS demonstrates strong cross-domain generalization. Light-R1 represents a significant advancement in making sophisticated reasoning models more accessible and implementable in real-world applications. Our models, training data and code have been made available at https://github.com/Qihoo360/Light-R1.
1 Introduction
Light-R1 addresses the resource barrier of training and deploying large long-COT models by developing compact reasoning models through open, reproducible curriculum post-training. The approach combines difficulty-progressive data, staged SFT and DPO, and RL for long-COT models.
- Large R1-level models impose prohibitive computational costs, limiting practical deployment on edge devices and in real-time applications.
- The work establishes reproducible evaluation protocols and reports results using averaged AIME24 pass@1 scores.
- Light-R1-32B uses two difficulty-progressive SFT stages followed by DPO and outperforms DeepSeek-R1-Distill-Qwen-32B in mathematical reasoning.
- RL on Light-R1-14B-DS increases response length and reward scores simultaneously without the initial length reduction typically observed.
- The curriculum approach progressively exposes models to increasingly difficult reasoning data and costs $1000 for 6 hours on 12×H800 GPUs.
- 3,000 challenging examples improve DeepSeek-R1-Distill models and produce the SOTA 7B model Light-R1-7B-DS.
2 The Origin of Everything: Stable and Trustworthy Evaluation of Long-COT Models
The paper emphasizes that long-COT evaluation requires many samples for stable pass@1 estimates, unlike earlier greedy-decoding evaluation. Its protocol uses 64 samples per query and reproduces reported baseline scores with approximately one-point deviation.
- Long-COT models generally perform better with sampling, but multiple samples increase evaluation burden compared with greedy decoding.
- 64 responses per query are needed to estimate pass@1 reliably; using 16 or fewer caused deviations exceeding 3 points across runs.
- The authors release evaluation code and logs adapted from prior work for stable and trustworthy evaluation.
- With 64 samples per query, the protocol reproduces DeepSeek-R1-Distill and QwQ scores with deviation around 1 point.
3 Light-R1-32B: Long-COT from Scratch with Curriculum SFT & DPO
Light-R1-32B is trained from public data through a staged pipeline that collects, cleans, difficulty-filters, and curriculum-orders mathematical problems before SFT and DPO. The resulting stages improve performance consistently, while math-focused training partially reduces GPQA performance but preserves strong generalization.
- Data preparation: The pipeline begins with approximately 1,000k math questions with ground-truth answers, then filters for diversity and removes contamination.
- Data generation: Difficulty filtering first retains approximately 76k questions using DeepScaleR-1.5B-Preview and then selects roughly 3k harder examples using DeepSeek-R1 responses.
- Post-training pipeline: The training curriculum comprises SFT Stage 1 on 76k problems, SFT Stage 2 on 3k challenging problems, and DPO with verified response pairs.
- Post-training pipeline: DPO uses verified correct DeepSeek-R1 answers as chosen responses and verified incorrect SFT-stage-2 responses as rejected responses.
- Results: Curriculum SFT and DPO produce consistent performance improvements, and TIES merging of trained variants yields additional gains.
- Generalization: Math-focused training decreases GPQA scores beginning at SFT Stage 2, but Light-R1-32B retains strong generalization compared with the base model.
- Results: Training only on the 3k Stage 2 dataset improves all DeepSeek-R1-Distill-Qwen models, indicating strong transferability of the refined data.
- Generalization: Light-R1-32B-DS shows unexpectedly high GPQA performance without science or code training, whereas Light-R1-7B-DS improves only on in-domain tasks.
4 Light-R1-14B-DS: Reinforcement Learning from Long-COT Models
This section presents a two-pass RL approach for improving already long-COT 14B models, combining offline difficulty filtering with online GRPO. Light-R1-14B-DS shows stable training dynamics, with response length and reward increasing together without initial length collapse.
- RL on DeepSeek-R1-Distill-Qwen-14B is presented as the first publicly documented demonstration of significant improvement on an already long-COT 14B model.
- The method uses offline data selection followed by online GRPO on prompts whose estimated pass rates fall between 0.25 and 0.625.The filtering removes prompts that are too easy or too hard and aligns training data with the rule-based verifier.
- Two-sided importance-sampling clipping and a modified length reward stabilize optimization and weaken the preference for shortening correct answers.Clipping limits extreme policy-ratio influence, while the length-reward modification helps prevent initial response-length collapse.
- RL training uses rule-based rewards on deduplicated Big-Math data, with offline selection taking 4 hours and online training taking 26 or 42 hours for 140 or 220 steps.
- Table 4 reports RL performance improvements for Light-R1-14B-DS and notes out-of-domain improvement on GPQA.
- Response length and reward increase simultaneously without an initial drop, supporting continuation to approximately 3 epochs or 220 steps.The first two evaluated epochs showed limited improvement, but the training curves were judged healthy.
5 Conclusion
The paper develops compact long-reasoning models through curriculum training, curated data, and reinforcement learning under resource constraints. Its released models, datasets, and code target broader access to reasoning systems and further research on generalization and RL efficiency.
- Light-R1 trains long-COT models from scratch through curriculum training and improves DeepSeek-R1-Distill models across 7B, 14B, and 32B sizes.
- The 3K dataset transfers across model sizes, while reinforcement learning on a multi-stage finetuned base model achieves superior performance with stable response-length growth.
- The open-source models, datasets, and code are intended to broaden access to R1-level reasoning and support research on curriculum design, data efficiency, and RL scalability.
A Light-R1 Series of Models
Table 5 distinguishes Light-R1 models by their starting model family: “-DS” models derive from DeepSeek-R1-Distill, while other models derive from Qwen-Instruct.
- “-DS” denotes models initialized from DeepSeek-R1-Distill; models without “-DS” are initialized from Qwen-Instruct.
B Dataset composition for full 59K questions
Table 6 summarizes the composition of the released data after first-stage diversity and difficulty filtering. OpenR1-Math-220k is the initial seed dataset and contributes the largest portion, while sources may overlap.
- The released-data composition is summarized after first-stage diversity and difficulty filtering, with potentially overlapping examples across sources.
- OpenR1-Math-220k serves as the initial seed dataset and therefore contributes the largest portion of the data.
C Data Decontamination
The section reports matched-prompt counts between open-source datasets and benchmarks.
- C Data Decontamination: Table 7 reports the number of matched prompts.These matches are evaluated between open-source datasets and benchmarks.
- C Data Decontamination: The table compares open-source datasets against benchmarks.
- C Data Decontamination: Matched-prompt counts provide the table’s reported measure for data decontamination.
D Training hyperparameters for Light-R1 series
Training sequence length is selected according to data characteristics, while GRPO uses a multi-factor balance and DPO uses sequence parallelism for 32k contexts.
- D Training hyperparameters for Light-R1 series: Sequence length is determined by the characteristics of the training data.
- D Training hyperparameters for Light-R1 series: GRPO sequence length balances rollout cost, inference cutoff ratio, and 32k-context evaluation performance.
- D Training hyperparameters for Light-R1 series: DPO with 32k context uses sequence parallelism to address GPU-memory limitations.The implementation comes from 360-LLaMA-Factory.