Source-linked AI summary
SRPO: A Cross-Domain Implementation of Large-Scale Reinforcement Learning on LLM
Xiaojiang Zhang, Jinghui Wang, Zifei Cheng, Wenhao Zhuang, Zheng Lin, Minglei Zhang, Shaojie Wang, Yinghan Cui, Chao Wang, Junyi Peng, Shimiao Jiang, Shiqi Kuang, Shouyu Yin, Chaohang Wen, Haotian Zhang, Bin Chen, Bing Yu
TL;DR
Large-scale RL has shown promise for improving LLM reasoning, but reproducing leading results across mathematical and coding domains is constrained by limited methodological transparency. SRPO combines staged cross-domain training and history resampling with curated data, achieving strong benchmark performance with substantially fewer training steps. The paper reports practical guidance for generalizable large-scale RL.
Problem
Limited methodological transparency and insufficient evidence on cross-domain generalization make it difficult to reproduce large-scale RL reasoning advances across mathematical and coding tasks.
Method
SRPO combines two-stage math-then-code training, history resampling, and a dedicated data curation pipeline for joint mathematical and coding RL.
Results
50.0 pass@1 on AIME24 and 41.6 pass@1 on LiveCodeBench surpass DeepSeek-R1-Zero-Qwen-32B while using about 1/10 of its training steps.
Takeaways & Limitations
SRPO provides an implementation and empirical insights for scaling RL reasoning across mathematical and coding domains.
Takeaways & Limitations
The reproducibility setup adopts a training template without explicit reflective reasoning cues or specific solution strategies.
Abstract
from arXiv · showhide
Recent advances of reasoning models, exemplified by OpenAI's o1 and DeepSeek's R1, highlight the significant potential of Reinforcement Learning (RL) to enhance the reasoning capabilities of Large Language Models (LLMs). However, replicating these advancements across diverse domains remains challenging due to limited methodological transparency. In this work, we present two-Staged history-Resampling Policy Optimization (SRPO), which surpasses the performance of DeepSeek-R1-Zero-32B on the AIME24 and LiveCodeBench benchmarks. SRPO achieves this using the same base model as DeepSeek (i.e. Qwen2.5-32B), using only about 1/10 of the training steps required by DeepSeek-R1-Zero-32B, demonstrating superior efficiency. Building upon Group Relative Policy Optimization (GRPO), we introduce two key methodological innovations: (1) a two-stage cross-domain training paradigm designed to balance the development of mathematical reasoning and coding proficiency, and (2) History Resampling (HR), a technique to address ineffective samples. Our comprehensive experiments validate the effectiveness of our approach, offering valuable insights into scaling LLM reasoning capabilities across diverse tasks.
1. Introduction
SRPO addresses limited transparency and cross-domain gaps in large-scale RL for LLM reasoning with staged training, history resampling, and curated data. It reports strong mathematical and coding performance while reducing ineffective training samples.
- Motivation: SRPO is proposed to address unresolved methodological transparency and insufficiently explored cross-domain generalization in large-scale RL for LLMs.The paper positions SRPO as a framework for mathematical and coding domains.
- Method: The two-stage paradigm trains primarily on mathematical data first, then integrates coding data to build coding skills on an established reasoning foundation.The first stage targets reflective, step-by-step reasoning; the second develops coding ability while preserving reasoning gains.
- Method: History resampling filters consistently correct answers to address zero-advantage groups, producing meaningful gradients and improving sample efficiency.The stated goal is to accelerate convergence by removing ineffective samples.
- Results: 50.0 pass@1 on AIME24 and 41.6 pass@1 on LiveCodeBench surpass DeepSeek-R1-Zero-Qwen-32B while using about 1/10 of its training steps.The comparison baseline reports 47.0 points on AIME and 40.2 on LiveCodeBench.
- Contributions: The paper combines SRPO with a dedicated math-and-code RL data curation pipeline and provides empirical analysis of large-scale RL training behavior.The dataset is described as cleaned, difficulty-categorized, and focused on challenging high-quality problems.
2. Preliminary
GRPO estimates baselines from relative rewards among grouped rollouts, while the training setup adopts a template without explicit reasoning cues. Code prompts with starter code include the problem description and function name.
- GRPO: GRPO generates grouped responses for each question and normalizes their rewards to compute relative advantages.This avoids a critic model and is described as improving training stability.
- Training Template: The adopted DeepSeek-R1-Zero training template provides no explicit reflective cues or solution strategies, leaving reasoning to the model's intrinsic capabilities.This choice is made for reproducibility.
- Training Template: For code questions with starter code, the prompt incorporates both the problem description and the starter's function name.
3. Approach
SRPO addresses cross-domain RL bottlenecks with staged math-and-code training and History Resampling, while using curated data and rule-based rewards. Comparative analyses indicate that staged training supports stronger reasoning patterns across both domains, and History Resampling improves gradient efficiency.
- Two-Stage Training: Directly mixing math and code data creates a response-length conflict that limits mathematical verbosity and coding performance.Math commonly elicits long step-by-step reasoning, whereas code training favors shorter responses.
- Two-Stage Training: SRPO first trains on challenging math data to elicit extended reasoning, then introduces coding data to integrate programming skills.The first stage targets reflective pauses, backtracking, and decomposition; the second builds on that reasoning foundation.
- Comparative Analysis of Training Strategies: Mixed training shows limited response-length growth and inferior benchmark performance, with code responses often producing immediate outputs rather than planned reasoning.This contrasts with the longer reasoning patterns sought for mathematical tasks.
- Comparative Analysis of Training Strategies: Math-only training increases response length and strengthens mathematical reasoning that can generalize to detailed reasoning on coding tasks, whereas code-only training develops less explicit reasoning.Code-only responses remain shorter and often lack substantial step-by-step analysis.
- Comparative Analysis of Training Strategies: Staged training produces detailed mathematical reasoning and structured programming reasoning, including occasional use of code generation as an auxiliary tool for mathematics.The reported behaviors combine domain-specific capabilities with broader procedural reasoning patterns.
- History Resampling: Nearly 50% of sampled groups produce identical rewards, creating minimal reward variance and ineffective gradient updates.This zero-advantage phenomenon is associated with consistently successful rollouts on easier problems.
- History Resampling: History Resampling filters out all-correct samples and retains mixed or all-incorrect samples to preserve positive reward variance and nonzero advantages.It reconstructs the next epoch’s dataset using rollout outcomes recorded during the current epoch.
- History Resampling: History Resampling significantly improves computational efficiency compared with dynamic sampling.Its reported effect on response-length growth is illustrated in Figure 9.
4. Experiments
SRPO experiments evaluate training configuration, benchmark performance, training dynamics, and emergent reasoning behaviors across mathematical and coding tasks. The results report strong benchmark performance, steady improvement, and increasingly reflective solution patterns.
- Evaluation Results: 50.0 pass@1 on AIME 24 and 41.6 pass@1 on LiveCodeBench surpass DeepSeek-R1-Zero-Qwen-32B while matching DAPO on AIME24.The experiments use the same LiveCodeBench version as DeepSeek’s report.
- Training Dynamics: Stage 2 begins after reward growth plateaus, initially lowers overall reward, and is followed by steady reward increases without a significant response-length increase.The initial decrease is attributed to the model’s previously untrained coding capability.
- Training Dynamics: History resampling provides consistently effective gradient updates, increasing informative gradients and improving sample efficiency.The passage connects improved sample efficiency with steady reward growth and enhanced training efficiency.
- Thinking Behavior Analysis: During training, the frequency of self-reflection, correction, and backtracking gradually increases, indicating the development of self-verification ability.The analysis identifies these behaviors as an adaptive behavior during policy optimization.
- Thinking Behavior Analysis: Early training shows little checking or reflection, whereas later responses exhibit stepwise reasoning, numerical substitution, one-by-one verification, and self-optimization.These response modes are reported as emerging as training progresses.
- Thinking Behavior Analysis: The model sometimes writes code after mathematical reasoning to verify solutions, supporting procedural self-correction and multi-method problem solving.The example describes mathematical reasoning followed by spontaneous code-based verification.
5. Conclusion
The conclusion presents SRPO as an implementation for math-and-code reasoning that combines staged training, history resampling, and data curation. It frames the experiments as practical guidance for generalizable large-scale reinforcement learning.
- Conclusion: SRPO combines a two-stage training paradigm, epoch-level history resampling, and a data curation pipeline for math-code joint RL training.The paper describes these as its main methodological components.
- Conclusion: The experiments and empirical insights offer practical guidance for future research on generalizable large-scale reinforcement learning.This is the paper’s stated conclusion about the broader value of its experiments.
Contributions
The supplied passages list the paper’s contributors across four author groups.
- The first contributor group includes Xiaojiang Zhang, Jinghui Wang, Zifei Cheng, Wenhao Zhuang, Zheng Lin, and Minglei Zhang.
- The second contributor group includes Shaojie Wang, Yinghan Cui, Chao Wang, Junyi Peng, Xiaojiang Zhang, Jinghui Wang, and Wenhao Zhuang.
- The third contributor group includes Jinghui Wang, Wenhao Zhuang, Xiaojiang Zhang, Zheng Lin, Shimiao Jiang, Minglei Zhang, Shiqi Kuang, Shouyu Yin, and Chaohang Wen.
- The fourth contributor group includes Haotian Zhang, Bin Chen, and Bing Yu.
A. More Show Cases of the Thinking Behavior
The supplied passages identify examples of self-correction, numerical substitution with verification, and self-optimization in model responses. They provide labels for these behaviors without further explanatory text.
- Thinking Behavior Examples: The examples include responses with self-correction, numerical substitution and one-by-one verification, and self-optimization.These behaviors are identified through the supplied figure labels.