Source-linked AI summary
DRIVE: Data Curation Best Practices for Reinforcement Learning with Verifiable Reward in Competitive Code Generation
Speed Zhu, Jianwei Cai, Guang Chen, Lulu Wu, Saiyong Yang, Wiggin Zhou
TL;DR
Competitive-programming code generation and RLVR prompt curation remain less explored than mathematics benchmarks and algorithm design. The paper proposes a two-stage RL approach that expands exploration before focusing on difficult problems. It reports state-of-the-art performance among 32B models, with improvements across benchmarks.
Problem
RLVR research has focused largely on mathematics and algorithm design, while competitive-programming applications and prompt construction remain comparatively underexplored.
Source: §1 · pp. 1–2Method
The paper combines distilled, augmented SFT with two RL stages: entropy expansion on broad problem data, followed by Pre-GRPO hard-focus training on challenging cases.
Source: §1 · p. 2, §3.1 · p. 3, §3.2 · p. 4Results
The method achieves state-of-the-art performance among 32B models, with improvements ranging from 13% to 58% across various benchmarks.
Source: §5 · p. 9Takeaways & Limitations
The analysis identifies large rollout budgets and progressive retention of the hardest problems as key elements for mastering challenging cases.
Source: §5 · p. 9The approach requires substantial computational resources, and the reported experiments were not continued to full convergence.
Source: §4.6 · p. 9, §5 · p. 10
Abstract
from arXiv · showhide
Recent reasoning-first models (e.g., OpenAI o1, DeepSeek R1) have spurred a resurgence of interest in RLVR. Nevertheless, advances are dominated by mathematics (e.g., AIME), with competitive-programming code generation underexplored and data curation receiving less attention than RL algorithm design. We investigate how to construct RLVR datasets (i.e., RL prompts) and present practical training techniques that yield strong performance on competitive-programming code generation. Our pipeline begins with supervised fine-tuning (SFT) distilled from strong open-source models, augmented with general-purpose and reasoning-intensive data. RL then follows a two-stage process with executable, testcase-driven rewards: first, training on a large, uniformly distributed set of competitive-programming problems using Group Relative Policy Optimization (GRPO) with 8 rollouts per prompt and a relatively short response-generation window (e.g., 32k during SFT and 24k in this stage) to expand entropy and mitigate repetition and truncation; second, we perform \textbf{Pre-GRPO}: updating on a small, high-quality set of challenging problems with a large rollout budget (64 rollouts per prompt) under a hard-focus curriculum that continuously retains the most difficult instances throughout training. We implement our method on Qwen2.5-32B and evaluate on LeetCode and Codeforces weekly contests to avoid data leakage. The resulting model achieves state-of-the-art performance among models of similar scale and is comparable to leading systems such as DeepSeek v3.1 and Doubao-1.5-Thinking. We also examine scaling trends and observe strong RL scaling on an internal large-scale MoE model. Our study distills concise best practices for data curation, entropy expansion, and curriculum design in RLVR for competitive-programming code generation.
1 Introduction
The paper studies RLVR for competitive-programming code generation, a less-explored application than mathematics, and focuses on data curation and curriculum design. Its two-stage framework expands exploration before focusing training on challenging problems.
1 Introduction
- Competitive-programming code generation remains relatively underexplored despite requiring logically correct, executable, efficient solutions that handle edge cases under strict computational constraints.Source: §1 · p. 1
- RLVR research has focused more on algorithms than prompt curation, while uniform training or simple difficulty categories may limit performance on the hardest problems.Large-scale training costs also make extensive experimentation difficult.Source: §1 · p. 2
- The proposed two-stage RL framework expands exploration entropy first, then applies a hard-focus curriculum to challenging problems.Source: §1 · p. 2
- The authors report that moderate rollout budgets suffice for entropy expansion, while large budgets are crucial for learning challenging problems; standard RL struggles with difficult cases.Source: §1 · p. 2
- The 32B model achieves state-of-the-art performance among similar-scale models, comparable performance to models with 5–10× more parameters, and up to 58% relative improvement over similar-sized models on Codeforces.The evaluation uses recent LeetCode and Codeforces weekly contests selected to avoid data contamination.Source: §1 · p. 2
- The study offers practical guidance on data curation, difficulty-aware curricula, and scaling, and argues that careful curricula and strategic computational resources can improve competitive-programming capabilities.Source: §1 · p. 2
2 Related Work
Prior RLVR research has advanced algorithms and training techniques, but has paid less attention to constructing suitable prompts, especially for competitive-programming code generation. The paper also addresses limited evidence about scaling RLVR beyond commonly evaluated dense models.
RLVR Algorithms
- RLVR studies have largely emphasized algorithm and training advances, leaving the construction of suitable RL prompts comparatively underexplored.Source: §2 · p. 2
RLVR Data Construction
- RL data construction for competitive-programming code generation remains largely unexplored, and this study examines its performance and scaling trends.Source: §2 · p. 2
RLVR performance scaling analysis
- Most RLVR evaluations focus on dense 7–32B models, leaving larger-scale behavior less studied; this work applies its Qwen2.5-32B recipe to an internal large-scale MoE model.Source: §2 · p. 3
3 Method
The method combines supervised fine-tuning with two RL stages. The first broadens exploration on a mixed problem set; the second uses Pre-GRPO to focus training on difficult cases.
3 Method
- The pipeline consists of supervised fine-tuning followed by a two-stage reinforcement-learning process with verifiable rewards.Source: §3 · p. 3
3.1 Supervised Fine-Tuning
- SFT distills strong open-source models and augments training with general-purpose coding and reasoning-intensive data; hard problems are duplicated to emphasize difficulty efficiently.The paper also reports that including general-purpose coding and reasoning-intensive problems significantly improves competitive-programming performance.Source: §3.1 · p. 3
3.2 Two-Stage Reinforcement Learning
- The SFT model shows low entropy, repetitive or truncated outputs, and weaker pass@10 on the harder Codeforces Weekly OJ than on LeetCode, motivating the two-stage RL design.Source: §3.2 · p. 4
- Stage 1 trains on approximately 9k mixed competitive-programming problems with 8 rollouts and a 24k-token total sequence length to expand entropy and reduce repetitive, truncated generations.Source: §3.2 · p. 4
- Stage 2 uses Pre-GRPO to carry low-pass-rate cases forward through a three-phase hard-focus curriculum, sampling 80 rollouts per prompt to target increasingly challenging problems.The phases use the 72, 50, and 25 hardest cases, respectively.Source: §3.2 · p. 4
4 Experiments
Across the experiments, the two-stage RL approach improves competitive-programming performance, while analyses examine the roles of hard-example training, rollout budgets, and curriculum design. The scaling study reports gains on an internal large-scale MoE model but notes that training did not reach full convergence.
4.1 Experimental Setup
- Experiments use Qwen2.5-32B, 470K distilled SFT prompts, 9K first-stage RL prompts, and 175 LiveCode V6 problems for Stage 2.Stage 1 uses 8 rollouts per prompt and a 24k-token sequence length; Stage 2 uses 64 rollouts and 32k tokens. Recent LeetCode and Codeforces contests are used to avoid data leakage.Source: §4.1 · p. 5
4.3 SFT Experiments
- Twice Hard Learning, which oversamples identified hard examples while restoring the training-token budget, performs best across the evaluated SFT benchmarks.Arena Learning reduces the dataset from 1.27M to 470K prompts while maintaining performance comparable to Basic SFT.Source: Table 3 · p. 6, §4.3 · p. 6
4.4 The RL Experiments
- The full two-stage RL method performs best across benchmarks; training on unfiltered easy and hard problems can fail to reach the best performance or collapse.Removing Stage 1 yields mixed results, including a 4.8% drop on LeetCode weekly OJ despite a 25.9% gain on contaminated LiveCodeV6.Source: Table 4 · p. 6, §4.4 · pp. 6–7
4.2 Experimental Results
- The RL model improves 13.0% on LiveCode weekly OJ and 58.3% on Codeforces weekly OJ, and outperforms the 32B OpenReasoning-Nemotron baseline across benchmarks.The reported benchmark results also show consistent gains over the SFT baseline.Source: §4.2 · pp. 5–6
4.5 Further Analysis
- Under standard RL, medium-initial-accuracy cases improve fastest, while low-accuracy cases stagnate; even 60 steps on a single hard case may not yield satisfactory performance.The analysis motivates targeting hard cases with large rollouts.Source: §4.5 · p. 7
- Adding 109 hard cases selected from 633 candidates to Stage 2 improves results consistently across the reported benchmarks.The study presents targeted hard-case inclusion as a scalable lever under limited compute.Source: §4.5 · p. 8
4.6 Scaling Trends
- On the internal MoE model, the final stage improves results across benchmarks, including +15.17% on LeetCode Weekly OJ and +7.51% on LiveCodeBench-V5 versus SFT.The experiments were resource-constrained and did not continue to full convergence.Source: Table 5 · p. 9, §4.6 · p. 9
5 Conclusion
The paper concludes that a two-stage RL approach combining entropy expansion with hard-focus curriculum learning achieves strong competitive-programming results. Its analyses identify difficult-case learning, rollout budgets, and curriculum design as important considerations, while noting the computational cost of large-scale rollout sampling.
5 Conclusion
- The two-stage framework combines entropy expansion with hard-focus curriculum learning and achieves state-of-the-art performance among 32B models, with 13%–58% improvements across benchmarks.The conclusion frames data curation and curriculum design as crucial alongside algorithmic innovation for RLVR.Source: §5 · p. 9
- The analysis finds that standard RL struggles on hard problems, large rollout budgets of 64+ samples are essential for challenging cases, and retaining the hardest cases outperforms uniform difficulty distribution.Ablations attribute robust generalization to entropy expansion and frontier-pushing to the hard-focus curriculum.Source: §5 · p. 9
- Scaling experiments on an internal large-scale MoE model support transfer of the training principles to larger scales, but large-rollout sampling requires substantial computation.The paper identifies adaptive curricula and more efficient sampling as future directions.Source: §5 · p. 10
A Few Cases Training
Few-case experiments reveal uneven learning across problems: some cases converge quickly, while difficult cases remain stagnant. Training multiple cases can help one difficult case, motivating the paper’s hard-case strategy.
A Few Cases Training
- With GRPO and 8 rollouts, two of four sampled cases rise to about 80% accuracy within 60 steps, while another reaches only 36.7% and one remains at 12.5%.The four cases were randomly selected from LiveCodeBench V6’s 175 problems.Source: A Few Cases Training · p. 12, Figure 7 · p. 12
- Case 1 learns more slowly in single-case training than in the four-case setting, which the authors interpret as evidence of positive transfer across problems.Case 1 plateaus at 36.7% after 60 steps, while Case 4 stagnates, motivating large-rollout training focused on hard cases.Source: A Few Cases Training · pp. 12–13, Figure 8 · p. 12
- The case-level results suggest standard RL favors easily learnable patterns while failing to break through on some challenging problems.The authors propose large rollout budgets in the second stage to address these learning barriers.Source: A Few Cases Training · pp. 12–13
B Case Study for Repetition Patterns
A case study of a graph-isomorphism optimization problem illustrates repetitive and cyclical reasoning in a model trained without extensive 24k-style data. The authors suggest that diverse reasoning examples within a 24k context may reduce these patterns.
B Case Study for Repetition Patterns
- The case study describes repeated permutation checks, edge-by-edge calculations, sample processing, and trial-and-error cycles in the model’s reasoning.The authors characterize these patterns as computational redundancy and missed opportunities for abstraction.Source: B Case Study for Repetition Patterns · pp. 13–15
- The authors suggest that without diverse reasoning examples in a 24k context, models tend toward repetitive computational habits with limited efficiency and abstraction.The displayed analysis is based on GPT-5’s review of the RL model’s response.Source: B Case Study for Repetition Patterns · p. 13
- The analysis identifies redundant calculations in the think section that could be simplified by extracting common functions and using systematic verification workflows.Examples include repeated permutation-cost calculations and repeated input-processing logic.Source: B Case Study for Repetition Patterns · pp. 13, 15