Breaking Training Bottlenecks: Effective and Stable Reinforcement Learning for Coding Models

Zongqian Li; Shaohan Huang; Zewen Chi; Yixuan Su; Lexin Zhou; Li Dong; Nigel Collier; Furu Wei

Source-linked AI summary

Breaking Training Bottlenecks: Effective and Stable Reinforcement Learning for Coding Models

Zongqian Li, Shaohan Huang, Zewen Chi, Yixuan Su, Lexin Zhou, Li Dong, Nigel Collier, Furu Wei

arXiv:2603.07777v1Mar 8, 2026cs.LGcs.CLcs.GL

TL;DR

Modern coding models require training methods and data that accommodate longer outputs and altered training dynamics. The paper introduces MicroCoder-GRPO, a challenging dataset, and a faster evaluator, reporting stronger coding performance and systematic training guidance.

Problem
Traditional reinforcement-learning methods and datasets are less effective for modern coding models with longer outputs, higher reasoning abilities, and changed training dynamics.
Method
The paper combines MicroCoder-GRPO’s three algorithmic modifications with MicroCoder-Dataset, MicroCoder-Evaluator, and analysis of more than thirty controlled experiments.
Results
MicroCoder-GRPO consistently outperforms GRPO and DAPO across datasets, difficulty levels, and model sizes, with gains becoming more pronounced under extended-context evaluation.
Takeaways & Limitations
The study provides 34 training insights and shows that 4K-context training can achieve performance comparable to 6K-context baselines while saving approximately 40–50% computational cost.

Abstract

from arXiv · show

Modern code generation models exhibit longer outputs, accelerated capability growth, and changed training dynamics, rendering traditional training methodologies, algorithms, and datasets ineffective for improving their performance. To address these training bottlenecks, we propose MicroCoder-GRPO, an improved Group Relative Policy Optimization approach with three innovations: conditional truncation masking to improve long output potential while maintaining training stability, diversity-determined temperature selection to maintain and encourage output diversity, and removal of KL loss with high clipping ratios to facilitate solution diversity. MicroCoder-GRPO achieves up to 17.6% relative improvement over strong baselines on LiveCodeBench v6, with more pronounced gains under extended context evaluation. Additionally, we release MicroCoder-Dataset, a more challenging training corpus that achieves 3x larger performance gains than mainstream datasets on LiveCodeBench v6 within 300 training steps, and MicroCoder-Evaluator, a robust framework with approximately 25% improved evaluation accuracy and around 40% faster execution. Through comprehensive analysis across more than thirty controlled experiments, we reveal 34 training insights across seven main aspects, demonstrating that properly trained models can achieve competitive performance with larger counterparts.

1. Introduction

Modern coding models have longer, more variable outputs and training dynamics that make traditional reinforcement-learning methods and datasets less effective. The paper responds with four contributions targeting algorithms, analysis, data quality, and evaluation infrastructure.

GRPO removes the separate value model by sampling multiple responses and computing relative advantages, but recent improvements were mainly validated on mathematical reasoning tasks.
Reinforcement-learning insights for code generation are less developed than in mathematical reasoning because coding requires passing all tests under runtime and other constraints.
GRPO on DeepCoder substantially improves Qwen 2.5 but minimally improves Qwen 3 because the mainstream dataset is harder for Qwen 2.5 and relatively simple for Qwen 3.
Qwen 3 models show increasing response lengths during training, while Qwen 2.5 models show stable or decreasing lengths.
The paper proposes four contributions: MicroCoder-GRPO, systematic analysis across over thirty experiments, MicroCoder-Dataset, and MicroCoder-Evaluator.
MicroCoder-GRPO achieves up to 17.6% relative improvement over strong baselines on LiveCodeBench v6 across multiple model scales.

2. Algorithms

MicroCoder-GRPO modifies GRPO with conditional truncation masking, diversity-determined temperature selection, and KL removal with high clipping. These changes target long-output growth, output diversity, and stable reinforcement-learning dynamics.

GRPO: GRPO samples G outputs per query, estimates relative advantages through group normalization, and optimizes the current policy without a separate value model.
MicroCoder-GRPO: MicroCoder-GRPO selectively masks qualifying maximum-length responses by zeroing their advantages, using correctness, non-repetition, completion, and random-probability criteria.
Conditional Truncation Mask: Higher masking rates accelerate response-length growth but also accelerate diversity decline and reduce diversity convergence values.
Conditional Truncation Mask: Conditional truncation masking achieves higher final performance and greater stability than no masking or complete masking, avoiding the rapid decline seen with complete masking.
Diversity-determined Temperature: Temperature robustness increases during training, while fixed-temperature output diversity tends to decline toward similar final values across settings.
Diversity-determined Temperature: Dynamic low-to-high temperature scheduling outperforms static approaches, whereas brief continuous temperature changes can irreversibly alter output diversity.
No KL Loss and High Clip Ratio: Removing KL loss and using high clipping improves output diversity and response length, supporting sustained performance improvements rather than the decline associated with standard KL loss.

3. Data

MicroCoder’s data pipeline builds challenging coding datasets through staged collection, processing, filtering, and verification. Compared with DeepCoder, MicroCoder produces stronger coding improvements and longer solution responses as training progresses.

Data Processing: MicroCoder’s four-stage pipeline collects diverse data, standardizes and validates it, filters by quality, relevance, and difficulty, then verifies the resulting problems.The stages are Collect, Process, Filter, and Verify.
Dataset Comparison: MicroCoder training yields rapid accuracy gains, whereas DeepCoder training shows minimal performance variation.The difference is reported in Figure 5’s training comparison.
Dataset Comparison: MicroCoder consistently generates lower critic rewards than DeepCoder, indicating greater problem complexity.Both datasets show similar critic-reward growth trends during training.
Dataset Comparison: MicroCoder’s test-set improvements show that training effectiveness on challenging problems translates more directly to generalization performance.The passage contrasts MicroCoder’s test-set gains with DeepCoder’s minimal performance variation.
Response Length: MicroCoder initially produces similar or shorter responses than DeepCoder but develops faster response-length growth and ultimately longer outputs.The passage links longer outputs to the longer solution paths required by challenging coding tasks.

4. Infrastructure

MicroCoder-Evaluator uses comprehensive output validation to assess coding solutions more flexibly and reliably than the LiveCodeBench evaluator. This evaluation approach is associated with higher rewards, fewer misjudgments, and more effective training.

Evaluator Design: MicroCoder-Evaluator compares outputs using 6-7 fallback methods, flexible type handling, approximate numeric matching, preprocessing, and fault tolerance.It continues trying comparison approaches when individual methods fail.
Evaluator Design: LiveCodeBench Evaluator relies on exact matching, Decimal-based numeric comparison, and minimal whitespace preprocessing.These implementation choices are contrasted directly with MicroCoder-Evaluator’s broader validation strategy.
Training Impact: MicroCoder-Evaluator achieves higher critic reward scores, reflecting more accurate assessment of solution quality.The passage associates comprehensive comparison methods with better recognition of valid solution variations.
Training Impact: MicroCoder-Evaluator’s evaluation approach translates to improved training effectiveness through fewer misjudgments, reduced noise, and faster test-accuracy improvement.The passage presents these outcomes as consequences of more comprehensive output validation.

5. Experimental Design

The experiments compare MicroCoder-GRPO with established baselines across datasets, evaluators, temperatures, model scales, and context lengths. The design evaluates both algorithmic robustness and the scalability of extended reasoning budgets.

Evaluation Infrastructure: Figure 6 compares MicroCoder-Evaluator with the LiveCodeBench Evaluator using accuracy and reward during training, plus evaluator execution efficiency.The right subfigure concerns parallel-processing efficiency relative to DeepCoder’s original single-threaded version.
Algorithmic Evaluation: Figure 7 compares MicroCoder-GRPO and baselines across model scales and output lengths, tracking accuracy, response length, and output diversity.The comparison covers both preset and extended context lengths.
Experimental Setup: Other analyses use DeepCoder and MicroCoder datasets, with testing on unseen AtCoder and LeetCode problems from LiveCodeBench v6.The experiments use Qwen3-1.7B Instruct and Qwen3-4B-Instruct-2507 models.

6. Results

MicroCoder-GRPO improves coding performance across datasets, difficulty levels, model sizes, and context settings, with stronger gains under extended contexts. Its conditional truncation masking supports faster, more stable long-term training than baseline approaches.

Training Dynamics: MicroCoder-GRPO achieves faster convergence and higher final accuracy than baselines when trained at 4K response length and tested at both 4K and 8K contexts.The 4K-trained model performs comparably to baselines trained with 6K contexts while saving approximately 40-50% computational cost.
Training Dynamics: MicroCoder-GRPO’s conditional truncation masking maintains stable long-term training dynamics, unlike DAPO’s pronounced performance decrease during extended training.DAPO reaches higher peak performance more rapidly, but its training is variable and declines in later stages.
Model Scale: 4B models exhibit greater response-length growth capacity than 1.7B models, while MicroCoder-GRPO increases length across both scales while preserving diversity and sustained accuracy.The result links model scale to response-length growth capacity without reporting a specific numeric difference.
Benchmark Results: MicroCoder-GRPO consistently outperforms GRPO and DAPO across benchmark datasets, difficulty levels, and model sizes.The reported gains are especially pronounced under extended-context evaluation and on Medium and Hard problems.
Benchmark Results: +3.6% improvement on LiveCodeBench is achieved by 1.7B models trained on 4K contexts and tested on 8K contexts.This result is reported as evidence of better scalability under extended context evaluation.

7. Analysis

Training dynamics depend strongly on batch configuration and context-length scheduling. Off-policy settings improve stability, while longer initial output limits support stronger accuracy, growth, and diversity that later extension cannot fully recover.

Batch configuration: Larger train batch sizes create more off-policy dynamics, whereas smaller sizes produce more on-policy behavior.The framework uses train batch size and PPO mini batch size to determine update iterations per training cycle.
Context length: Longer maximum output lengths correlate with higher final accuracy, faster output growth, greater final lengths, and increased diversity.These relationships are shown across context-length settings in Figure 9.
Context length: Small initial maximum output lengths reduce generation and diversity, producing persistent performance effects after later length extensions.The effects grow with more limiting initial settings and cannot be compensated by subsequent context extension.
Batch configuration: Off-policy configurations increase training stability, while intermediate settings achieve optimal performance.This comparison is reported across different train batch size settings.

8. Conclusions

The paper addresses training bottlenecks in modern coding models with algorithmic, data, and evaluation improvements. Its broader analysis derives systematic reinforcement-learning guidance from controlled experiments and motivates extending these methods to diverse coding tasks.

8. Conclusions: MicroCoder-GRPO combines conditional truncation masking, diversity-determined temperature selection, and KL-loss removal with high clipping ratios.These innovations target long-output potential, training stability, output variability, and solution diversity.
8. Conclusions: The paper introduces a more challenging, higher-quality dataset and a robust code evaluator with faster testing speed.These components complement the proposed training algorithm.
8. Conclusions: More than thirty controlled experiments yield 34 training insights providing systematic guidance for reinforcement learning in code generation.The insights span Sections 2, 3, 4, and 7.
8. Conclusions: Future work will apply the methods and insights to diverse coding tasks and examine their performance, efficiency, and generalization capabilities.The stated direction is to leverage the demonstrated properties across additional code-generation domains.