StepCoder: Improve Code Generation with Reinforcement Learning from Compiler Feedback

• Long code sequences make RL exploration difficult, while unexecuted snippets can make whole-sequence optimization imprecise; existing APPS data also has quality limitations for RL training.Source: §1 · p. 1
• CCCS breaks code generation into progressively harder completion subtasks, while FGO dynamically masks unexecuted code so optimization uses relevant segments.The curriculum increases the portion of code to complete until generation from human requirements alone is the goal.Source: §1 · pp. 1–2
• APPS+ excludes syntax-error, irrelevant, and no-output code segments and standardizes unit-test input and output formats for deterministic comparisons.Source: §1 · p. 2
• StepCoder improves exploration and outperforms other reinforcement-learning methods on code-generation benchmarks.The evaluation includes MBPP and HumanEval.Source: §1 · p. 2
• For the illustrated APPS solution and input, only 75% of the code fragment is executed.Source: Figure 1 · p. 2

• More conditional statements can sharply reduce the probability of generating a correct solution, making exploration harder when unit-test rewards arrive only after complete code execution.The passage describes the probability reduction as potentially exponential as requirements become more complex.Source: §2 · pp. 2–3
• The RL formulation treats generated tokens as actions, token history as state, a generated solution as an episode, and compiler feedback as its reward.Policy updates use an advantage estimated from reward with GAE.Source: §2 · p. 3
• Compiler rewards depend on the code executed by a test, so optimizing all generated actions can produce inaccurate advantages when some code is irrelevant to that reward.In the example, 75% of the code fragment is executed for the given input.Source: §2 · p. 3

3.1 Priliminaries

• The training data pairs each human requirement, canonical solution, and unit-test set with AST-derived conditional-statement spans; code passes the tests to count as a final state.Each span records its start and end positions, and spans are ordered by start position.Source: §3.1 · p. 3

3.2 StepCoder

• CCCS breaks long, sparse-reward code-generation exploration into shorter code-completion subtasks, using a canonical-solution prefix as the prompt.As training progresses, the completion tasks become harder until the model generates from the human requirement alone.Source: §3.2 · pp. 3–4, Figure 2 · p. 4
• CCCS advances its starting point toward the beginning of the canonical solution when the cumulative proportion of correctly generated code segments exceeds threshold ρ.Later in training, the starting point reaches s*=0, so the model uses only the human requirement as input.Source: §3.2 · p. 4
• The curriculum samples conditional-statement starts and sets each sample’s curriculum count to ⌈√Ei⌉, where Ei is its number of conditional statements.Programs with more conditional statements have more independent paths and are sampled more frequently.Source: §3.2 · p. 5
• FGO masks tokens not executed in unit tests and computes the policy-update loss using only executed tokens.The method treats unexecuted code as irrelevant to the unit-test reward, making optimization over the full trajectory imprecise.Source: Figure 2 · p. 4, §3.2 · p. 5

4.1 Dataset Preprocessing

• APPS+ contains 7,456 instances with problem descriptions, canonical solutions, function names, unit tests, and starter code.Preprocessing removes incomplete or irrelevant code, syntax errors, API misuse, and missing dependencies, and standardizes test inputs and outputs.Source: §4.1 · p. 5

4.2 Experiment Details

• MBPP and HumanEval evaluation uses models fine-tuned only on APPS+, while the comparison on APPS+ includes RL baselines using the same DeepSeek-Coder-Instruct backbone.The study reports Pass@k and describes the benchmark evaluation as zero-shot.Source: §4.2 · pp. 5–6

4.3 Experimental Results on APPS+

• 59.7% on Introductory, 23.5% on Interview, and 8.6% on Competition are StepCoder’s APPS+ scores, the highest among the baselines.The authors also report that StepCoder exceeds same-backbone supervised fine-tuning and other RL approaches.Source: Table 1 · p. 6, §4.3 · p. 7

4.5 Results on MBPP and HumanEval

• StepCoder is superior to the compared models on both MBPP and HumanEval after models are fine-tuned on APPS+ and evaluated in a zero-shot benchmark setting.Table 2 reports pass@1 results.Source: Table 2 · p. 7, §4.5 · p. 7

4.4 Ablation Studies

• Ablations show that both CCCS and FGO improve generated-code quality over vanilla PPO; CCCS helps on Competition-level problems, while FGO boosts unit-test pass rate.The authors associate the gains with easier exploration and using executed code snippets in compiler-feedback optimization.Source: §4.4 · p. 7

4.5 Results on MBPP and HumanEval

• Only 0.2% of APPS+ lines overlap more than halfway with MBPP solutions and 7.1% with HumanEval solutions; the authors also report SFT performance degradation on both benchmarks.They suggest single-dataset SFT may impair instruction following and generalization, whereas RL-based methods can improve performance on unseen code-generation tasks.Source: Figure 3 · p. 7, §4.5 · pp. 7–8

4.6 Analysis by Unit Test Results

• StepCoder reduces compilation errors, particularly on Interview- and Competition-level problems, but runtime errors and failures remain more common than compilation errors across LLMs.The authors identify reducing runtime errors as a direction for future work.Source: §4.6 · p. 8, Figure 4 · p. 8

5.1 Large Language Models for Code Generation

• Code-generation research includes pretrained and supervised-fine-tuned language models, while RL methods such as CodeRL, PPOCoder, and RLTF use unit-test feedback but still face sparse-reward exploration challenges.RLTF supplies fine-grained rewards from error locations, yet the cited discussion says these methods fall short on complex tasks.Source: §5.1 · p. 8

5.2 Exploration in Reinforcement Learning

• Prior exploration work includes SFT initialization and process-supervised reward models, but the latter require large preference datasets; StepCoder instead combines conditional-statement-based curriculum starts with coverage-informed FGO.The cited curriculum approaches address gaming and robotic manipulation rather than code generation.Source: §5.2 · p. 8

6 Conclusion

• StepCoder breaks down complicated exploration problems while providing fine-grained optimization.The paper also describes APPS+ as a high-quality dataset specifically constructed for code generation.Source: §6 · pp. 8–9
• Experiments indicate that StepCoder improves generated-code quality via reinforcement learning compared with other approaches.Source: §6 · p. 9

A Instance of the APPS+ Dataset

• An APPS+ dataset example is shown in Figure 5.Source: A Instance of the APPS+ Dataset · p. 11

B Experiments Setup in Detail

• The section elaborates on the compared baselines and implementation details of the method.Source: B Experiments Setup in Detail · p. 11

B.1 Benchmarks

• APPS+ contains Introductory (2,850), Interview (4,020), and Competition (586) instances, with about 25% each selected for validation and test.Its problems average 255.3 words and solutions 21.9 lines; each instance averages three unit tests and includes a conditional-statement attribute.Source: §B.1 · p. 11
• MBPP comprises 974 crowd-sourced introductory Python problems, each with a task description, code solution, and three automated test cases.Models are fine-tuned only on APPS+ and then evaluated on MBPP.Source: §B.1 · p. 11
• HumanEval consists of 164 hand-written Python problems covering language comprehension, algorithmic thinking, and basic mathematics.The models are evaluated on HumanEval in a zero-shot setting.Source: §B.1 · p. 11

B.2 Baselines

• StarCoder [15] is a 15.5B-parameter model trained on more than 80 programming languages and fine-tuned on 35 billion Python tokens.Its training corpus contains one trillion tokens, and its context length is 8K.Source: §B.2 · p. 11
• CodeLlama [27] spans 7B–34B parameters and has base, Python-focused, and instruction-following variants.Source: §B.2 · p. 11
• WizardCoder [23] is fine-tuned on complex code-related instructions adapted from Evol-Instruct [42].The passage describes Evol-Instruct as an improvement on self-instruct [40].Source: §B.2 · p. 11
• DeepSeek-Coder [8] comprises 1B–33B models trained from scratch on two trillion code and natural-language tokens.The models use a 16K context window and include a fill-in-the-blank training task.Source: §B.2 · p. 11
• PPOCoder [33] combines PPO [30] with discrete compiler feedback and syntax and semantics matching scores to guide code generation.Source: §B.2 · pp. 11–12
• RLTF [20] generates training data in real time, uses multigranularity unit-test feedback, and penalizes code sections identified as erroneous.Source: §B.2 · p. 12

D The prompts used for code generation

• Prompts for PPOCoder, RLTF, and StepCoder are aligned with DeepSeek-Coder-Instruct, whose prompt instructs Python algorithm generation from the task description.Prompts for CodeLlama, WizardCoder, and StarCoder follow previous studies.Source: D The prompts used for code generation · p. 12

C The algorithm of CCCS and FGO

• The algorithm initializes a curriculum stage and stride for each training sample using its number of code segments.Source: D The prompts used for code generation · p. 13
• FGO forms a mask that distinguishes executed from unexecuted code snippets after calculating the reward.Source: D The prompts used for code generation · p. 13