PaperlayerView exact revision on arXiv

Source-linked AI summary

ACECODER: Acing Coder RL via Automated Test-Case Synthesis

Huaye Zeng, Dongfu Jiang, Haozhe Wang, Ping Nie, Xiaotong Chen, Wenhu Chen

arXiv:2502.01718v4cs.SEcs.AIcs.CL

TL;DR

Reinforcement learning for code generation remains underexplored amid challenges in coder-model training. ACECODER uses automated large-scale test synthesis to train reward models and tune coder models, improving best-of-N performance and producing gains across reinforcement-learning evaluations, including 25% on HumanEval-plus and 6% on MBPP-plus from 80 optimization steps.

  • Problem

    Reinforcement learning has been much less explored than supervised fine-tuning for coder models because of key training challenges.

  • Method

    ACECODER synthesizes large-scale reliable test cases, uses pass rates from sampled programs to train Bradley-Terry reward models, and applies reinforcement learning to coder models.

  • Results

    The method improves coding performance across multiple benchmarks, including 25% on HumanEval-plus and 6% on MBPP-plus after 80 optimization steps from Qwen2.5-Coder-7B-Base.

  • Takeaways & Limitations

    ACECODER substantially improves Best-of-N performance, while its reinforcement-learning gains are less pronounced and remain an area for future improvement.

  • Takeaways & Limitations

    Synthesized test cases do not necessarily guarantee program correctness, introducing noise into reward-model training and reinforcement-learning signals.

Abstract

from arXiv · show

Most progress in recent coder models has been driven by supervised fine-tuning (SFT), while the potential of reinforcement learning (RL) remains largely unexplored, primarily due to the lack of reliable reward data/model in the code domain. In this paper, we address this challenge by leveraging automated large-scale test-case synthesis to enhance code model training. Specifically, we design a pipeline that generates extensive (question, test-cases) pairs from existing code data. Using these test cases, we construct preference pairs based on pass rates over sampled programs to train reward models with Bradley-Terry loss. It shows an average of 10-point improvement for Llama-3.1-8B-Ins and 5-point improvement for Qwen2.5-Coder-7B-Ins through best-of-32 sampling, making the 7B model on par with 236B DeepSeek-V2.5. Furthermore, we conduct reinforcement learning with both reward models and test-case pass rewards, leading to consistent improvements across HumanEval, MBPP, BigCodeBench, and LiveCodeBench (V4). Notably, we follow the R1-style training to start from Qwen2.5-Coder-base directly and show that our RL training can improve model on HumanEval-plus by over 25\% and MBPP-plus by 6\% for merely 80 optimization steps. We believe our results highlight the huge potential of reinforcement learning in coder models.

1 Introduction

ACECODER targets the limited use of reinforcement learning for code generation by synthesizing and filtering large-scale test cases, then using them for reward modeling and RL. The resulting reward models improve best-of-N coding performance and RL yields further gains across multiple benchmarks.

  • Approach: ACECODER synthesizes question–test-case pairs from seed code data, filters noisy tests with strong models, and uses them to construct program preferences for reward-model training and RL.The workflow uses test cases to harvest positive and negative program pairs.
  • Motivation: Reinforcement learning remains underexplored for coder models because reliable code-generation rewards and large-scale coding data are difficult to obtain.Code quality typically requires execution-based test-case evaluation, complicating reward design.
  • Results: 10.7 points is the average improvement for Llama-3.1-8B-Instruct with ACECODE-RM-32B across HumanEval, MBPP, BigCodeBench, and LiveCodeBench.The corresponding improvement with ACECODE-RM-7B is 8.4 points.
  • Results: 4.7 points is the average improvement for Qwen2.5-Coder-7B-Instruct with ACECODE-RM-32B across four coding benchmarks.The 7B reward-model combination produces a 2.6-point average improvement for this model.
  • Results: ACECODE-RM-32B achieves the highest RM Bench average score of 76.1 and leads in Coding, Chat, Normal, and Hard categories.ACECODE-RM-7B is particularly competitive in Coding and Hard categories.
  • Results: 25% improvement on HumanEval-plus and 6% improvement on MBPP-plus follow RL from Qwen2.5-Coder-7B-base after 80 optimization steps.The paper also reports improvements generalizing to more difficult benchmarks.

2 Methodology

ACECODER trains code-generation reward models and policies using synthesized, filtered test cases and pass-rate preferences. The methodology combines Bradley-Terry reward modeling, Best-of-N selection, and reinforcement learning, while addressing PPO’s additional value-model requirement.

  • Reward-model training: The Bradley-Terry objective trains the reward model to assign higher scores to preferred programs and lower scores to non-preferred programs.The loss is applied to pairs of sampled responses whose target rewards are test-case pass rates.
  • Best-of-N sampling: After training, Best-of-N sampling selects the response with the highest predicted reward-model score among sampled candidates.The reward model is used for test-time selection rather than directly modifying the policy.
  • Reinforcement learning: PPO updates the policy using reward-model outputs, advantages, and clipped probability ratios, but ordinarily requires an additional value model.Reinforcement++ is described as eliminating the value-model requirement by computing advantages from rewards and token-level KL divergence.
  • ACECODE-87K construction: ACECODER synthesizes reliable test cases from coding questions, filters hallucinated tests with a stronger coder model, and constructs ACECODE-87K.The final dataset contains 87.1K coding questions and 1.38M cleaned test cases.
  • Quality verification: The methodology verifies synthesized-test quality through filtering and a human case study of 200 manually annotated test cases.The cited case study found 3 invalid test cases among 200 reviewed.
  • Preference-pair construction: Sampled programs are ranked by test-case pass rates to construct positive and negative preference pairs for reward-model training.The selection rules account for pass-rate differences and require preferred programs to exceed specified quality thresholds.

3 Experiments

ACECODER evaluates reward-model selection and reinforcement learning using automated test-case rewards across coding benchmarks. The experiments show gains from reward-model backbone choice, test-case filtering, Best-of-N selection, and RL from both instruct and base models, while also identifying reward hacking as a limitation.

  • Reward Model Training Setup: ACECODER trains reward models primarily from Qwen2.5-Coder-7B-Instruct and creates around 300K preference pairs from 46,618 questions.The reward-model training uses 16 sampled responses per question and retains questions with at least one qualifying preference pair.
  • Reward Model Results: Best-of-N sampling consistently improves inference-model performance, with gains exceeding 10 points on weaker Mistral and Llama-3.1 models.The reward model selects the highest-scored completion among sampled candidates; gains are larger when sampled-completion quality varies more.
  • Reward Model Results: ACECODE-RM is effective when Best-of-N performance surpasses greedy decoding, whereas general-purpose reward models may fail to improve or decrease performance.The comparison attributes ACECODE-RM’s advantage to more reliable identification of correct generated programs.
  • Reinforcement Learning Results: RL tuning improves performance across initial policies and reward types, including more than 3 points on Plus benchmarks from Qwen2.5-Instruct-7B.Experiments use Reinforcement++ with both ACECODE-RM and binary test-case pass-rate rewards.
  • Reinforcement Learning Results: RL from Qwen2.5-Coder-7B-base yields significant improvements on HumanEval, MBPP, and BigCodeBench-I after only 80 training steps.The authors report especially strong gains from test-case pass rewards and suggest that further scaling could produce larger gains.
  • Ablation Studies: Test-case filtering significantly improves reward-model performance on difficult questions, while Qwen2.5-Coder backbones outperform Llama-3.1 backbones for Best-of-16.The paper attributes filtering gains to greater consistency among retained tests and backbone gains to more code-related pretraining data.

4 Related Works

Prior work uses automated test generation and execution-based rewards for code, but scalable RL and coding reward models remain constrained by unreliable tests and limited annotated data.

  • Test-case generation: Automatic test generation verifies LLM-generated programs, but self-generated tests can hallucinate; Algo instead filters tests with an oracle solution.This paper uses a stronger coder LLM rather than exhaustive variable enumeration to generate the oracle solution.
  • Reinforcement learning: RL coding methods use execution accuracy, PPO, granular line-level rewards, or self-generated tests across single- and multi-turn settings.Examples include PPOCoder, CodeRL, RLEF, StepCoder, and DSTC.
  • Research gap: Prior coding RL approaches rely mainly on APPS, which has only 5,000 examples and usually one test case per problem.The limited data availability challenges scalable RL training.
  • Research gap: The potential of reward models for coding remains largely unexplored despite RL progress in conversational interaction and mathematical reasoning.Existing RL algorithms train models using learned reward models or other reward signals.

5 Conclusion

ACECODER trains reward models and tunes code-generation policies using large-scale, reliable synthesized test cases. The approach improves Best-of-N performance, while its RL gains are less pronounced and remain an area for future work.

  • Contribution: ACECODER trains reward models and performs RL tuning for code generation using large-scale, reliable test-case synthesis.Its data pipeline produces verifiable code without relying on the most advanced models.
  • Results: ACECODER significantly improves Best-of-N performance.The conclusion distinguishes these gains from the less pronounced improvements observed during RL training.
  • Future work: RL training gains are less pronounced than the Best-of-N improvements, leaving enhancement of RL performance as future work.

Limitations

The paper identifies two main limitations: synthesized tests can be inaccurate, and current RL gains depend on the model being tuned and potentially noisy rewards.

  • Test-case synthesis: Passing all synthesized tests does not guarantee program correctness, introducing noise into reward-model training and RL tuning signals.Errors can arise from incorrect reference solutions or tests that miss difficult edge cases.
  • RL scope: The RL study uses three models and two reward types: RM-based and rule-based.This defines the scope of the reported RL experiments.
  • RL limitations: RL gains are less pronounced when tuning Qwen2.5-Coder-7B-Instruct because of its strong initial ability.The authors suggest current reward signals may still contain noise.
  • Future work: Further improvement is especially needed for tuning Qwen2.5-Coder-7B-Base through finer-grained reward design.The authors identify this as a future direction alongside refined RL tuning strategies.

Ethical Statements

The authors state that the work complies with the ACL Ethics Policy and that they know of no ethical issues in the paper.

  • Ethical statement: The paper declares compliance with the ACL Ethics Policy and reports no known ethical issues.

A.1 More related works

Coding models are specialized LLMs trained through pre-training and fine-tuning, while reward models assign scalar values to response pairs for alignment and test-time selection. General-purpose reward models may struggle with code-specific evaluation.

  • Coding models such as Code Llama and Qwen Coder were developed for code-generation tasks.
  • These models typically undergo pre-training followed by fine-tuning.
  • Reward models assign scalar values to response pairs according to criteria such as human preference or accuracy.
  • Reward models support reinforcement learning with human feedback and Best-of-N sampling.
  • General-purpose reward models are effective for human-preference assessment but often struggle with code-specific evaluation.

A.2 ACECODE-RM Model Breakdown

This section identifies the models involved in ACECODE-RM training and evaluation and presents them in tabular form.

  • The training and evaluation models for ACECODE-RM are presented in a concise table.
  • Table 9 lists the models used during ACECODE-RM training and evaluation.

A.3 Prompt Template

The appendix includes prompt templates for converting seed code into LeetCode-style questions and test cases, alongside HumanEval case studies illustrating RL-model behavior. These examples cover an incorrect regular-expression solution, an error caught by RL, and a runtime improvement that avoids timeout.

  • A.3 Prompt Template: Table 10 presents a prompt for converting a seed code dataset into LeetCode-style questions and test cases.
  • A.3 Prompt Template: Table 11 presents a prompt that performs the conversion using only the reference program without an instruction.
  • A.4 Case Studies on HumanEval: Figure 2 shows an RL model attempting regular-expression matching but omitting whitespace handling and mishandling the three-digit filename constraint.
  • A.4 Case Studies on HumanEval: Figure 3 shows the RL model catching a baseline error involving repetition of the whole string to create a palindrome.
  • A.4 Case Studies on HumanEval: Figure 4 reports identical outputs but a runtime for the RL model that is half the baseline’s because it uses a for loop instead of recursive calls.
  • A.4 Case Studies on HumanEval: The Figure 4 case passed the test for the RL model, whereas the baseline received a time-out.
Loading 2502.01718v4…