Source-linked AI summary
CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learning
Hung Le, Yue Wang, Akhilesh Deepak Gotmare, Silvio Savarese, Steven C. H. Hoi
TL;DR
Program synthesis models often overlook unit-test signals and rely on token prediction despite its mismatch with functional correctness. CodeRL combines actor-critic reinforcement learning with critical sampling and an improved CodeT5 backbone, achieving new state-of-the-art results on APPS and MBPP while requiring human verification for security concerns.
Problem
Pretrained language models for program synthesis commonly use next-token supervision while neglecting unit-test signals that directly determine functional correctness.
Method
CodeRL treats the code-generating language model as an actor, trains a correctness-predicting critic for dense feedback, and uses critic-guided critical sampling during inference.
Results
CodeRL achieves new state-of-the-art results on APPS and MBPP, including 2.69% pass@1, 6.81% pass@5, 20.98% pass@1000, and 63.0% pass@80 on MBPP.
Takeaways & Limitations
The framework consistently improves pretrained language models for code generation and transfers zero-shot from APPS training to unseen MBPP tasks.
Takeaways & Limitations
Because pretraining code may contain vulnerabilities, CodeRL outputs warrant verification by qualified human developers.
Abstract
from arXiv · showhide
Program synthesis or code generation aims to generate a program that satisfies a problem specification. Recent approaches using large-scale pretrained language models (LMs) have shown promising results, yet they have some critical limitations. In particular, they often follow a standard supervised fine-tuning procedure to train a code generation model only from the pairs of natural-language problem descriptions and ground-truth programs. Such paradigm largely ignores some important but potentially useful signals in the problem specification such as unit tests, which thus often results in poor performance when solving complex unseen coding tasks. To address the limitations, we propose "CodeRL", a new framework for program synthesis tasks through pretrained LMs and deep reinforcement learning (RL). Specifically, during training, we treat the code-generating LM as an actor network, and introduce a critic network that is trained to predict the functional correctness of generated programs and provide dense feedback signals to the actor. During inference, we introduce a new generation procedure with a critical sampling strategy that allows a model to automatically regenerate programs based on feedback from example unit tests and critic scores. For the model backbones, we extended the encoder-decoder architecture of CodeT5 with enhanced learning objectives, larger model sizes, and better pretraining data. Our method not only achieves new SOTA results on the challenging APPS benchmark, but also shows strong zero-shot transfer capability with new SOTA results on the simpler MBPP benchmark.
1 Introduction
CodeRL addresses limitations of pretrained language models for program synthesis by using unit-test signals in reinforcement learning and inference-time regeneration. Combined with an improved CodeT5 foundation model, it achieves strong results on APPS and MBPP.
- Motivation: Program synthesis generates executable programs that satisfy natural-language problem specifications, with potential impacts on programming productivity and accessibility.Pretrained Transformer language models frame synthesis as sequence-to-sequence generation from problem descriptions to code.
- Limitations: Next-token prediction can cause accumulating generation errors, while token-matching metrics correlate poorly with functional correctness.These limitations make conventional supervised training insufficient for complex program synthesis tasks.
- Limitations: Unit tests provide meaningful correctness signals that current approaches neglect during both model optimization and generation.The tests can be incorporated into learning objectives and used as example feedback during inference.
- CodeRL: CodeRL uses an actor-critic reinforcement-learning strategy in which a pretrained language model generates synthetic programs and a critic provides correctness-related feedback.The framework treats the pretrained LM as the actor and trains a critic to evaluate generated programs.
- CodeRL: At inference, critical sampling uses example unit tests and critic scores to seed resampling, select promising failed programs, and automatically refine or repair outputs.The procedure combines regeneration from selected subsequences with candidate-based repair.
- Results: More than 2% pass@1, 6% pass@5, and 20% pass@1000 were achieved on APPS, while MBPP reached 63.0% pass@80 versus GPT-137B’s 61.4%.The approach also uses improved CodeT5 pretraining objectives, larger model sizes, and more pretraining data.
2 Related Work
Related work spans program synthesis, visual program generation, reinforcement learning for sequence generation, and code completion. CodeRL differs by targeting general-purpose programs from natural-language specifications and using unit-test signals tied to functional correctness.
- Program Synthesis: Earlier program-synthesis tasks used input-output examples and were limited to probabilistic methods or simple programming concepts.Later work expanded synthesis toward general-purpose programming languages and natural-language specifications.
- Visual Program Synthesis: Visual program-synthesis research infers scene, shape, or graphics programs for image and video applications.These systems typically use domain-specific languages defined for visual objects rather than general-purpose programming languages.
- RL for Sequence Generation: Reinforcement learning has been applied to sequence generation by optimizing nondifferentiable task metrics such as BLEU and ROUGE, including with actor-critic methods.These approaches provide precedents for applying RL to generation problems.
- RL for Program Synthesis: In program synthesis, token-based similarity metrics show very low correlation with functional correctness, motivating unit-test signals during optimization and test-time generation.Unit tests directly expose whether generated programs function correctly.
- Code Completion: Code completion generates code conditioned on partial programs, whereas program synthesis generates complete programs from scratch and evaluates them through unit tests.CodeRL focuses on synthesis from natural-language problem descriptions while adopting a related candidate-generation strategy.
3 CodeRL
CodeRL formulates program synthesis as reinforcement learning, using an actor-critic framework that exploits unit-test signals during optimization and inference. It extends CodeT5 and uses critic-guided refining and repairing to improve generated programs.
- 3 CodeRL: The framework extends CodeT5 with a multilingual encoder-decoder backbone, larger Python pretraining data, and a next-token prediction objective aligned with synthesis.The GCPY dataset contains 10.5B Python tokens and is 10x larger than CodeSearchNet; next-token prediction is added to reduce the gap between pretraining and synthesis.
- 3 CodeRL: CodeRL treats the pretrained language model as a stochastic policy whose token actions generate synthetic programs for reinforcement-learning optimization.The model receives returns based on the functional correctness of completed programs, and policy-gradient methods estimate updates from these non-differentiable returns.
- 3.3 Program Synthesis as an RL Problem: A critic predicts program outcomes across CompileError, RuntimeError, FailedTest, and PassedTest, using its representations to estimate returns for actor updates.Programs and problem specifications are inputs to the critic, whose hidden states provide feedback beyond a single terminal correctness signal.
- 3.3.1 Defining Return by Unit Test Signals: Heuristic returns assign -1.0 to compilation failures, -0.6 to runtime errors, -0.3 to failed tests, and +1.0 to programs passing all tests.Because these returns can produce high-variance gradients, CodeRL compares generated samples with a greedy baseline and uses relative returns.
- 3.3.4 Generating Programs with Example Unit Tests and Critic: During inference, Critic Sampling refines passed programs from critic-selected subsequences and repairs failed programs selected by predicted test-passing probability.The repair model receives problem descriptions, buggy programs, test outcomes, and error subtypes; refining re-samples partial programs from seed sequences.
4 Experiments
Experiments show that CodeRL improves program synthesis across APPS metrics, model backbones, training objectives, sampling procedures, and zero-shot transfer to MBPP, while example-test limitations remain.
- APPS results: 2.69% pass@1, 6.81% pass@5, and 20.98% pass@1000 establish new APPS SOTA results for CodeRL+CodeT5.The same model also reaches 8.48% 1@k and 12.62% 5@k on filtered samples.
- APPS results: CodeRL can match AlphaCode’s performance with k = 1000, compared with AlphaCode’s much larger generation budget of k = 50000.Performance also benefits from increasing the number of generated samples from 1 to 1000.
- Ablation studies: Relative token-level returns from a trained critic achieve the best pass@1 and pass@5 performance, whereas absolute returns without a baseline produce the largest drop.Rule-based linear-decay returns also underperform critic-based estimates.
- Ablation studies: Combining cross-entropy and reinforcement-learning objectives gives more consistent gains, while using only reinforcement learning causes vanishing gradients and using only cross-entropy degrades test performance.The cross-entropy-only degradation is attributed to overfitting the training data.
- Ablation studies: Critical sampling improves all metrics most strongly when program refining and repairing are combined, although refining alone has reduced impact on 1@1000 and n@k.The authors recommend candidate ranking as additional post-processing for refining, particularly on n@k metrics.
- Zero-shot transfer: 63.0% pass@80 on MBPP surpasses GPT-137B’s 61.4% pass@80 in zero-shot evaluation, while APPS-to-MBPP program overlap is generally minimal.More than half of MBPP programs have zero overlap with APPS training data, and 90.9% have no more than three overlapping lines.
- Test outcomes: CodeRL increases the likelihood of passing example and hidden tests and reduces compiling errors, but CodeRL programs still show runtime errors and example tests are less comprehensive.The gap between example- and hidden-test outcomes limits the positive impact of the generation procedure through false positives.
5 Limitations and Broader Impacts
The paper discusses societal benefits of program synthesis alongside risks from biased or insecure generated code, added critic-training cost, and persistent alignment failures.
- Program synthesis could transform software tools, increase developer productivity, and improve programming education accessibility and quality.
- Code models may generate biased code or toxic natural-language comments, motivating alignment and safety controls beyond functional correctness.
- Public GitHub pretraining data may contain vulnerabilities, so CodeRL outputs require verification by qualified human developers.
- Training the critic adds computational cost, although a GPT-2-small critic achieved over 75% error-prediction accuracy on synthetic samples.
- CodeRL can still produce code that fails unit tests, although incorporating problem descriptions and tests allows stronger tailoring to user intent.
6 Conclusion
CodeRL combines pretrained language models with reinforcement learning and unit-test feedback during training and inference. Integrated with CodeT5-large, it achieves new state-of-the-art results on APPS and MBPP while outperforming much larger pretrained models.
- CodeRL exploits unit-test signals during both training and inference through an actor-critic reinforcement-learning framework.
- Its critic provides dense feedback on synthetic code samples, while critical sampling regenerates programs using unit tests and critic scores.
- The framework consistently improves conventional pretrained language models and can incorporate stronger backbones or finer-grained feedback such as static analysis.
A Critic Sampling Procedure
The section points readers to Algorithm 1 for a step-by-step account of CodeRL’s critic sampling procedure.
- Algorithm 1 provides the step-by-step explanation of the critic sampling procedure.
B.1 CodeXGLUE Benchmark Results
The CodeXGLUE evaluation covers code-to-text, text-to-code, and code-to-code generation using benchmark-specific generation metrics, while Algorithm 1 specifies CodeRL’s critic-sampling workflow.
- B.1 CodeXGLUE Benchmark Results: CodeT5-large is evaluated on CodeXGLUE code-to-text, text-to-code, and code-to-code generation tasks.
- B.1 CodeXGLUE Benchmark Results: CodeXGLUE uses similarity-based metrics including BLEU and CodeBLEU, unlike the pass-based APPS and MBPP evaluations.
- A Critic Sampling Procedure: Algorithm 1 takes a problem, language model, critic, and repair model as input and outputs N generated solution programs.
- A Critic Sampling Procedure: The procedure extracts example unit tests, generates programs, repairs failed programs, and filters candidates using test results.
- A Critic Sampling Procedure: Among failed programs, critic scores rank candidates, while subsequences from passed programs seed regeneration.
- B.1 CodeXGLUE Benchmark Results: The benchmark tables report CodeXGLUE results for code-to-text, text-to-code, and code-to-code generation.
B.2 MBPP Benchmark Results
On MBPP, CodeT5 variants benefit from larger models, more pretraining data, and improved objectives, while CodeRL strengthens zero-shot transfer from APPS. With 1,000 samples, CodeT5+CodeRL reaches a new SOTA result.
- Higher temperature sampling benefits CodeT5, with 1.2 producing less-greedy, more-diverse decoding than GPT’s temperature of 0.5.
- 63.0% pass@80 is achieved by CodeT5+CodeRL on MBPP in the zero-shot setting, up from 60.2% for CodeT5 finetuned on APPS.
- CodeT5-770M reaches 46.8% pass@80, surpassing GPT-8B’s 40.6% with a smaller model.
- Table 10 evaluates pass@80 and pass@1000 across CodeT5 variants differing in model size, pretraining data, and objectives.
- 81.8% pass@1000 is the new SOTA result achieved by CodeT5+CodeRL when the sampling budget increases to 1,000.
C.1 Failure Analysis
Failure analysis shows that compilation failures are dominated by syntax mistakes, while runtime failures commonly involve incorrect input processing, values, indices, or data types.
- More than 90% of compiling errors arise from syntactical mistakes, despite models generally producing correct indentation.
- Only 4% of problems involve wrong tab tokens and 5% involve wrong indentation levels among the analyzed compilation failures.
- Runtime errors most often involve wrong data indices, inappropriate values, or mismatched data types.
- Many runtime failures occur while preprocessing test inputs, indicating a potential improvement target for constructing proper input variables.
C.2 Example Generated Programs
The examples illustrate CodeRL+CodeT5 succeeding directly, refining programs after example-test feedback, repairing and refining failed programs, and still encountering timeout failures.
- CodeRL+CodeT5 can generate correct programs without critical sampling, as illustrated by the successful example in Figure 11.
- Figure 10 distinguishes compiling errors from runtime errors, with runtime categories defined in Table 11.
- Critical sampling can condition on prior programs to refine code that passes example tests but fails hidden tests, including by reordering elif blocks.
- When generated programs fail example tests, critical sampling can first repair and then refine them until they pass hidden tests.
- A demonstrated failure case remains incorrect after critical sampling because execution still times out.