Source-linked AI summary
CodeT: Code Generation with Generated Tests
Bei Chen, Fengji Zhang, Anh Nguyen, Daoguang Zan, Zeqi Lin, Jian-Guang Lou, Weizhu Chen
TL;DR
Selecting one correct program from multiple language-model-generated candidates remains difficult, while manually creating test cases is costly. CodeT automatically generates test cases and selects solutions through dual execution agreement, significantly improving pass@1 across benchmarks, including 65.8% on HumanEval for code-davinci-002.
Problem
Selecting a single correct solution from multiple language-model-generated candidates remains difficult, limiting the practical usefulness of code generation models.
Method
CodeT uses the same pre-trained language model to generate test cases, then selects code through agreement with those tests and other candidate solutions.
Results
CODET significantly improves pass@1 consistently across four benchmarks and models; for code-davinci-002 on HumanEval, it reaches 65.8%, an 18.8% absolute gain over baseline.
Takeaways & Limitations
The experiments support dual agreement as an effective basis for selecting correct code solutions from multiple generated candidates.
Takeaways & Limitations
CODET can rank incorrect solutions highest when descriptions are ambiguous, corner cases are uncovered, or imports are missing.
Abstract
from arXiv · showhide
The task of generating code solutions for a given programming problem can benefit from the use of pre-trained language models such as Codex, which can produce multiple diverse samples. However, a major challenge for this task is to select the most appropriate solution from the multiple samples generated by the pre-trained language models. A natural way to evaluate the quality and correctness of a code solution is to run it against a set of test cases, but the manual creation of such test cases is often costly and time-consuming. In this paper, we propose a novel method, CodeT, that leverages the same pre-trained language models to automatically generate test cases for the code samples, thus reducing the human effort and increasing the coverage of the test scenarios. CodeT then executes the code samples using the generated test cases, and performs a dual execution agreement, which considers both the consistency of the outputs against the generated test cases and the agreement of the outputs with other code samples. We conduct comprehensive experiments on four benchmarks, HumanEval, MBPP, APPS and CodeContests, using five different pre-trained language models with varying sizes and capabilities. Our results show that CodeT can significantly improve the performance of code solution selection over previous methods, achieving remarkable and consistent gains across different models and benchmarks. For instance, CodeT improves the pass@1 metric on HumanEval to 65.8%, which represents an absolute improvement of 18.8% over the code-davinci-002 model, and an absolute improvement of more than 20% over the previous state-of-the-art results.
1 INTRODUCTION
CodeT addresses the difficult problem of selecting one correct solution from multiple code samples by generating test cases with the same language model and selecting solutions through dual execution agreement. The method avoids labelled data and additional rankers while being evaluated across multiple models and benchmarks.
- Motivation: Codex reaches 77.4% pass@100, underscoring the challenge of selecting a single correct solution from multiple generated candidates.The passage defines pass@100 as passing when at least one of 100 generated solutions succeeds.
- Motivation: Execution-guided verification depends heavily on the quality and quantity of available test cases.Checking whether a solution passes corresponding test cases is widely used across code generation, translation, and program synthesis.
- Method: CodeT uses the same pre-trained language model to generate many test cases for each programming problem from an elaborate prompt.The method then applies a dual execution agreement approach inspired by RANSAC.
- Evaluation: CodeT is designed to be simple and efficient without labelled data or additional rankers.The method was evaluated with five pre-trained code-generation models, including Codex, INCODER, and CODEGEN, across HumanEval, MBPP, and APPS.
2 METHODOLOGY
CODET generates test cases with the same pre-trained language model used for code solutions, then selects the best solution through dual execution agreement over solution–test-case pairs and consensus sets.
- Code and test generation: CODET samples code solutions X from context c, which contains a natural-language programming problem and code snippet.The selected solution ˆx is intended to correctly solve the programming problem.
- Code and test generation: The model generates test cases Y by completing an added instruction with plausible input–expected-output pairs for the function.The same pre-trained language model M generates both code solutions and test cases; example cases are removed from context before generation.
- Dual execution agreement: A solution passes a test case when it executes without errors and matches the expected output, making the pair a hypothetical inlier.Pairs that fail are treated as outliers because they do not describe the correct functionality.
- Dual execution agreement: For an inlier pair, CODET forms a consensus set from solutions passing exactly the same test cases and those test cases, scoring it as f(S) = |Sx||Sy|.The score equals the number of agreeing solution–test-case pairs and reflects consensus around a hypothetical functionality.
- Dual execution agreement: CODET repeats consensus-set construction and selects a solution from the highest-scoring set; when feasible, it evaluates all solution–test-case pairs directly.The direct version groups solutions that pass the same test cases and scores each resulting consensus set.
3 EXPERIMENTAL SETUP
The experiments evaluate code-solution selection across four public benchmarks in the zero-shot setting using Codex, INCODER, and CODEGEN models. Performance is measured with unbiased pass@k, using ground-truth tests to assess functional correctness and generated samples for selection.
- Models: Experiments use Codex, INCODER, and CODEGEN, including code-cushman-001, code-davinci-001, and code-davinci-002.The three Codex models provide different capabilities.
- Evaluation Settings: For CODET, temperature is set to 0.8 and the sampling number is set to 100.These settings are reported for the HumanEval and MBPP pass@k results.
- Metrics and Baseline: Pass@k measures the percentage of problems solved when any of k sampled code solutions passes all ground-truth test cases.For each problem, n code solutions are sampled and k are selected for evaluation.
- Metrics and Baseline: The unbiased definition of pass@k is used as the baseline for performance evaluation.Ground-truth test cases determine the functional correctness of code solutions.
- Benchmarks: Four public code-generation benchmarks are evaluated in the zero-shot setting, including HumanEval and MBPP.HumanEval contains handwritten Python programming problems, with original example input-output cases removed to avoid exposing real test cases.
4 EXPERIMENTAL RESULTS
CODET improves code-solution selection across five pretrained models and four benchmarks through test-driven dual execution agreement. Its gains persist across model capabilities, while test-case quality and model limitations affect performance.
- Benchmark results: 18.8% absolute improvement boosts code-davinci-002 pass@1 to 65.8% on HumanEval.This also exceeds previously reported results by more than 20% in absolute pass@1 improvement.
- Benchmark results: 4.2%–13.1% absolute pass@1 gains appear for INCODER-6B and CODEGEN-MONO-16B, with INCODER-6B gaining 13.1% on MBPP.For three Codex models, CODET pass@1 improvements are about 10% and consistently above 10% on HumanEval.
- Benchmark results: CODET consistently outperforms AlphaCode-C because its dual execution agreement incorporates test-case information.The experiments also report duplicated generated solutions and test cases, while deduplication has little influence on CODET results.
- Test-case analysis: Code-davinci-002 generates the highest-quality test cases, which improve other models’ pass@1 by 1.8%–4.3% for mediocre models and 6.2%–15.9% for INCODER and CODEGEN.Test-case correctness is evaluated by whether canonical solutions pass the generated cases.
- Case studies and limitations: 53 of 164 HumanEval problems with code-cushman-001 contain correct solutions that are absent from CODET’s top consensus set.Among these cases, 20% are attributed to ambiguous descriptions, uncovered corner cases, or missing imports; the remainder reflect failures to understand the problem description.
5 RELATED WORK
Related work spans large-model code generation, automated test-case generation, and selecting correct solutions from multiple generated samples. Prior methods include competition-scale models, automated testing systems for statically typed languages, and learned ranking or verification approaches.
- Code Generation with Large Models: Large pre-trained language models leverage billions of parameters and massive public source-code corpora for code generation.AlphaCode reportedly outperformed half of human competitors in real-world programming competitions, while Codex powers Copilot’s real-time coding assistance.
- Automatic Test Case Generation: Automated test-case generation reduces developers’ manual effort, with early systems targeting statically typed languages such as Java.Examples include Randoop, EvoSuite, MOSA, DynaMOSA, and MIO; Pynguin was later proposed for Python.
- Code Selection from Multiple Samples: Because large models require many samples to find correct code, prior work selects candidates using trained verifiers or jointly trained generators and rankers.These approaches include choosing the highest-ranked solution with a trained verifier and using a multi-task framework to train generation and ranking together.
6 CONCLUSION AND FUTURE WORK
CODET generates code solutions and test cases with pre-trained language models, then selects solutions using dual execution agreement. Experiments show significant pass@1 improvements, while future work targets executable-code limitations, test-generation cost, and more difficult programming problems.
- Conclusion: CODET uses pre-trained language models to generate code solutions and test cases, executes solutions against those tests, and selects the best solution via dual execution agreement.The agreement considers both generated-test outputs and consistency with other solutions.
- Conclusion: Experimental results demonstrate CODET’s superiority through significant pass@1 improvements across various benchmarks.
- Limitations: CODET currently works only for executable code generation and introduces additional computation cost for test-case generation.
- Future Work: Future work will address these challenges and improve CODET for more difficult programming problems.
A MORE IMPLEMENTATION DETAILS
The implementation uses fixed sampling, generation, execution-timeout, and benchmark-specific test-case settings. CODET scores consensus sets with the square root of |Sx| to reduce the impact of code solutions.
- A MORE IMPLEMENTATION DETAILS: Sampling uses temperature 0.8, top p 0.95, and a maximum generation length of 300.Baseline pass@1 instead uses greedy search with temperature 0.
- A MORE IMPLEMENTATION DETAILS: The timeout for executing each test case is 0.1 seconds.
- A MORE IMPLEMENTATION DETAILS: Test-case sampling is 100 per problem for HumanEval and MBPP, and 50 per problem for APPS and CodeContests.CODET scores consensus sets using the square root of |Sx| to reduce the impact caused by code solutions.
B RESULTS ON ORIGINAL HUMANEVAL
On original HumanEval contexts, baseline pass@10 and pass@100 outperform results on the modified benchmark, motivating an additional experiment on the effect of retaining original contexts.
- Results on Original HumanEval: Baseline pass@10 and pass@100 outperform the corresponding results on modified HumanEval when evaluated with original contexts.The experiment examines the influence of removing example input-output cases from the original contexts.
C ANALYSIS ON CODE SOLUTIONS … F REMOVING TRIVIAL CODE SOLUTIONS
The analysis examines consensus-set scoring, de-duplication, sampling temperature, and trivial-solution filtering in CODET. It finds that preserving generated samples, using diverse sampling, and filtering trivial solutions affect evaluation in distinct ways, while CODET remains robust to trivial outputs.
- C ANALYSIS ON CODE SOLUTIONS: CODET groups code solutions that pass exactly the same test cases into consensus sets, whose number can increase with diverse high-temperature sampling.The main experiments use top-p sampling with temperature 0.8, allowing substantial functional variation among generated solutions.
- C ANALYSIS ON CODE SOLUTIONS: CODET weights the square root of a consensus set’s solution count because passing more test cases is considered more important than having more solutions.The paper illustrates this with one solution passing five tests versus five solutions passing one test.
- D INFLUENCE OF DE-DUPLICATION: De-duplication has slight and inconsistent effects on CODET performance across benchmarks.HumanEval favors code-solution de-duplication for pass@1, whereas MBPP achieves its best pass@1 without de-duplication.
- D INFLUENCE OF DE-DUPLICATION: The main experiments retain all generated code solutions and test cases rather than de-duplicating them.The authors leave more advanced de-duplication methods for future work.
- E SENSITIVITY TO THE TEMPERATURE: Temperature strongly affects generated code and test-case quality, while CODET uses temperature 0.8 to benefit from more diverse samples.Sensitivity is evaluated by comparing baseline pass@100 and CODET pass@1 across temperatures for code-cushman-001 on HumanEval.
- F REMOVING TRIVIAL CODE SOLUTIONS: Trivial solutions that return an input argument or constant value can pass generated tests and hinder CODET’s ranking on difficult APPS and CodeContests problems.These solutions are identified among incorrect code-davinci-002 outputs.
- F REMOVING TRIVIAL CODE SOLUTIONS: Removing a prominent percentage of trivial solutions yields little performance gain, demonstrating CODET’s robustness.The comparison is reported in Table 8 for zero-shot APPS and CodeContests results using code-davinci-002 and CODET.
G RESULTS ON APPS AND CODECONTESTS IN THE ONE-SHOT SETTING … H.2 CODE COVERAGE OF TEST CASES
In the one-shot APPS and CodeContests setting, example-based filtering improves baseline performance, while CODET filtering adds gains on APPS. The paper also analyzes generated test-case validity, quantity, and code coverage on HumanEval.
- G RESULTS ON APPS AND CODECONTESTS IN THE ONE-SHOT SETTING: One-shot APPS and CodeContests append a single input-output example as a formatting hint, then filter solutions that fail the example.The paper calls this approach the “Baseline Filter” method.
- G RESULTS ON APPS AND CODECONTESTS IN THE ONE-SHOT SETTING: CODET Filter applies CODET to solutions remaining after example-based filtering.This separates CODET Filter from the Baseline Filter procedure.
- G RESULTS ON APPS AND CODECONTESTS IN THE ONE-SHOT SETTING: One-shot CODET performance is much better than the reported zero-shot performance, while example-based filtering significantly improves baseline results.These comparisons are summarized in Table 9 relative to Table 3.
- G RESULTS ON APPS AND CODECONTESTS IN THE ONE-SHOT SETTING: CODET Filter further outperforms Baseline Filter on APPS, especially for introductory and interview problems.The passage contrasts this APPS result with the separate CodeContests outcome.
- H.1 STATISTICS ON TEST CASES: For HumanEval, test-case generation samples each problem 100 times, potentially producing multiple test cases per sample.Generated samples are post-processed into individual syntactically correct test cases.
- H.1 STATISTICS ON TEST CASES: The paper reports extracted test-case counts for five models on HumanEval and evaluates CODET Pass@k under different test-case numbers.Table 10 covers extracted counts, while Table 12 varies sampling number and extraction limits.
- H.2 CODE COVERAGE OF TEST CASES: Test-case quality is assessed using statement coverage and branch coverage on canonical HumanEval solutions.Statement coverage measures executed statements, while branch coverage measures executed control-flow branches.
H.3 RESULTS OF REDUCING THE NUMBER OF TEST CASES
The paper evaluates how reducing the number of generated test cases affects CodeT’s dual execution agreement on HumanEval. It varies test-case-related settings while accounting for the fact that the maximum number of cases is not always reached.
- Ablation setup: CodeT’s fewer-test-case performance is studied through an ablation on HumanEval using code-cushman-001 and code-davinci-002.The ablation varies the number of test cases participating in dual execution agreement.
- Ablation setup: 100 test case samples are used for HumanEval in the main experiments.The number of test case samples is one of the two hyper-parameters related to the number of test cases.
- Test-case extraction: Limit is set to 5 across benchmarks, controlling how many syntactically correct test cases are extracted from each sample.A valid test case is an assertion statement beginning with “assert” and containing the corresponding entry point function’s name.
- Test-case extraction: Limit multiplied by Sampling Number gives the maximum, not exact, number of test cases because some samples contain fewer valid cases.Not every sample contains the Limit number of valid test cases.
I ABLATION STUDY ON THE SCORE OF CONSENSUS SET · J MORE EXAMPLES FOR CASE STUDY
The ablation compares consensus scoring based on code-solution count, test-case count, or their product. Case studies show that CODET can select correct consensus sets when solutions or tests are fewer, but can fail on uncovered corner cases or misunderstood problems.
- I ABLATION STUDY ON THE SCORE OF CONSENSUS SET: CODET scores a consensus set as f(S) = |Sx||Sy|, combining its code solutions and test cases.Sx denotes code solutions and Sy denotes test cases in the consensus set.
- I ABLATION STUDY ON THE SCORE OF CONSENSUS SET: The first ablation variant uses f′(S) = |Sx|, counting code solutions with the same functionality in a self-consistency-style score.This variant is described as aligning with the idea of self-consistency.
- I ABLATION STUDY ON THE SCORE OF CONSENSUS SET: The second ablation variant uses f′′(S) = |Sy|, counting the test cases that each code solution can pass.This contrasts with the product score by considering only the number of test cases.
- I ABLATION STUDY ON THE SCORE OF CONSENSUS SET: AlphaCode-C provides a related clustering baseline by grouping code solutions by functionality and scoring each cluster by size.AlphaCode-C requires additional test inputs to obtain outputs used to determine functional equivalence.
- J MORE EXAMPLES FOR CASE STUDY: In the remove-vowels case, CODET ranks the correct consensus set first despite containing fewer solutions, because it passes more test cases.The incorrect second-ranked set has 41 solutions and 147 test cases, while the correct top-ranked set has 31 solutions and 170 test cases.
- J MORE EXAMPLES FOR CASE STUDY: CODET can fail when partially correct solutions miss corner cases, as in the prime-length example where incorrect solutions fail on strings of length 1.The incorrect top-ranked set has 20 solutions and passes 205 test cases, whereas the correct third-ranked set has 222 test cases but a lower consensus score.