Source-linked AI summary
Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation
Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, Lingming Zhang
TL;DR
Existing code-generation benchmarks use limited tests that may not fully assess whether LLM-generated code is functionally correct. EvalPlus augments these benchmarks with LLM- and mutation-based test generation, and HumanEval+ reveals previously undetected errors while changing model comparisons.
Problem
Existing programming-benchmark test cases can be limited in quantity and quality for fully assessing the functional correctness of LLM-generated code.
Method
EvalPlus augments evaluation datasets with automatically generated tests using LLM-based seed generation, mutation-based diversification, and optional test-suite reduction.
Results
HumanEval+ expands HumanEval’s test cases by 80× and reduces pass@k by up to 19.3%-28.9% across evaluated LLMs.
Takeaways & Limitations
More rigorous testing can detect previously undetected wrong code and change relative model rankings, including making WizardCoder-CodeLlama and Phind-CodeLlama outperform ChatGPT on HumanEval+.
Takeaways & Limitations
ChatGPT is unsuitable for large amounts of automated test generation because querying it has undesirable speed and cost.
Abstract
from arXiv · showhide
Program synthesis has been long studied with recent approaches focused on directly using the power of Large Language Models (LLMs) to generate code. Programming benchmarks, with curated synthesis problems and test-cases, are used to measure the performance of various LLMs on code synthesis. However, these test-cases can be limited in both quantity and quality for fully assessing the functional correctness of the generated code. Such limitation in the existing benchmarks begs the following question: In the era of LLMs, is the code generated really correct? To answer this, we propose EvalPlus -- a code synthesis evaluation framework to rigorously benchmark the functional correctness of LLM-synthesized code. EvalPlus augments a given evaluation dataset with large amounts of test-cases newly produced by an automatic test input generator, powered by both LLM- and mutation-based strategies. While EvalPlus is general, we extend the test-cases of the popular HumanEval benchmark by 80x to build HumanEval+. Our extensive evaluation across 26 popular LLMs (e.g., GPT-4 and ChatGPT) demonstrates that HumanEval+ is able to catch significant amounts of previously undetected wrong code synthesized by LLMs, reducing the pass@k by up-to 19.3-28.9%. We also surprisingly found that test insufficiency can lead to mis-ranking. For example, both WizardCoder-CodeLlama and Phind-CodeLlama now outperform ChatGPT on HumanEval+, while none of them could on HumanEval. Our work not only indicates that prior popular code synthesis evaluation results do not accurately reflect the true performance of LLMs for code synthesis, but also opens up a new direction to improve such programming benchmarks through automated testing. We have open-sourced our tools, enhanced datasets as well as all LLM-generated code at https://github.com/evalplus/evalplus to facilitate and accelerate future LLM-for-code research.
1 Introduction
LLM code-generation benchmarks rely on limited manually constructed tests, raising doubts about whether reported functional correctness reflects true correctness. EvalPlus addresses this gap by automatically expanding benchmark tests with LLM- and mutation-based generation.
- Motivation: Existing programming benchmarks often use fewer than 10 manually constructed tests per problem, which may miss functional behaviors and corner cases.The paper illustrates this issue with incorrect ChatGPT code that passes HumanEval’s base tests.
- Motivation: These test limitations occur across popular code-generation benchmarks and motivate the question of whether LLM-generated code is really correct.
- Proposal: EvalPlus augments existing code benchmarks with automatically generated inputs to evaluate the functional correctness of LLM-synthesized code more rigorously.Its generator combines LLM-based seed creation with type-aware mutation and can reduce the resulting suite through greedy set covering.
- Contribution: 80× more test cases are used to extend HumanEval into HumanEval+, while HumanEval+-Mini reduces the expanded suite by 47× with similar testing effectiveness.
- Contribution: The study identifies test inadequacy as a source of substantially over-approximated functional-correctness results in current programming benchmarks.
2 Approach
EvalPlus builds rigorous evaluation suites by combining ChatGPT-generated seed inputs, type-aware mutation, contract filtering, differential testing, and optional test-suite reduction. Its reduction strategies preserve selected testing requirements while lowering evaluation cost.
- Evaluation and Contracts: Generated tests evaluate LLM code through differential testing against the ground-truth implementation, while programming contracts filter invalid inputs.
- Automated Test Input Generation: EvalPlus prompts ChatGPT with ground-truth code, demonstration inputs, and instructions to generate high-quality seed tests targeting difficult cases.
- Automated Test Input Generation: Type-aware mutation expands valid seed inputs by applying structure-preserving changes informed by input data types.Mutated inputs that comply with program contracts are retained in the seed pool for continued generation.
- Test-Suite Reduction: Test-suite reduction selects smaller suites using branch coverage, mutant killings, LLM sample killings, or a combination of these requirements.Mutation testing measures whether tests detect artificially seeded bugs, while sample killings record wrong LLM outputs detected by each test.
3 Evaluation
EvalPlus evaluates 26 LLMs on HUMANEVAL and HUMANEVAL+, using unbiased pass@k under multiple sampling settings and a reduced-test analysis. HUMANEVAL+ consistently exposes more incorrect code, changes some model rankings, and reveals defects in HUMANEVAL ground truths.
- Evaluation setup: 26 LLMs were evaluated with unbiased pass@k, random sampling, greedy decoding, and temperature settings.Each model generated program samples for each task; greedy decoding reports pass@1⋆.
- Evaluation setup: HUMANEVAL contains 164 Python tasks judged by a handful of manual tests, while HUMANEVAL+ adds 80× unique test-cases and fixes incorrect ground truths.Four HUMANEVAL tasks contain over 100 cross-checked tests; otherwise the maximum and average counts are 26 and 7.3.
- LLM evaluation results: Up to 23.1% for pass@1⋆, 19.3% for pass@1, 24.9% for pass@10, and 28.9% for pass@100 reductions occurred when evaluating LLMs on HUMANEVAL+.Drops also occurred for CodeGen-16B, CodeLlama-34B, StarCoder, ChatGPT, and GPT-4.
- LLM evaluation results: HUMANEVAL+ changes relative model rankings: WizardCoder-CodeLlama and Phind-CodeLlama outperform ChatGPT despite not doing so on HUMANEVAL.Other reversals include SantaCoder-1B surpassing INCODER-6.7B and VICUNA-7B outperforming INCODER-1.3B.
- Test-suite reduction: HUMANEVAL+ can be reduced from 764.1 average tests per task to 16.1 while retaining almost the same pass@1⋆ drop as the full suite.The reduced suite is 47× smaller; empirical LLM sample killings provide the strongest effectiveness, while coverage and mutation analysis provide theoretical guarantees.
- LLM evaluation results: HUMANEVAL+ detects misidentified solutions across programming-task difficulty levels, while tasks differ in both difficulty and the difficulty of generating counter-examples.Multiple conditions, completeness, reasoning, and efficiency requirements are identified as especially hard task characteristics.
- Benchmark validity: 18 HUMANEVAL ground truths contain defects: five unhandled edge-cases, 10 bad-logic implementations, and three performance issues.The defects were detected by differential testing between a re-implemented ground truth and the original.
4 Related Work
This section situates LLM code synthesis within program-synthesis research, code benchmarks, and automated test generation, highlighting limitations of existing testing approaches.
- LLMs have recently been applied to direct code generation, joining established deductive, inductive, and neural-guided program-synthesis techniques.
- HUMANEVAL evaluates code synthesis through Python function signatures, docstrings, reference implementations, and correctness test cases.
- Automated test generation includes black-box, white-box, and grey-box methods that differ in their access to source code and coverage feedback.
- Traditional automated testing methods may not generate semantically meaningful inputs for arbitrary dynamically typed programs.
- EvalPlus addresses this gap by using ChatGPT to initialize interesting seeds from ground-truth code and type-aware mutation to scale test inputs.
5 Conclusion & Future Work
EvalPlus is a rigorous program-synthesis evaluation framework that augments HUMANEVAL with automatically generated tests and supports smaller test suites with similar effectiveness.
- EvalPlus combines LLM- and mutation-based input generation to obtain diverse tests for evaluating the correctness of LLM-generated code.
- HUMANEVAL+ extends HUMANEVAL with additional high-quality, automatically generated test inputs.
- HUMANEVAL+-MINI is 47× smaller than HUMANEVAL+ while preserving similar test effectiveness through test-suite reduction.
- EvalPlus evaluation shows that HUMANEVAL+ identifies previously undetected wrong code generated by LLMs.
- Future work includes applying EvalPlus to MBPP and integrating it with formal verification and validation techniques.
A Detailed Experimental Setup
The experiments evaluate 26 LLMs under controlled sampling and decoding settings, with model-specific prompting and per-test-case runtime-based correctness oracles.
- 26 LLMs, including code-specialized and general-purpose models, are evaluated to represent recent and widely used systems.
- Each model randomly samples 200 programs across temperatures {0.2,0.4,0.6,0.8} and greedy decoding with zero temperature.
- Generated outputs are capped at 512 new tokens and truncated using end-of-string identifiers associated with HUMANEVAL and particular models.
- Prompts follow official model examples, with instruction-based prompts used for ChatGPT, GPT-4, and WizardCoder-CodeLlama.
- A solution is correct only when all test outputs match expected values within runtime limits, using exact matching by default.
- Each test case receives a timeout of max(200ms,4×tgt), where tgt is the corresponding ground-truth execution time.