PaperlayerView exact revision on arXiv

Source-linked AI summary

CoderEval: A Benchmark of Pragmatic Code Generation with Generative Pre-trained Models

Hao Yu, Bo Shen, Dezhi Ran, Jiaxin Zhang, Qi Zhang, Yuchi Ma, Guangtai Liang, Ying Li, Qianxiang Wang, Tao Xie

arXiv:2302.00288v3cs.SE

TL;DR

Existing benchmarks largely focus on standalone functions, although non-standalone functions account for more than 70% of functions in popular open-source projects. CoderEval is a curated benchmark for contextual code generation, and the evaluated models perform substantially better on standalone than non-standalone functions.

  • Problem

    Most existing benchmarks focus on standalone functions, even though non-standalone functions constitute more than 70% of functions in popular open-source projects.

    Source: §1 · p. 2

  • Method

    CoderEval curates real-world Python and Java tasks across six levels of contextual dependency and provides an execution platform to assess generated code.

    Source: §1 · p. 2

  • Results

    The three evaluated models perform substantially better on standalone than non-standalone functions in both CoderEval for Python and CoderEval for Java.

    Source: §1 · p. 2, §5.1.3 · p. 8

  • Takeaways & Limitations

    The authors identify improving models' ability to consider and use contextual information as important for their practical value.

    Source: §1 · p. 2

    The experiments cover three industry models and omit other recent models, including CodeGeex, StarCoder, and WizardCoder, due to limited computing resources.

    Source: §7 · p. 11

Abstract

from arXiv · show

Code generation models based on the pre-training and fine-tuning paradigm have been increasingly attempted by both academia and industry, resulting in well-known industrial models such as Codex, CodeGen, and PanGu-Coder. To evaluate the effectiveness of these models, multiple existing benchmarks are proposed, including only cases of generating a standalone function, i.e., a function that may invoke or access only built-in functions and standard libraries. However, non-standalone functions, which typically are not included in the existing benchmarks, constitute more than 70% of the functions in popular open-source projects, and evaluating models' effectiveness on standalone functions cannot reflect these models' effectiveness on pragmatic code generation scenarios. To help bridge the preceding gap, in this paper, we propose a benchmark named CoderEval, consisting of 230 Python and 230 Java code generation tasks carefully curated from popular real-world open-source projects and a self-contained execution platform to automatically assess the functional correctness of generated code. CoderEval supports code generation tasks from six levels of context dependency, where context refers to code elements such as types, APIs, variables, and consts defined outside the function under generation but within the dependent third-party libraries, current class, file, or project. CoderEval can be used to evaluate the effectiveness of models in generating code beyond only standalone functions. By evaluating three code generation models on CoderEval, we find that the effectiveness of these models in generating standalone functions is substantially higher than that in generating non-standalone functions. Our analysis highlights the current progress and pinpoints future directions to further improve a model's effectiveness by leveraging contextual information for pragmatic code generation.

1 INTRODUCTION

Existing benchmarks largely test standalone functions, although non-standalone functions make up more than 70% of functions in popular open-source projects. CoderEval evaluates pragmatic code generation with curated project functions and contextual information, and the evaluated models perform substantially better on standalone than non-standalone functions.

1 INTRODUCTION

  • More than 70% of functions in popular open-source projects are non-standalone, while most existing benchmarks focus on standalone functions.DS-1000 is an exception, but its non-standalone cases use APIs from only seven specified data-science libraries.Source: §1 · p. 2
  • CoderEval divides tasks into six levels of contextual dependency, from self-contained functions to project-level functions.The benchmark targets dependencies defined outside the function, including types, APIs, variables, and constants.Source: §1 · p. 2
  • The benchmark contains 230 Python and 230 Java problems curated from real-world open-source projects, with human-labeled docstrings and added tests to improve coverage.The authors also provide all accessible context and the context actually used by each function for fine-grained evaluation.Source: §1 · p. 2
  • The three evaluated models perform substantially better on standalone than non-standalone functions in both CoderEval for Python and CoderEval for Java.The authors identify correct use of oracle_context as important for pragmatic code generation.Source: §1 · p. 2
  • The evaluation compares CodeGen, PanGu-Coder, and ChatGPT on CoderEval and HumanEval, examining contextual use and the effects of original versus human-labeled docstrings.The authors report that docstring choice affects code-generation effectiveness.Source: §1 · pp. 2–3

2 BACKGROUND

The background reviews code-generation models and benchmarks, emphasizing that existing benchmark coverage of project-level functions is limited and difficult to construct. It positions CoderEval against prior benchmarks and reports that its functions have more lines of code on average than HumanEval functions, with similar cyclomatic complexity.

2.1 Large Language Models for Code Generation

  • Code-generation models discussed include PanGu-Coder, CodeGen, Codex, AlphaCode, and InCoder, which use different training and generation approaches.For example, InCoder supports both program synthesis and editing using bidirectional context.Source: §2.1 · p. 3

2.2 Benchmarks for Code Generation

  • CoderEval is described as supporting project-level code generation with Pass@K to validate generated code's functional correctness.The authors characterize it as the only benchmark with both features at the time of writing.Source: §2.2 · p. 3
  • Most existing benchmarks focus on standalone functions; DS-1000 covers non-standalone functions only when they invoke APIs from seven specified data-science libraries.Project-level benchmarks are difficult to build and sandbox, select targeted tests for, and supplement with tests for uncovered functions.Source: §2.2 · p. 3
  • Concode is a dataset of over 100,000 functions from approximately 33,000 Java repositories, rather than a benchmark.CoderEval differs by executing generated code, providing human-labeled docstrings, selecting functions carefully, supporting Python and Java, and including newer code.Source: §2.2 · p. 3

2.3 Statistical Comparison with Different Benchmarks

  • CoderEval functions have cyclomatic complexity similar to HumanEval functions, but more lines of code on average.Table 1 compares CoderEval, HumanEval, and Concode.Source: §2.3 · p. 3, Table 1 · p. 4

3 CODEREVAL BENCHMARK

CoderEval's construction process comprises dataset collection, dataset inspection, and evaluation.

3 CODEREVAL BENCHMARK

  • CoderEval's construction process has three phases: dataset collection, dataset inspection, and evaluation.The overview figure presents the construction process.Source: §3 · p. 4

3.1 Dataset Collection

CoderEval selects functions from real-world open-source projects, screens them for pragmatic relevance, and constructs function-specific tests. Its collection comprises 230 Java problems from 10 projects and 230 Python problems from 43 projects.

3.1.1 Problem Selection.

  • Candidate projects are selected from GitHub's 14 most frequent tags, prioritizing projects with high star counts.Functions are filtered for English comments, successful execution, passing original tests, and suitability after manual screening.Source: §3.1.1 · p. 4
  • The collection contains 230 problems from 10 Java projects and 230 from 43 Python projects.The project-selection process aims to compile fewer projects while retaining the same total number of selected functions.Source: §3.1.1 · p. 4
  • The selection rules favor functions judged useful in real development, with informative docstrings and more than three implementation lines, while excluding tests and deprecated functions.Functions with fewer than ten contextual tokens are selected to avoid dependencies considered too difficult for the benchmark.Source: §3.1.1 · p. 4
  • When existing tests provide less than 100% branch coverage for a selected function, the first author adds tests aiming to reach full coverage when possible.Coverage is measured using the projects' original unit tests.Source: §3.1.2 · p. 4

3.1.2 Test Construction.

  • CoderEval selects tests that can reach each target function using a static call graph, then converts them into NewTestFile wrappers with a main function.This avoids running every project test and standardizes how function-specific tests are invoked.Source: §3.1.2 · p. 4

3.2 Dataset Inspection

CoderEval’s dataset inspection examines human-labeled prompts, identifies code dependencies outside target functions, and classifies tasks by runnable context. Most Python tasks require file-level context, unlike the standalone-function focus of HumanEval.

3.2.1 Human-labeled Docstring.

  • Thirteen experienced engineers, each with at least three years of Python or Java experience, supplied human-labeled descriptions for CoderEval problems.The paper uses these prompts to study prompt effects, mitigate possible pretraining memory of original descriptions, and provide high-quality docstrings; original descriptions vary in role and content.Source: §3.2.1 · pp. 4–5

3.2.2 Contextual Dependency Identification.

  • Program analysis identifies external dependencies by building a language-and-library knowledge base, parsing definitions in the source file, and classifying dependent references as types, variables, or API invocations.The knowledge base covers Python 3.0–3.10, Java 1.8–17, and publicly available PyPI and Maven Central libraries.Source: §3.2.2 · p. 5
  • Oracle_context records identified dependencies for assessing generated code or prompting models, whereas all_context collects types, variables, and APIs defined or imported in the function’s file.The experiments did not use all_context in inference; the authors identify its use in prompts as future work.Source: §3.2.2 · p. 5
  • The six runnable levels specify which scope’s dependencies a function requires, from self-contained through project-runnable, while excluding dependencies defined only at subsequent levels.A plib-runnable function can run outside its class, source file, and project when its required Python version and libraries are available and imported.Source: §3.2.3 · p. 5, §3.2.1 · p. 5

3.2.3 Runnable Level Classification.

  • Python tasks are 84% file-runnable and 54% class-runnable; slib-runnable tasks are 17% in CoderEval Python, 35% Java, versus 100% HumanEval.The authors interpret the high file-runnable share as making belonging-file oracle_context necessary and critical for generating most Python tasks correctly.Source: §3.2.3 · pp. 5–6, Figure 2 · p. 6

3.3 Evaluation Process

CoderEval’s evaluation platform runs generated functions in project environments and uses compilation and test execution to determine whether the corresponding tests pass.

3.3 Evaluation Process

  • The platform uses a Linux Docker image as a virtual sandbox for automatically executing and evaluating generated functions.The sandbox is intended to provide a ready runtime environment and enable easy and safe distribution.Source: §3.3 · p. 6
  • For Java, the platform incrementally compiles changed files, then runs NewTestFile and treats its return value as the test outcome.Compilation failure is counted as failure to pass all corresponding tests; the classpath includes the test file and dependent files or libraries.Source: §3.3 · p. 6

4 EXPERIMENTAL SETUP

The experiments compare three Python-and-Java code-generation models on CoderEval and HumanEval, examining correctness, contextual-token generation, and prompt effects. They use repeated runs, project-specific execution environments, and Pass@K and Acc@K metrics.

4 EXPERIMENTAL SETUP

  • Experiments ran each model 10 times; the standard deviation of Pass@K was about 1%, and reported results focus on the median experiment group by Pass@10.The authors state that experimental results and related artifacts are publicly available.Source: §4 · p. 6

4.1 Research Questions

  • The research questions ask how models perform on standalone versus non-standalone functions, whether they use oracle_context correctly, and how prompts affect effectiveness..Source: §4.1 · p. 6

3.3.1 Evaluation for Python.

  • Python evaluation uses a project-specific Python version and virtual environment, installs dependencies, checks project setup, then replaces the target function and runs its tests.Java projects are run with Java 17; the platform compiles the test file before replacing the target function.Source: §4.2.2 · p. 6, §3.3.2 · p. 6

4.2 Model Selection and Settings

  • The study compares CodeGen, PanGu-Coder, and ChatGPT because each supports Python and Java, with the selection spanning specialized and multilingual models.The authors excluded CodeGeeX because its 13B size exceeded their available computing resources at the time.Source: §4.2.1 · p. 6

4.2.2 Model settings.

  • CodeGen uses its 350M Mono model for Python and Multi model for Java; PanGu-Coder uses 300M models, and ChatGPT uses gpt-3.5-turbo.Inference uses a maximum window length of 1024, nucleus sampling, 10 samples per function, and temperature 0.8.Source: §4.2.1 · pp. 6–7

4.3 Evaluation Metric

  • Pass@K measures the percentage of problems with at least one correct solution among the top K samples, using 10 total samples and K values of 1, 5, and 10.The study uses an unbiased estimator to reduce high sampling variance and chooses 10 samples as a practical cost and response-time setting.Source: §4.3 · p. 7
  • Acc@K measures the percentage of target functions whose individual oracle_context tokens appear in at least one of the top K generated samples.Metrics requiring all oracle_context tokens, with or without their sequential order, are described as more challenging and reserved for future work.Source: §4.3 · p. 7

5 EXPERIMENTAL RESULTS

The experiments compare three models across CoderEval and HumanEval, examine standalone versus non-standalone generation, contextual-token accuracy, and prompt wording. Results show substantial difficulty with non-standalone functions and language-dependent effects.

5.1 RQ1: How Do CodeGen, PanGu-Coder, and ChatGPT Perform, Especially in Generating Standalone vs. Non-Standalone Functions?

  • ChatGPT consistently outperforms CodeGen and PanGu-Coder on HumanEval and on CoderEval for both Python and Java.The authors note that CoderEval's Python and Java problems do not overlap, so their results cannot be directly compared across those languages.Source: §5.1.1 · p. 7

5.1.3 Effectiveness comparison between the generation of standalone

  • Across Python and Java CoderEval, all three models perform substantially better on standalone than on non-standalone functions.The authors identify correctly generating oracle_context information as vital because non-standalone functions make up more than 70% of functions in open-source projects.Source: Table 4 · p. 8, §5.1.3 · p. 8

5.1.2 Complementarity and overlapping of three models.

  • Among the three models, 91 Python and 136 Java problems were solved by at least one model; 32 Python and 56 Java problems were solved by all three.Across CoderEval, 52 of the 88 overlapping problems were standalone, although standalone functions account for only about 30% of the benchmark.Source: §5.1.2 · p. 8, Figure 3 · p. 8

5.2 RQ2: How Do These Models Perform in Correctly Incorporating the Oracle_context Information in the Generated Code?

  • In Python, VarReference generation is weakest and TypeReference strongest; in Java, TypeReference is weakest and APIInvocation strongest.The relative ability to generate these token categories varies across languages.Source: §5.2 · p. 9
  • Acc@K for generating oracle_context tokens is consistent with Pass@K for generating functions that pass tests.The authors report that token-category performance differs between Python and Java.Source: §5.2 · p. 9

5.3 RQ3: How Do Different Prompts Impact the Effectiveness of These Models?

  • Original and human-labeled docstrings are more similar for Python than Java, and code-generation results using the two versions are likewise more similar for Python.On Java, CodeGen and ChatGPT perform better with original docstrings, while PanGu-Coder performs better with human-labeled docstrings.Source: §5.3 · p. 9, Table 5 · p. 9
  • On Java, CodeGen and ChatGPT perform better with original docstrings, while PanGu-Coder performs better with human-labeled docstrings.The authors relate these differences to docstring similarity and the proportion of target-language code in model training data; only PanGu-Coder was pretrained on Java code.Source: §5.3 · p. 9

6 RELATED WORK

Related benchmarks span standalone-function tests, multilingual variants, library-based tasks, and contextual Java code. The cited accounts distinguish them by task scope and how generated code is evaluated.

HumanEval

  • HumanEval contains 164 hand-written Python problems with function signatures, docstrings, reference solutions, and unit tests.It evaluates functional correctness of code generated from docstrings [7].Source: §6 · pp. 9–10

AiXBench

  • AiXBench provides 175 problems for automated Java evaluation and 161 for manual evaluation, with metrics for both forms of assessment.Its authors introduce an automatic correctness metric and manual quality criteria.Source: §6 · p. 10

MultiPL-E

  • MultiPL-E extends HumanEval and MBPP to 18 languages through translated benchmark material and an evaluation framework.Its portions cover varied programming paradigms, language features, and popularity levels.Source: §6 · p. 10

DS-1000

  • DS-1000 includes 1,000 Python problems across seven data-science libraries, but no tasks with dependencies on user-defined functions outside the target function.Its authors also manually modify functions to mitigate data leakage.Source: §6 · p. 10

Concode

  • Concode contains over 100,000 Java problems with contextual dependencies, but evaluates generated functions using BLEU rather than execution-based correctness.Its problems lack test cases, and its data is split by repository.Source: §6 · p. 11

7 THREATS TO VALIDITY

The authors identify two threats to validity: the representativeness of the evaluated models and possible deviation in statistics on standalone versus non-standalone functions.

Model representativeness

  • The experiments use three industry models, and omit newer models including CodeGeex, StarCoder, and WizardCoder because of limited computing resources.The authors say broader subject coverage could reduce this threat to representativeness.Source: §7 · p. 11

Statistics accuracy

  • The standalone versus non-standalone statistics may deviate because the authors cannot guarantee collecting every third-party library from PyPI and Maven.These libraries are used to identify dependencies on third-party libraries.Source: §7 · p. 11

8 CONCLUSION

The conclusion presents CoderEval as a benchmark for pragmatic code generation and reports that its experiments expose weaknesses in generating non-standalone functions.

Benchmark contribution

  • CoderEval evaluates model effectiveness in pragmatic code-generation scenarios using problems selected from varied open-source projects.The conclusion contrasts it with the widely used HumanEval benchmark.Source: §8 · p. 11

Experimental finding

  • Experiments on CoderEval reveal strengths and weaknesses of three LLMs.Source: §8 · p. 11
  • The results highlight limitations of existing models in generating non-standalone functions.Source: §8 · p. 11
Loading 2302.00288v3…