PaperlayerView exact revision on arXiv

Source-linked AI summary

DevEval: A Manually-Annotated Code Generation Benchmark Aligned with Real-World Code Repositories

Jia Li, Ge Li, Yunfei Zhao, Yongmin Li, Huanyu Liu, Hao Zhu, Lecheng Wang, Kaibo Liu, Zheng Fang, Lanshen Wang, Jiazheng Ding, Xuanming Zhang, Yuqi Zhu, Yihong Dong, Zhi Jin, Binhua Li, Fei Huang, Yongbin Li

arXiv:2405.19856v1cs.CLcs.SE

TL;DR

Existing benchmarks are poorly aligned with real-world code repositories, leaving the evaluation of LLM coding ability in those settings an open question. DevEval is a repository-aligned benchmark with comprehensive developer annotations, and evaluation shows lower performance than on previous benchmarks, including a 53.04% Pass@1 score for gpt-4-turbo-1106 on DevEval versus 80% on HumanEval.

  • Problem

    Existing code-generation benchmarks are poorly aligned with real-world repositories, and none satisfies all the features the authors identify for evaluating LLMs in real development settings.

    Source: §1 · pp. 1–2

  • Method

    DevEval provides repository-aligned samples with developer-written requirements, reference code and dependencies, and evaluates repository-level code generation using Pass@k and dependency Recall@k.

    Source: §1 · p. 2

  • Results

    gpt-4-turbo-1106 achieves 80% Pass@1 on HumanEval but 53.04% Pass@1 on DevEval, and the evaluated models perform lower on DevEval than on previous benchmarks.

    Source: §1 · p. 2

  • Takeaways & Limitations

    DevEval offers a more challenging and practical evaluation scenario for examining LLM strengths and weaknesses in real repositories.

    Source: §7 · p. 9

    DevEval is monolingual, with English requirements and Python code, and does not cover other natural or programming languages.

    Source: §9 · p. 10

Abstract

from arXiv · show

How to evaluate the coding abilities of Large Language Models (LLMs) remains an open question. We find that existing benchmarks are poorly aligned with real-world code repositories and are insufficient to evaluate the coding abilities of LLMs. To address the knowledge gap, we propose a new benchmark named DevEval, which has three advances. (1) DevEval aligns with real-world repositories in multiple dimensions, e.g., code distributions and dependency distributions. (2) DevEval is annotated by 13 developers and contains comprehensive annotations (e.g., requirements, original repositories, reference code, and reference dependencies). (3) DevEval comprises 1,874 testing samples from 117 repositories, covering 10 popular domains (e.g., Internet, Database). Based on DevEval, we propose repository-level code generation and evaluate 8 popular LLMs on DevEval (e.g., gpt-4, gpt-3.5, StarCoder 2, DeepSeek Coder, CodeLLaMa). Our experiments reveal these LLMs' coding abilities in real-world code repositories. For example, in our experiments, the highest Pass@1 of gpt-4-turbo is only 53.04%. We also analyze LLMs' failed cases and summarize their shortcomings. We hope DevEval can facilitate the development of LLMs in real code repositories. DevEval, prompts, and LLMs' predictions have been released.

1 Introduction

Existing code-generation benchmarks poorly reflect real-world repositories, and none meets all the features the authors identify for a repository-aligned benchmark. DevEval addresses this gap with annotated repository data, repository-level code generation, and evaluation of popular LLMs.

1 Introduction

  • Existing benchmarks are poorly aligned with real-world repositories, and none satisfies all four features the authors say a repository-oriented benchmark should meet.The features cover real repositories, realistic code and dependency distributions, comprehensive annotations, and robust evaluation metrics.Source: §1 · pp. 1–2
  • DevEval contains 1,874 test samples from 117 real-world repositories across 10 domains, with annotations by 13 developers including requirements, reference code, and dependencies.The benchmark's code and dependency distributions are reported as consistent with those of 500 real-world repositories.Source: §1 · p. 2
  • Repository-level code generation asks models to write code from requirements and a complete repository, simulating developers' coding process in a working repository.DevEval checks predictions with test cases using Pass@k and evaluates predicted dependencies with Recall@k.Source: §1 · p. 2
  • gpt-4-turbo-1106 scores 80% Pass@1 on HumanEval but only 53.04% Pass@1 on DevEval, illustrating lower performance in repository-level evaluation.The authors also analyze failed cases and summarize shortcomings of existing LLMs.Source: §1 · p. 2
  • The paper evaluates eight popular LLMs on DevEval to examine their strengths and shortcomings in repository-level code generation.The evaluated models include gpt-4, gpt-3.5, DeepSeek Coder, StarCoder 2, and CodeLLaMa.Source: §1 · p. 2

2 Benchmark - DevEval

DevEval combines repository context, implementation and dependency references, requirements, and tests in a repository-level code-generation benchmark. Its features are designed to reflect real repositories and support evaluation of both functional correctness and dependency use.

2.1 Overview

  • Each DevEval sample includes a function signature, requirement, repository contexts, reference code, reference dependencies, and test cases.The benchmark contains 1,874 samples from 117 real-world repositories.Source: §2.1 · p. 3

2.2 Task Definition

  • Models receive a function signature, requirement, and complete repository, then generate a function that is inserted into the repository and checked for correctness.Source: §2.2 · p. 3

2.3 Evaluation Metrics

  • Pass@k measures functional correctness by executing tests on generated programs and computing an unbiased score from n ≥ k samples and c correct programs.Source: §2.3 · p. 3
  • Recall@k measures how many reference dependencies appear among generated programs’ dependencies, which a parser extracts for comparison with the references.The paper considers intra-class, intra-file, and cross-file dependencies.Source: §2.3 · pp. 3–4

2.4 Features of DevEval

  • DevEval’s code, dependency, and repository-scale distributions align with 500 real repositories; its distribution across dependency types is also closer to them than previous benchmarks.Source: §2.4 · p. 4, Table 2 · p. 4, Table 3 · p. 4
  • Thirteen developers provide comprehensive annotations; requirements specify functionality and input-output parameters, while dependency paths disambiguate same-named functions.Requirements average 91.5 tokens, compared with 41.5 tokens for CoderEval.Source: §2.4 · p. 4
  • Unlike simple requirement-to-code tasks, repository-level generation simulates coding in a working repository and evaluates functional correctness and dependency use.Source: §2.4 · p. 4
  • DevEval offers a shared arena for comparing retrieval, long-context, and decision-making approaches, while allowing models to generate diverse programs.Source: §2.4 · pp. 4–5

3 Benchmark Construction

DevEval is constructed from real-world Python repositories through repository selection, function and test extraction, developer annotation, and distribution-informed sample selection.

Stage ❶: Repository Selection

  • The authors select 500 PyPI projects from the top 10 topics, requiring open-source licenses, non-fork and non-malicious status, and explicit unit tests.They download the latest released versions in November 2023.Source: §3 · p. 5

Stage ❷: Function Parse

  • Function parsing removes empty and initialization functions, extracts signatures and reference code, and treats other repository programs as context, yielding 590,365 candidates.Source: §3 · p. 5

Stage ❸: Tests Construction

  • Test construction retains functions with executable tests that pass on reference code and fail on incorrect programs, leaving 3,764 candidates.Source: §3 · p. 5

Stage ❹: Annotation

  • Thirteen annotators draft and double-check natural, functionally clear requirements, then review reference code and label dependencies; 2,846 functions remain.Trivial functions and those violating the ethical code are excluded.Source: §3 · p. 5

Stage ❺: Benchmark Construction

  • Benchmark construction selects for consistency with the 500-repository distribution while including as many functions as possible, producing 1,874 non-standalone and 706 standalone functions.Source: §3 · p. 5

4 Experiments

The experiments evaluate eight LLMs on repository-level code generation across three context settings, using Pass@k and Recall@k. Performance is low without context, improves with local-file context, and remains constrained by long, heterogeneous inputs.

4.1 Studied LLMs

  • The evaluation covers eight popular LLMs, including two closed-source and six open-source models, reproduced through official interfaces or implementations.Source: §4.1 · p. 5

4.4 Main Results

  • 17.40 Pass@1 for gpt-4 without context contrasts with 88.4 Pass@1 on HumanEval; local-file contexts improve gpt-4 Pass@1 by 205% and 173% across two settings.Table 5 reports Pass@k and Recall@k across the models and settings.Source: Table 5 · p. 6, §4.4 · pp. 6–7

4.2 Experimental Settings

  • Because repositories average 1.1 million tokens, the experiments compare generation without context with local-file completion and infilling using code around the target.Completion uses code above the target; infilling uses code both above and below it.Source: §4.2 · p. 6

4.3 Evaluation

  • The evaluation uses Pass@k and Recall@k for k=1, 3, 5, and 10; for k>1, it samples 20 programs per requirement with nucleus sampling.At k=1, the evaluation uses greedy search.Source: §4.3 · p. 6

4.4 Main Results

  • Successful cases reflect a synergy: local contexts supply repository knowledge, including 72% intra-class and intra-file dependencies, while detailed requirements help locate relevant code.The authors also attribute improvements to requirements that describe program purposes and repository background.Source: §4.4 · p. 7
  • Failure analysis finds models may miss valid functions in context; repositories are about nine times gpt-4-1106’s context window, and partial contexts can still match or exceed most models’ windows.The authors also identify heterogeneous snippets from multiple files as a challenge for models trained to predict from continuous contexts.Source: §4.4 · p. 7, Figure 4 · p. 7
  • Without context, gpt-family models have higher Pass@k and lower Recall@k, while other models show the reverse and sometimes reason or guess dependencies.The authors speculate that instruction-tuned gpt models generate conservatively, whereas other models may draw on learned code naming conventions.Source: §4.4 · pp. 7–8

4.5 Empirical Lessons

  • The empirical lessons are that DevEval poses a harder repository-level challenge, local-file context and detailed requirements help, and long heterogeneous contexts remain difficult.The authors note that long, heterogeneous contexts can lead models to disregard contextual knowledge and generate hallucinated functions.Source: §4.5 · p. 8

5 Discussion

The discussion finds that repository context improves generation and dependency recovery, while LLMs still struggle to use available context. It also describes data-leakage risk and a slight downward bias in dependency recall.

Results on different program types

  • Adding local files raises Pass@1 on non-standalone functions from 11.04 to 53.58, and contexts also benefit standalone functions.The authors attribute the standalone improvement to domain knowledge in contexts that helps models understand requirements.Source: §5 · p. 8

Results on different dependency types

  • With contexts, gpt-4 improves dependency generation, but Recall@1 remains a challenge, especially for cross-file dependencies.Without context, dependency Recall@1 is low across intra-class, intra-file, and cross-file types; models may ignore dependencies already present in contexts.Source: §5 · p. 8, Figure 6 · p. 8

Data leakage

  • The authors judge data-leakage risk to have only slight impact, citing new manually written requirements and a 0.36 average Pass@1 difference between repository groups.They compare that difference with average variation of 5.15 between LLMs and encourage excluding selected repositories from future training data.Source: §5 · p. 8

The bias of Recall@k

  • Static dependency extraction may miss Python dependencies determined only at runtime, making Recall@k slightly lower than actual values.A comparison on 50 gpt-4 programs found a 0.16 Recall@1 bias, which the authors consider slight relative to 2.16 average variation between LLMs.Source: §5 · p. 9

6 Related Work

Related benchmarks cover function-, competition-, class-, and repository-level code tasks, but several repository benchmarks target code completion or issue repair rather than generating new programs from requirements. DevEval is positioned as a repository-aligned benchmark with more comprehensive annotations.

Code Generation Benchmarks

  • Earlier benchmarks cover relatively simple function tasks, competition-style problems, class-level generation, and non-standalone programs.The examples include HumanEval and MBPP for function generation, APPS for competition-style problems, and ClassEval for class-level generation.Source: §6 · p. 9
  • Compared with existing benchmarks, DevEval aligns with real repository code and dependency distributions and offers more comprehensive annotations, including reference dependencies.Its annotations include requirements, original repositories, reference code, and reference dependencies.Source: §6 · p. 9
  • CrossCodeEval, RepoBench, and RepoEval evaluate code completion, while SWE-bench focuses on repository issue repair.DevEval instead targets generating new repository programs from natural-language requirements.Source: §6 · p. 9

7 Conclusion and Future Work

The paper presents DevEval as a repository-aligned benchmark and evaluates eight LLMs to reveal their strengths and weaknesses in real repositories. Future work will expand the benchmark and explore ways to improve context-based code generation.

Conclusion

  • DevEval aligns with real repositories in code distributions, dependency coverage, and repository scale, and the authors describe it as a more challenging and practical evaluation scenario than previous benchmarks.The paper hopes the benchmark will facilitate LLM applications in practical repositories.Source: §7 · p. 9
  • Evaluation of eight popular LLMs on DevEval reveals strengths and weaknesses in real repositories.The supplied conclusion does not report additional comparative scores.Source: §7 · p. 9

Future Work

  • Future work will add multilingual samples, projects, and test cases, and explore retrieval-augmented and tool-augmented generation for context-based code generation.Source: §7 · p. 9

9 Limitations

The authors identify limits in language coverage, dependency-recall measurement, and repository context scope. DevEval currently uses English requirements, Python code, and local-file contexts, while its parser may miss runtime-only dependencies.

Limitations

  • DevEval is monolingual, with English requirements and Python code, and therefore omits other natural and programming languages.The authors plan to build a multilingual version.Source: §9 · p. 10
  • Recall@k may slightly understate actual values because the Python parser can miss dependencies identifiable only at runtime.The authors report a 0.16 Recall@1 bias, which they consider slight compared with 2.16 average variation between LLMs.Source: §9 · p. 10
  • Experiments use code contexts from local files only; broader contexts such as imported and sibling files remain future work.Source: §9 · p. 10

10 Ethics Consideration

DevEval samples are manually checked for private information, offensive content, and normal program behavior; malicious programs are excluded.

10 Ethics Consideration

  • All DevEval samples are manually checked.Source: §10 · p. 10
  • The checks ensure samples contain no private information or offensive content.Source: §10 · p. 10
  • The checks ensure programs behave normally, and malicious programs are excluded.Source: §10 · p. 10

Related papers

Loading 2405.19856v1…