EvoCodeBench: An Evolving Code Generation Benchmark Aligned with Real-World Code Repositories

1 Introduction

• EvoCodeBench is collected from real-world repositories and aligns with their code and dependency distributions across multiple dimensions.The paper reports that its distributions are consistent with those in 500 real-world repositories.Source: §1 · p. 2
• EvoCodeBench provides requirements, reference code and dependencies, and test cases; it reports Pass@k and proposes Recall@k for predicted dependencies.The annotations also include original repositories and reference dependencies.Source: §1 · p. 2
• The evolving benchmark is intended to avoid data leakage through periodic updates; EvoCodeBench-2403 contains 275 samples from 25 real-world repositories.The paper gives six months as an example update interval.Source: §1 · p. 2
• Repository-level code generation asks models to generate code from requirements and a complete repository, simulating developers’ coding process in a working repository.This task is presented as a more realistic scenario than a simple requirement-to-code task.Source: §1 · p. 2
• gpt-4-turbo-1106 achieves 80% Pass@1 on HumanEval but only 20.73% on EvoCodeBench, and the evaluated LLMs show low performance compared with previous benchmarks.The paper evaluates 10 popular LLMs and analyzes their failed cases and shortcomings in repository-level code generation.Source: §1 · p. 2
• The paper identifies five desired benchmark features, including real-world repositories, realistic code and dependency distributions, comprehensive annotations, robust metrics, and leakage avoidance.Real-world repository samples should cover standalone and non-standalone code and reflect realistic code and dependency distributions.Source: §1 · pp. 1–2

2.2 Task Definition

• Repository-level code generation asks models to generate code from a requirement and a working repository, unlike simple requirement-to-code tasks.Each sample includes a function signature, requirement, repository, reference implementation, reference dependencies, and test cases.Source: §2.2 · p. 3, §2.1 · p. 3

2.3 Evaluation Metrics

• Pass@k evaluates functional correctness through tests, while Recall@k measures how many reference dependencies appear in generated programs.For Recall@k, the authors parse predicted dependencies and compare them with reference dependencies.Source: §2.3 · pp. 3–4

2.4 EvoCodeBench-2403

• EvoCodeBench-2403 comprises 275 samples collected from 25 real-world repositories; its code distribution is consistent with that of 500 real-world repositories, and its average number of dependencies per program is close to theirs.Source: §2.4 · p. 4
• Each sample provides requirements, reference code, reference dependencies, and a complete repository; dependency paths disambiguate same-named functions.The paper contrasts these annotations with prior works that provide dependency names only.Source: §2.4 · p. 4
• EvoCodeBench-2403 uses repositories created from 2023-10 to 2024-2, and future versions are intended to use continually updated repositories.The paper relates this collection window to the latest LLM's training data ending in 2023-9.Source: §2.4 · p. 4

Repository Selection and Function Parse

• Repository selection targets recent, non-fork, non-malicious open-source Python projects with more than 50 stars and explicit unit tests.The pipeline then extracts function signatures, reference code, and dependencies, excluding trivial functions.Source: §3 · pp. 4–5

Tests Construction

• Functions without executable tests are excluded after the pipeline extracts invoking tests and runs them with Pytest.Required packages are installed with pip for each repository.Source: §3 · p. 5

Deduplication

• Repository-level deduplication uses Jaccard similarities in code files and imports to remove textual duplicates and overly similar domains..Source: §3 · p. 5

Requirement Annotation

• LLMs generate natural-language requirements from a few-shot prompt specifying functional descriptions and input-output parameters.The paper motivates this step as an alternative to time-consuming manual requirement writing.Source: §3 · p. 5

Benchmark Construction

• Final sample selection targets the code distribution and average dependency count observed in 500 real-world repositories while including as many samples as possible.The paper describes an automatic pipeline for collecting EvoCodeBench from the latest repositories.Source: §3 · pp. 4–5

4.1 Studied LLMs

• The study evaluates 10 LLMs, including closed-source gpt-4 and gpt-3.5 and open-source models such as DeepSeek Coder, StarCoder 2, CodeLLaMa, Gemma, and Qwen 1.5.The authors use official interfaces or implementations.Source: §4.1 · p. 5

4.2 Experimental Setting

• The three settings provide no context, preceding local-file code for completion, or code above and below the target for infilling; other context-extraction approaches are left for future work.Repositories average 1.1 million tokens, exceeding the cited 128k-token gpt-4 context window.Source: §4.2 · p. 5

4.3 Evaluation

• Pass@k and Recall@k are reported on EvoCodeBench-2403; k=1 uses greedy generation, while k>1 samples 20 programs with top-p 0.95 and temperature 0.4.The paper cautions that metrics across benchmark versions are not comparable.Source: §4.3 · pp. 5–6

4.4 Main Results

• 7.27 on Pass@1 on EvoCodeBench-2403 versus 88.4 on HumanEval for gpt-4; local-file context improves gpt-4 Pass@1 by 104% and 152%.The improvements correspond respectively to Local File (Completion) and Local File (Infilling).Source: §4.4 · p. 6
• 29 of 50 gpt-4 errors in Local File (Infilling) involved implementation logic, 20 involved missing context, and one involved a vague requirement.The authors conclude that reasoning and coding abilities need improvement and that using more context warrants exploration.Source: §4.4 · p. 7
• Without context, models still generate some dependencies by reasoning from requirements or guessing from dependency names and functionality.The authors attribute such guesses to learned naming conventions in large code corpora.Source: §4.4 · p. 7

4.5 Empirical Leassons

• More context improves performance, while gpt-family models show higher Pass@k and lower Recall@k than other models, which show the opposite pattern.The paper's empirical lessons also identify repository-level generation as a challenge and logic errors and incomplete context as common failure sources.Source: §4.4 · p. 7, §4.5 · pp. 7–8

Evaluation of auto-generated requirements

• On 41 of 50 functions, gpt-4 and developers tied; gpt-4 won 5 and developers won 4, with missing hyperparameters among gpt-4’s errors.Two developers wrote requirements and two others evaluated completeness and clarity; the paper reports that gpt-4’s requirements were comparable to human-written ones in 92% of cases and cost less time and money.Source: §5 · p. 8

Retrieval-Augmented Generation

• Adding five similarly named functions as context improved both LLMs’ performance by supplying relevant algorithms and dependencies.The retrieval process tokenizes function names by underscores or camel case; Table 6 reports Pass@1 and Recall@1 with retrieval-augmented generation.Source: §5 · p. 8, Table 6 · p. 8

Results on different program types

• Code contexts significantly improved Pass@1 on non-standalone functions and slightly increased it on standalone functions, though results remained low for both.The authors speculate that context containing domain knowledge, such as private objects, helps models understand requirements.Source: §5 · p. 8

Results on different dependency types

• Local files improve Recall@1 for intra-class and intra-file dependencies; copied cross-file dependencies can raise recall while lowering Pass@1 when their arguments are inconsistent.Without context, models show low Recall@1 across all three dependency types.Source: Figure 5 · p. 9, §5 · p. 9

The bias of Recall@k

• The parser’s Recall@1 bias was 0.16, slight beside the 7.77 average variation between LLMs, and the authors conclude Recall@k can still rank them effectively.Because Python is dynamically typed, dependencies identified only at runtime may escape the static-analysis parser.Source: §5 · p. 9

Code Generation Benchmarks

• Earlier benchmarks cover Python functions, competition problems, classes, and non-standalone programs, while repository-level completion benchmarks do not provide the requirements needed for code generation.SWE-bench instead focuses on repairing repository issues by revising existing programs.Source: §6 · p. 9

EvoCodeBench

• Compared with existing benchmarks, EvoCodeBench aligns with real repositories in code and dependency distributions, provides more comprehensive annotations, and is dynamically updated to address data leakage.The paper describes it as an extension to code-generation benchmarks.Source: §6 · p. 9

Code Generation Benchmarks

• EvoCodeBench targets generating new repository programs from requirements and includes natural-language requirements, original repositories, reference code, and reference dependencies.This contrasts with completion benchmarks CrossCodeEval, RepoBench, and RepoEval, and repair-focused SWE-bench.Source: §6 · p. 9

Conclusion

• EvoCodeBench aligns with real repositories in multiple dimensions, is updated periodically, and supports repository-level code generation evaluation.The paper describes the benchmark as a more challenging and realistic evaluation scenario than previous benchmarks.Source: §7 · p. 10
• Evaluations of 10 popular LLMs reveal strengths and weaknesses in real-world repositories.Source: §7 · p. 10

Future Work

• Future work includes adding multilingual samples and exploring retrieval-augmented and tool-augmented generation to improve repository-level performance.Source: §7 · p. 10

Benchmark limitations

• EvoCodeBench uses English requirements and Python code, so it does not cover other natural or programming languages.Source: §8 · p. 10
• Auto-generated requirements can omit necessary details, despite being comparable to human-written requirements in the paper’s evaluation.Source: §8 · p. 10
• Runtime-only dependencies can escape the parser, making Recall@k slightly lower than actual values; the measured bias was 0.16.The authors report that the bias is slight relative to average variation between LLMs.Source: §8 · p. 10

Evaluation limitations

• Limited computing budgets constrained evaluation to 10 mainstream LLMs, straightforward context methods, and no careful tuning of prompts or sampling hyperparameters.The paper proposes evaluating more model sizes and fine-tuned models, exploring cross-file context, and tuning settings in future work.Source: §8 · p. 10

• EvoCodeBench samples come from real-world open-source repositories.Source: §9 · p. 10
• All samples were manually checked to exclude private information and offensive content.Source: §9 · p. 10
• Programs were checked for normal behavior, and malicious programs were excluded.Source: §9 · p. 10

• The process manually writes a function with requirements as a demonstration example.Source: §A.1 · p. 13
• GPT-4 generates corresponding requirements after the input functions are supplied.Source: §A.1 · p. 13
• Greedy search is used to generate a requirement for each function.Source: §A.1 · p. 13

B.1 Base LLMs

• Six popular LLMs are selected as base models for evaluation on EvoCodeBench-2403.Source: §B.1 · p. 13
• The evaluation uses gpt-4-1106 and gpt-3.5-turbo-1106; their training data extends through April 2023 and September 2021, respectively.Source: §B.1 · p. 13
• The evaluated code-model variants include CodeLLaMa-Python-{7B, 13B} and DeepSeek Coder-{6.7B, 33B}.Source: §B.1 · p. 13
• StarCoder2-{7B, 15B} is evaluated, using models trained on over 3.5 trillion tokens in 17 programming languages from Stack v2.Source: §B.1 · p. 14
• The evaluated variants also include Gemma-7b and Qwen1.5-7b.Source: §B.1 · p. 14

B.2 Prompt Templates

• Instruction-tuned models use prompt templates shown in Figures 7–10, with placeholders for function name, surrounding context, signature, and requirement.Source: §B.2 · p. 15, Figure 7 · p. 15, Figure 8 · p. 15
• Standard-model prompts use no context, local-file completion, or local-file infilling; infilling prompts include official special tokens.Source: §B.2 · p. 15