Source-linked AI summary
PolyCodeEval: Benchmarking Multilingual Code Generation from Functions to Repositories
Bowen Yang, Jiajun Jiang, Luxue Yu, Yihao Wang, Fengjie Li, Dong Wang
TL;DR
Existing code-generation benchmarks fragment language coverage, task granularity, and evaluation, leaving limited unified evidence for comparing repository-constrained generation. PolyCodeEval builds a multilingual, multi-granularity benchmark from executable repositories and evaluates generated artifacts through build and functional testing. Results show level-specific bottlenecks, including a function-level full-pass advantage for RepoCoder with GPT-5.4 over LLM-direct generation.
Problem
Method
PolyCodeEval constructs tasks from shared repositories across generation scopes and evaluates integrated artifacts through build and functional testing.
Source: §2 · p. 5Results
RepoCoder with GPT-5.4 achieves a 71.7% function full-pass rate versus 48.8% for LLM-direct generation, while repository-level full-pass rates remain low.
Source: §1 · p. 3Takeaways & Limitations
The findings identify engineering integration, interface and dependency modeling, and function-context selection as distinct improvement needs across generation levels.
Source: §1 · p. 3The benchmark favors actively maintained open-source repositories with mature engineering practices and testing, so findings may not generalize directly to private software or legacy systems.
Source: §6 · p. 19
Abstract
from arXiv · showhide
As large language models increasingly move toward repository-level software engineering, existing code-generation benchmarks remain fragmented across language coverage, task granularity, and evaluation protocols, impeding systematic comparison. To address this gap, we present PolyCodeEval, a unified multilingual and multi-granularity benchmark for code generation. It comprises 2,590 code generation tasks spanning functions to repositories, derived from 58 real, executable open-source repositories in five programming languages. All tasks are evaluated under a unified execution-based protocol with integration procedures tailored to their generation targets. Building on this benchmark, we evaluate frontier large language models, state-of-the-art specialized methods, and general coding agents. Our results show that existing approaches still struggle to correctly generate complete code fragments across granularities and languages. Specifically, the studied methods generate at most 71.7%, 76.7%, and 31.0% correct functions, files, and repositories, respectively, with performance varying widely across languages. Paired experiments further show that implementation context from related functions in the same file improves the executable correctness of function generation. Method rankings also vary across task granularities and programming languages, highlighting the importance of multilingual, multi-granularity evaluation for comprehensively assessing code generation capabilities.
1 Introduction
Existing benchmarks fragment language coverage, task granularity, and validation, limiting comparisons of code generation under repository constraints. PolyCodeEval provides a shared multilingual benchmark and execution-based evaluation platform spanning functions to repositories.
1 Introduction
- PolyCodeEval defines four generation levels across 58 executable open-source repositories in five languages, comprising 2,590 tasks for controlled cross-level comparison.The levels cover complete-repository generation, skeleton-conditioned repository generation, file generation, and function generation.Source: §1 · p. 2
- Its unified parsing and execution pipeline reintegrates generated artifacts into Oracle-derived repositories and validates tasks through build, test, coverage, and Oracle checks.L0/L1 use interface-level functional tests, while L2/L3 use repository backfilling followed by build and test execution.Source: §1 · p. 2
- The empirical study compares direct generation, repository-aware methods, and coding agents across task levels under a unified execution-based protocol.Evaluated systems include specialized methods at L3, direct generation and agents at L2, and coding agents at L0/L1.Source: §1 · pp. 2–3
- RepoCoder with GPT-5.4 achieves a 71.7% function full-pass rate versus 48.8% for LLM-direct generation, while repository-level full-pass rates remain low.Build-successful repository generations typically pass a high proportion of tests, but few pass the complete suite.Source: §1 · p. 3
- Paired experiments find that related-function implementation context improves executable correctness, while capability bottlenecks differ across repository, file, and function generation.The findings point to engineering integration, interface and dependency modeling, and context selection as distinct improvement needs at those levels.Source: §1 · p. 3
2 Related Work and Limitations
Prior benchmarks cover functions, repository-context tasks, repository generation, or multilingual code generation, but few combine broad language coverage with multiple generation scopes. Their different data sources and evaluation protocols also make cross-granularity comparisons difficult.
2 Related Work and Limitations
- Function-level benchmarks such as HumanEval and MBPP focus on isolated programs, while repository-context benchmarks extend toward cross-file completion and issue resolution.The cited work includes multilingual cross-file completion and benchmarks built around repository issues and patches.Source: §2 · pp. 3–4
- Repository-level benchmarks evaluate whole-project generation, but differ in inputs and assessment, including simulated interaction, code similarity, online judging, and code review.ProjDevBench reports 27.38% acceptance across 20 problems, while the passage describes ProjectEval's distinct evaluation approach.Source: §2 · p. 4
- Existing benchmarks rarely jointly cover multiple languages and scopes from functions through files to complete repositories.MultiPL-E and xCodeEval broaden language coverage, while many other benchmarks center on Python or local code units.Source: §2 · pp. 4–5
- Granularity-specific datasets, input formats, and evaluation protocols confound performance differences with repository, language, domain, and task variation.Textual similarity or exact match also provides limited evidence about build success, interface compatibility, dependencies, and integration.Source: §2 · p. 5
- PolyCodeEval constructs tasks from shared repository and functional foundations while varying generation scope and context, then evaluates integrated artifacts through build and functional testing.The construction aims to support systematic comparison across languages and granularities.Source: §2 · p. 5
3 PolyCodeEval Benchmark
PolyCodeEval derives repository-, file-, and function-generation tasks from a shared pool of executable repositories. It standardizes their inputs and validates generated artifacts through level-specific execution procedures.
3.1 Benchmark Overview
- 2,590 tasks span repository, file, and function generation across five languages and 58 executable repositories, with all levels derived from a shared Oracle pool.The shared pool is intended to enable more controlled comparisons across generation granularities.Source: §3.1 · pp. 5–6
3.2 Multi-Granularity Task Definition
- L0–L3 vary artifact scope and visible repository information, progressing from empty-workspace repositories through skeletons and files to embedded functions.The function-level tasks retain repository context and interface information rather than isolating the function from its software system.Source: §3.2 · p. 6
3.3 Benchmark Construction
- Construction yields 58 tasks each for L0 and L1, 150 file-level tasks, and 2,324 function-level tasks from 58 qualified Oracle repositories.Repositories are screened for reproducible builds and tests, adequate structure, and usable language-specific source code.Source: §3.3 · pp. 7–8
3.4 Input Construction and Quality Assurance
- Inputs combine level-permitted repository evidence with behavior specifications, reviewed by three models and manually revised until every input scores above 3.5.The Oracle implementation grounds specification construction and quality checks but is not exposed to the code-generation model.Source: §3.4 · pp. 8–9
3.5 Execution-Based Validation
- 93.9% mean line coverage after supplementation, with all 58 repositories above 85%; all 2,590 tasks also pass Oracle validation.L0/L1 use interface-level tests, while L2/L3 backfill completions into Oracle-derived repositories for testing.Source: §3.5 · pp. 10–11
4 Case Study Design
The case study compares code-generation methods across repository, file, and function tasks, including the effects of context. It evaluates outputs with paired execution-based metrics under controlled task environments.
4.1 Research Questions
- The study asks how methods perform across languages at repository, file, and function levels, and how context affects generation.These are the four research questions guiding the case study.Source: §4.1 · p. 11
4.2 Studied Methods
- The comparison includes direct models, general-purpose coding agents, and specialized methods selected for task alignment, technical distinction, reproducibility, and cross-language feasibility.For L2, agent-mediated repository inspection is compared with direct generation; L3 methods use the same LLM backends to compare context-processing strategies.Source: §4.2 · pp. 11–12
4.3.1 Procedure.
- Methods receive task-level-appropriate inputs, and frozen outputs are reintegrated into Oracle-derived repositories for evaluation using the same dependencies, build commands, and tests.L0/L1 outputs reconstruct repositories, L2 replaces files, and L3 backfills functions; L2/L3 use associated tests while L0/L1 run the complete suite.Source: §4.3.1 · pp. 12–13
4.3.2 Evaluation Metrics.
- Evaluation reports Build, conditional Test, and strict Full Pass rates, with paired task-level differences tested using McNemar or Wilcoxon tests and Holm correction.The significance threshold is α=0.05.Source: §4.3.2 · p. 13
4.3.3 Configurations and Environments.
- Specialized methods use their default settings and LLMs are accessed through APIs or native CLIs; experiments run in isolated workspaces and fixed Docker environments.Direct API calls and specialized-method final generation use temperature 0.2.Source: §4.3.3 · p. 13
5 Evaluation Results
Across repository, file, and function tasks, execution results vary by method and language, with trade-offs in correctness, context use, and resource consumption. Related implementation context improves function-level full-pass rates, while repository skeletons raise full-pass rates but reduce build success.
5.1 RQ1: Effectiveness in Repository-Level Code Generation
- At L0, general-purpose agents outperform ChatDev, with Claude Code strongest; at L1, Claude Code reaches 31.0% Full pass despite lower Build success than at L0.Codex is more cost-efficient than Claude Code, while repository skeletons improve full-pass rates but increase resource use.Source: §5.1 · pp. 13–14, Table 4 · p. 14
5.2 RQ2: Effectiveness in File-level Code Generation
- Codex with GPT achieves 88.0% Build and 76.7% Full, improving over Direct with GPT by 17.3 and 21.4 percentage points, respectively.The build-success improvement is statistically significant under the exact McNemar test after Holm correction (p=3.08 × 10−5).Source: §5.2 · p. 15, Table 5 · p. 16
- Claude Code with Sonnet shows no clear overall file-level improvement over Direct with the same backbone, and some agent runs time out, stop before implementation, or introduce interface errors.Repository access supplies useful information but does not ensure effective file completion.Source: §5.2 · p. 15
- File-level gains vary by language, while function-level repository-aware methods use 3.5x–10.4x the tokens of Direct with GPT.Codex improves file-level Build most strongly for Java and C++, whereas function-level retrieval and multi-stage context processing add substantial token overhead.Source: §5.2 · p. 15, §5.3 · p. 17
5.3 RQ3: Effectiveness in Function-level Code Generation
- RepoCoder with GPT achieves the highest overall function-level Build and Full rates, at 85.4% and 71.7%, respectively.Under the same GPT backbone, RepoCoder and HCP significantly outperform Direct on Build, Test, and Full; AlignCoder’s improvement is smaller and significant only on Build.Source: Table 6 · p. 16, §5.3 · p. 17
- Build-successful functions often pass many tests without passing the complete suite, and files show the same gap between Test rates of 93.0%–97.4% and Full rates of 55.3%–76.7%.Generated code may miss boundary cases, exceptional paths, or repository-specific constraints.Source: §5.2 · p. 15, §5.3 · p. 17
5.4 RQ4: The Impact of Context on Code Generation Performance
- Repository skeletons increase Full and Test rates but decrease Build for both agents; Claude Code’s Full rate rises from 22.4% to 31.0%.Skeletons expose file organization and interfaces, but fixed structural and dependency constraints can prevent builds.Source: §5.4 · p. 18, Figure 4 · p. 18
- Related implementation context raises function-level Full rates by 4.9 percentage points for GPT-5.4 and 11.4 points for Claude Sonnet 4.6 across 1,891 shared tasks.Both increases are statistically significant.Source: §5.4 · p. 18, Figure 4 · p. 18
6 Threats to Validity
The study’s findings are bounded by its repository selection, automatically generated task descriptions, and possible training-data contamination.
6 Threats to Validity
- The benchmark favors actively maintained open-source repositories with mature engineering and testing practices, so findings may not generalize directly to private, legacy, or other repository types.The reported results are measurements on public, real-world repository tasks.Source: §6 · p. 19
- Task descriptions are generated by Claude Sonnet 4.6 and may not fully capture intended functionality, although independent LLM review and manual revision address low-scoring tasks.Source: §6 · p. 19
- Because public repositories may occur in model pretraining data, contamination or memorization cannot be excluded despite withholding implementations and executing artifacts in controlled environments.Source: §6 · p. 19
7 Conclusion
PolyCodeEval unifies multilingual code-generation evaluation across repository, file, and function tasks. Its empirical study finds incomplete correctness and highlights context use and cost-effectiveness as important dimensions of performance.
7 Conclusion
- PolyCodeEval is a multilingual, multi-granularity benchmark for code generation in real-world software repositories.It spans five programming languages and four task levels, from whole-repository generation to function-level completion.Source: §7 · p. 19
- A unified build-and-test protocol supports consistent evaluation of direct generation, specialized methods, and general-purpose agentic frameworks.Source: §7 · p. 19
- The study finds existing methods’ performance unsatisfactory and identifies effective context use and the cost-effectiveness trade-off as key findings.The authors underscore the need for execution-driven, multilingual, multi-granularity evaluation.Source: §7 · p. 19