PaperlayerView exact revision on arXiv

Source-linked AI summary

MultiPL-E: A Scalable and Extensible Approach to Benchmarking Neural Code Generation

Federico Cassano, John Gouwar, Daniel Nguyen, Sydney Nguyen, Luna Phipps-Costin, Donald Pinckney, Ming-Ho Yee, Yangtian Zi, Carolyn Jane Anderson, Molly Q Feldman, Arjun Guha, Michael Greenberg, Abhinav Jangda

arXiv:2208.08227v4cs.LGcs.PL

TL;DR

Code generation models showed promise in Python, but their ability to generalize across programming languages was poorly understood. MultiPL-E translates unit-test-driven benchmarks into 18 additional languages and evaluates three models across them, finding strong performance in several languages, especially JavaScript. The resulting benchmarks support broader multilingual evaluation while retaining important representativeness and translation limitations.

  • Problem

    The central gap is limited evidence about how well code generation models perform across programming languages beyond Python.

  • Method

    MultiPL-E translates HumanEval and MBPP from Python into 18 additional languages and uses the parallel benchmarks to evaluate Codex, CodeGen, and InCoder.

  • Results

    Across the multilingual benchmarks, performance is consistently high on JavaScript and TypeScript and sometimes exceeds Python, while Codex performs strongly across several languages.

  • Takeaways & Limitations

    The benchmark suite enables comparative study of language features and popularity while making multilingual code-correctness evaluation possible.

  • Takeaways & Limitations

    The translated benchmarks may not represent real-world programming tasks because the evaluated languages are often used for different kinds of work.

Abstract

from arXiv · show

Large language models have demonstrated the ability to generate both natural language and programming language text. Such models open up the possibility of multi-language code generation: could code generation models generalize knowledge from one language to another? Although contemporary code generation models can generate semantically correct Python code, little is known about their abilities with other languages. We propose MultiPL-E, a system for translating unit test-driven code generation benchmarks to new languages. We create the first massively multilingual code generation benchmark by using MultiPL-E to translate two popular Python code generation benchmarks to 18 additional programming languages. We use MultiPL-E to extend the HumanEval benchmark and MBPP benchmark to 18 languages that encompass a range of programming paradigms and popularity. Using these new parallel benchmarks, we evaluate the multi-language performance of three state-of-the-art code generation models: Codex, CodeGen, and InCoder. We find that Codex matches or even exceeds its performance on Python for several other languages. The range of programming languages represented in MultiPL-E allow us to explore the impact of language frequency and language features on model performance. Finally, the MultiPL-E approach of compiling code generation benchmarks to new programming languages is both scalable and extensible, making it straightforward to evaluate new models, benchmarks, and languages.

1 INTRODUCTION

MultiPL-E addresses the limited evaluation of code generation beyond Python by translating unit-test-driven benchmarks into parallel versions for 18 additional languages. The framework supports multilingual model evaluation and investigation of language-related performance factors.

  • Existing code generation models are often trained on multiple languages but typically evaluated only on Python.
  • MultiPL-E translates Python code generation benchmarks into 18 additional programming languages and produces parallel problems across languages.
  • Its lightweight compilers translate function signatures, unit tests, behavioral comments, and type annotations without translating function bodies.
  • The framework extends HumanEval and MBPP to 18 languages spanning varied language features, application areas, and popularity levels.
  • MultiPL-E evaluates Codex, CodeGen, and InCoder across the resulting multilingual benchmarks.
  • The evaluation examines language frequency, type annotations, prompt translation sensitivity, and fine-grained errors.

2 CODE GENERATION

Code generation models complete natural-language-to-code prompts token by token, using signatures, descriptions, and examples as implicit language cues. Multi-language evaluation therefore requires translating prompt components and judging generated functions with unit tests.

  • Code generation models can condition on both natural-language descriptions and programming-language text when synthesizing programs.
  • Natural-language-to-code generation asks a model to complete a function body from a natural-language description.
  • A code-generation prompt can contain a function signature, behavioral comment, and optional doctest examples.
  • Prompt translation must address syntax, terminology, and doctest regions because each implicitly signals the target language.
  • Completions are token sequences rather than abstract syntax trees, so stop sequences control output that may continue beyond one function.
  • Generated functions are evaluated by whether they pass all hidden unit tests, with multiple stochastic completions commonly sampled per problem.

3 THE MULTIPL-E APPROACH

MultiPL-E translates Python code-generation benchmarks into parallel versions for many languages by compiling prompts, tests, types, and terminology. Its design supports broad language coverage while retaining explicit constraints where translation is difficult.

  • Benchmark Selection: MultiPL-E translates HumanEval and MBPP, two widely used Python benchmarks, into multiple programming languages for parallel evaluation.
  • Benchmark Selection: MBPP prompts are standardized by adding function signatures and hiding assertions so the assertions serve as hidden unit tests.
  • Language Selection: MultiPL-E supports 19 programming languages spanning four frequency classes, with half statically type-checked and eight previously unused for NL2Code evaluation.
  • Compiling Python Benchmarks: Each lightweight compiler translates function signatures, unit tests, behavioral comments, and type annotations rather than function bodies.
  • Compiling Unit Tests: Unit-test compilation converts Python values and equality checks into language-specific forms, including deep equality and imported testing-library functions where needed.
  • Limitations of Our Approach: Translation is constrained by unsupported higher-order-function tests, untranslatable types, excluded HumanEval problems, and target languages lacking natural Python data-type analogues.

4 CODE GENERATION MODELS

The study evaluates three Transformer-based code-generation models trained on mixtures of natural language and code. It compares their multilingual performance using pass@k estimates across translated benchmark languages.

  • Models: Three Transformer-based models—InCoder, CodeGen, and Codex—are evaluated using their largest, best-performing versions.
  • Models: InCoder is a 6.7B-parameter model supporting code infilling and completion, but the evaluation tests only completion.
  • Models: CodeGen is a 16.1B-parameter multilingual model trained with next-token prediction and subsequently fine-tuned on six programming languages.
  • Models: Codex is evaluated with the multilingual codex-davinci-002 model through the public OpenAI API, although its training-set details and size are not public.
  • Evaluation: Performance is reported with pass@k, where pass@1 is the likelihood that one completion passes all unit tests and pass@10 or pass@100 allow multiple sampled completions.

5 EVALUATION

The evaluation benchmarks three code generation models across translated HumanEval and MBPP tasks, revealing strong performance in JavaScript and several less common languages. Results also show model- and benchmark-specific differences, with language frequency generally associated with performance but not determining it uniformly.

  • Codex performs similarly to Python on C++, JavaScript, Scala, and TypeScript in MultiPL-HumanEval.
  • InCoder performs significantly better on Python than every other MultiPL-HumanEval language.
  • Codex performance on JavaScript is higher than Python by 2.3%, but the difference is not significant (p = 0.43).
  • Codex achieves pass@1 above 40% on C++, Java, TypeScript, PHP, Ruby, Rust, Scala, and Lua.
  • All three models perform best on high-frequency languages, although Codex performs very well on some low- and niche-frequency languages such as Lua.
  • Perplexity does not strongly correlate with Codex pass@1, so it may not reliably evaluate NL2Code correctness.
  • Across MultiPL-MBPP, Codex and CodeGen remain strong on JavaScript and other languages, while InCoder exceeds Python on TypeScript, JavaScript, and PHP.
  • MBPP generally yields higher pass@1 than HumanEval, suggesting HumanEval may provide a more useful indication of performance with a more efficient sample size.

6 FACTORS IN CODE GENERATION SUCCESS

The follow-up experiments examine translation choices, typing, prompt formatting, and problem types as determinants of code generation success. Doctest presence and language-specific prompt design matter, while broad effects of type annotations and problem type are limited.

  • Prompt translation: Translating doctests and Python-specific terminology has little impact on better-performing languages but matters more for Bash, PHP, Perl, R, Rust, Swift, and TypeScript.
  • Prompt translation: Full Translation differs significantly from Test-Only Translation (p = 0.03), while No Doctests differs from Test-Only Translation (p < 0.001).
  • Prompt translation: The experiments suggest Python terminology translation has a small reliable effect, whereas doctest presence is important but doctest translation is not.
  • Type annotations: Codex shows no overall effect of type annotations on MultiPL-HumanEval (p = 0.33) or MultiPL-MBPP (p = 0.23).
  • Type annotations: Removing precise types from TypeScript lowers Codex pass@1 by 2.5% (p < 0.001), while removing Python annotations has no significant effect (p = 0.23).
  • Prompt design: Changing from single-line to multi-line comments improves Racket pass@1 by 1.9% but decreases PHP pass@1 by 3.1%.
  • Prompt design: Omitting Perl’s argument-naming prompt lowers Codex pass@1 by 8% (p < 0.001).
  • Problem types: No significant effect of problem type is found when programming language is treated as a random effect, although tuple questions appear difficult for many languages.

7 THREATS TO VALIDITY

The paper’s main validity concern is that translated benchmark problems may not represent real programming tasks in each language. Results may also vary with prompt design and completion-sampling choices.

  • Translated benchmarks may not represent the problems programmers typically solve in each language, limiting real-world interpretation.
  • The benchmarks mix scripting and systems languages on the same task, despite programmers often using them for different tasks.
  • Pass rates may improve with more language-specific prompt engineering because code generation models are sensitive to small prompt changes.
  • The study uses sampling settings optimized for Python, although other languages may benefit from different configurations.

8 RELATED WORK

Related work includes multilingual evaluation, monolingual benchmarks, alternative metrics, and other code-generation tasks and models. MultiPL-E distinguishes itself through compiler-based translation, hidden unit-test evaluation, and broader language and benchmark coverage.

  • Early approaches: Earlier work often used textual similarity metrics, but prior findings show weak correlation between such metrics and code correctness.
  • Extensions: The approach could extend to other Python benchmarks such as MathQA-Python and APPS.
  • Other multi-language evaluation: Xu et al. evaluate 12 languages with perplexity, whereas MultiPL-E uses unit tests and checks code correctness beyond Python.
  • Other multi-language evaluation: HumanEval-X manually translates four languages, while MultiPL-E supports 18 languages and both HumanEval and MBPP through compiler-based translation.
  • Comparison with MBXP: MultiPL-E evaluates correctness with hidden unit tests, unlike MBXP, which exposes the same tests used for evaluation to the model.
  • Comparison with MBXP: MultiPL-E translates Python types more faithfully into typed languages, including algebraic datatypes for types such as Either[X,Y] and Optional[X].
  • Comparison with MBXP: MultiPL-E uses sampling rather than MBXP’s greedy decoding, following prior evidence that sampling improves generated-code correctness.

9 CONCLUSION

MultiPL-E provides a massively parallel, multilingual code-generation benchmark by translating HumanEval and MBPP into 18 additional languages. The evaluation shows strong cross-language Codex performance, predictable language-frequency effects, mixed type-annotation effects, and recurring language-specific errors.

  • Benchmark and evaluation: MultiPL-E translates HumanEval and MBPP into 18 additional programming languages for parallel code-correctness evaluation.The benchmark is designed to compare models on consistent problems across languages.
  • Results: Codex performs similarly to Python on several languages, most notably JavaScript.The paper evaluates Codex, CodeGen, and InCoder using the translated benchmarks.
  • By-language analysis: Language frequency has a predictable relationship with performance, while type annotations produce mixed effects.The detailed analysis examines both language frequency and annotation choices.
  • Error analysis: Error analysis across four languages identifies model errors that resemble and differ from human programmers’ errors.The authors caution that the four analyzed languages are not representative of all benchmark languages.
  • Future use: The publicly available benchmark is easy to extend to new problems and languages, supporting future multilingual code-generation evaluation.The paper presents extensibility as a practical property of the released benchmark.

APPENDIX B DATASHEET

The datasheet describes MultiPL-HumanEval and MultiPL-MBPP as translated, cleaned benchmarks for evaluating code-generation models across programming languages. It documents their construction, contents, availability, and maintenance.

  • Purpose and provenance: The datasets were created to evaluate code-generation models across programming languages by translating the original Python benchmarks.HumanEval and MBPP were modified by the paper’s authors.
  • Dataset contents: The datasets represent programming problems in 18 programming languages, with each instance containing a natural-language description, function signature, and unit tests.Instances are labeled by function name and programming language.
  • Dataset size: MultiPL-HumanEval contains 3,059 instances, while MultiPL-MBPP contains 7,619 instances.These totals reflect cleaned Python problem sets multiplied by 19 programming languages.
  • Scope and maintenance: MultiPL-HumanEval excludes 3 of the 164 original problems, and the datasets are subject to updates and repository-based error acknowledgement.The datasheet does not state whether older versions will continue to be supported.
  • Availability and reuse: The datasets are self-contained, publicly available, and intended for evaluating other code-generation language models.The repository is identified as the distribution location.
  • Construction: The translated versions and prompt variations were produced by a suite of compilers after manual cleaning of the original Python datasets.Preprocessing added missing type annotations, standardized docstrings, and changed random tests into unit tests in two problems.

C.1 MultiPL-HumanEval Mixed-Effects Results from §5.1

The appendix describes mixed-effects analyses comparing model performance across languages, benchmarks, translation settings, type annotations, comment formats, and prompt features. Codex language comparisons use Python as the reference, while other models receive analogous analyses with adjustments for unstable random effects.

  • Language comparisons: Language comparisons model pass@1 completion rates with programming language as a fixed effect and problem number as a random effect, using Python as the reference.Codex analyses include random effects for problem number; broader comparisons may also include language.
  • Language comparisons: Separate tables report language-comparison estimates for Codex, CodeGen, and InCoder on MultiPL-HumanEval and MultiPL-MBPP.The appendix provides corresponding model-result tables for each model and benchmark.
  • Language frequency: Codex language-frequency analyses use mixed-effects models with frequency as a fixed effect and language and problem as random effects.Separate analyses are reported for MultiPL-HumanEval and MultiPL-MBPP.
  • Translation ablations: Ablation models compare Doctest-Only Translation, Full Translation, No Translation, and Remove Doctests for InCoder and Codex.The Codex analysis includes language-by-experiment interaction models, while InCoder omits problem random effects because of unstable estimates.
  • Type annotations and checking: Type-related experiments test static type-checking, Python annotation removal, and weakened or absent TypeScript annotations.The TypeScript analysis separately compares precise types with Any types, TypeScript with JavaScript, and checked with unchecked execution.
  • Prompt-format experiments: Additional models compare comment formats in PHP and Racket, argument-naming lines in Perl, and encoding comments and natural-language translation in Bash.Problem number is treated as a random effect in these language-specific experiments.
  • Analysis scope: The error analysis covers four languages, but their theme prevalence is not representative of all benchmark or unstudied languages.Error labels receive different levels of manual assessment, so themes are treated as more informative than individual labels.

D.2 Complete Error Themes

The appendix catalogs Codex error categories and illustrates concrete failures across Python, C#, Swift, and Racket. The examples include undefined identifiers, invalid types or APIs, language-specific syntax, unsupported inputs, and generation in another language.

  • Error taxonomy: The error tables organize language-specific failures into categories, themes, and labels, with frequent errors highlighted for each language.The appendix reports roughly 32,000 Codex completions per language for the full translation.
  • Undefined identifiers: C# and Racket examples show local-context errors such as calling a nonexistent method or using Python’s return keyword in Racket.These examples are classified under undefined-identifier errors.
  • Python failures: Python examples include an unimported Counter dependency and a completion consisting only of raise, producing runtime failures.The examples correspond to NameError and NotImplementedError failures.
  • Generation and evaluation errors: Other examples include Markdown generated instead of Racket, user-input calls causing EOFError, and comments or code produced in another language.The examples span generation-format, input-handling, and cross-language-generation themes.
  • Language-specific errors: Swift examples demonstrate invalid optional handling, missing argument labels, and integer indexing of strings.These failures reflect Swift-specific type and API requirements.

D.5 Additional Error Examples

This section presents manually inspected Codex errors outside the structured analysis, including untranslated or Python-like constructs in generated code across languages.

  • Manual inspection identified additional Codex output errors outside the structured analysis.
  • The inspected examples include a sum_squares function and a Lua will_it_fly function declaration.
  • The JavaScript example is described as a complete executable function body, but it is functional-style Python.
  • The Lua example calls an undefined helper and uses Python-like sum syntax, indicating incomplete translation.
Loading 2208.08227v4…