PaperlayerView exact revision on arXiv

Source-linked AI summary

Multi-lingual Evaluation of Code Generation Models

Ben Athiwaratkun, Sanjay Krishna Gouda, Zijian Wang, Xiaopeng Li, Yuchen Tian, Ming Tan, Wasi Uddin Ahmad, Shiqi Wang, Qing Sun, Mingyue Shang, Sujan Kumar Gonugondla, Hantian Ding, Varun Kumar, Nathan Fulton, Arash Farahani, Siddhartha Jain, Robert Giaquinto, Haifeng Qian, Murali Krishna Ramanathan, Ramesh Nallapati, Baishakhi Ray, Parminder Bhatia, Sudipta Sengupta, Dan Roth, Bing Xiang

arXiv:2210.14868v3cs.LGcs.CL

TL;DR

Code-generation research lacks broad multilingual execution-based evaluation. The paper introduces converted benchmarks and model analyses spanning languages, training settings, prompting, translation, and related code tasks, finding broad cross-language generalization and advantages for multilingual models. The benchmarks and synthetic solutions support further multilingual evaluation, while language-specific evaluations remain necessary for real-world software complexity.

  • Problem

    Code-generation evaluation needs execution-based benchmarks that assess program synthesis abilities across multiple programming languages and related tasks.

  • Method

    The paper converts Python execution-based datasets into multilingual benchmarks by transforming prompts and test statements, then trains and evaluates models across multilingual, monolingual, prompting, translation, and code-task settings.

  • Results

    The evaluations find that multilingual models often outperform monolingual models, generalize to out-of-domain languages, benefit from few-shot prompting, and support zero-shot translation.

  • Takeaways & Limitations

    The released benchmarks and synthetic canonical solutions provide resources for studying multilingual code generation, robustness, summarization, translation, and code insertion.

  • Takeaways & Limitations

    Translation performance depends on language compatibility, and additional language-specific evaluations may be needed for real-world software-development complexity.

Abstract

from arXiv · show

We present new benchmarks on evaluation code generation models: MBXP and Multilingual HumanEval, and MathQA-X. These datasets cover over 10 programming languages and are generated using a scalable conversion framework that transpiles prompts and test cases from the original Python datasets into the corresponding data in the target language. Using these benchmarks, we are able to assess the performance of code generation models in a multi-lingual fashion, and discovered generalization ability of language models on out-of-domain languages, advantages of multi-lingual models over mono-lingual, the ability of few-shot prompting to teach the model new languages, and zero-shot translation abilities even on mono-lingual settings. Furthermore, we use our code generation model to perform large-scale bootstrapping to obtain synthetic canonical solutions in several languages, which can be used for other code-related evaluations such as code insertion, robustness, or summarization tasks. Overall, our benchmarks represents a significant step towards a deeper understanding of language models' code generation abilities. We publicly release our code and datasets at https://github.com/amazon-research/mxeval.

1 INTRODUCTION

The paper addresses execution-based evaluation of code-generation abilities by introducing a scalable framework that converts Python datasets into multilingual benchmarks. These benchmarks enable multilingual assessment across many languages and support analyses of model generalization and related code tasks.

  • Evaluation motivation: Execution-based evaluation runs generated candidate code against test cases and measures success rate, allowing functionally correct implementations that differ from reference code.This contrasts with match-based evaluation, which compares generated code with reference source using n-gram metrics such as BLEU.
  • Benchmark contribution: The framework converts prompts and test statements from Python execution-based datasets into multiple target languages without requiring canonical solutions.The approach relies on function-completion evaluation and static analysis of basic programming problems.
  • Benchmark contribution: MBXP, Multilingual HumanEval, and MathQA-X extend MBPP, HumanEval, and MathQA into Java, JavaScript, TypeScript, Go, Ruby, Kotlin, PHP, C#, Scala, C++, Swift, and Perl.The release also includes a code package for execution in all supported languages.
  • Benchmark contribution: Large-scale bootstrapping supplies synthetic canonical solutions, enabling code insertion, translation, robustness, and summarization evaluations.The paper processes MBXP into t-MBXP, r-MBXP, i-MBXP, and s-MBXP variants.
  • Research scope: The evaluation studies in-domain and out-of-domain generation, few-shot prompting, zero-shot translation, prompt robustness, code summarization, and code insertion across model sizes.Models span approximately 100M to 10B parameters and are trained in multilingual and monolingual settings.

2 FINDING HIGHLIGHTS

The reported findings cover multilingual training, cross-language generalization, few-shot learning, and translation. Together, they show that multilingual and even monolingual models can generate or translate code beyond their intentionally trained languages, with prompting improving unfamiliar-language performance.

  • Multilingual versus monolingual models: At equal model size, multilingual models often outperform the best equivalently resourced monolingual models, especially at larger scales.The authors attribute this to sufficient model capacity and training across programming languages.
  • Out-of-domain generation: Models can generate syntactically correct code that passes unit tests in programming languages they were not intentionally trained on.The authors hypothesize that cross-language code spillover provides enough exposure for large models to learn embedded languages.
  • Multilingual versus monolingual models: Multilingual data can explain multilingual models’ superior performance by allowing knowledge from several languages to be combined when generating code in one language.The stated mechanism is data spillover across languages in natural data.
  • Few-shot prompting: Few-shot prompting significantly improves out-of-domain code generation by teaching knowledge of an unfamiliar language and reducing compilation or parsing errors.The error analysis identifies compilation and parsing errors as major sources of unfamiliar-language failures.
  • Zero-shot translation: Language models exhibit zero-shot code translation abilities, using reference code in one language to improve generation in another.The authors report that difficult problems can become easier when another language’s solution is available, including for monolingual models.
  • Additional capabilities: Multilingual models are more robust to prompt perturbations and better at summarizing code.

3 CONVERSION OF EXECUTION-BASED EVALUATION DATASETS

The conversion framework transforms execution-based function-completion datasets from Python into target programming languages by converting prompts, test statements, and represented data. It validates compatibility between generated prompts and tests, including inferred types for typed languages.

  • Dataset representation: Each function-completion problem contains a prompt, test statement, and canonical solution, with the prompt specifying a function signature and desired behavior.
  • Prompt conversion: The framework maps a Python function signature and docstring to a valid target-language prompt that models complete with a function body.
  • Type inference: For Java and other typed languages, input and output types are inferred by parsing test cases and selecting sufficiently general types across heterogeneous values.
  • Test conversion: Converted test cases are made compatible with the target prompt by referencing its defined class, method, and argument list.

4 MULTI-LINGUAL EVALUATION OF CODE GENERATION MODELS

The evaluation shows that multilingual models generalize to untrained languages, with larger models outperforming monolingual counterparts and translation or few-shot context improving code completion. These results are linked to cross-language data spillover and demonstrate that source-language solutions can support target-language generation.

  • Generalization to out-of-domain languages: Non-zero pass@k scores show that models generate valid code in languages absent from their training data.The authors attribute this out-of-domain capability to cross-language knowledge.
  • Multilingual versus monolingual models: At 2.7B and 13B parameters, multilingual models outperform the best equivalent monolingual models across evaluation languages.The gains are especially significant for out-of-domain languages such as PHP and remain noticeable for in-domain languages such as JavaScript and Java.
  • Knowledge spillover: Natural code co-occurrences help explain why multilingual models outperform monolingual models and why some monolingual models generate code in other languages.For example, JavaScript can appear inside Python strings, while Java and JavaScript data contain relatively little Python spillover.
  • Zero-shot code translation: Prepending a source-language solution lets models preserve its logic and semantics while producing syntactically valid code in the target language.The translation setup uses a source-language function as additional context for target-language completion.
  • Zero-shot code translation: Translation can make otherwise difficult tasks nearly solvable, reaching almost 100% pass@100 on MathQA in some settings.The result is reported for problems requiring complex reasoning but using simple arithmetic syntax.
  • Zero-shot code translation: A Java monolingual model improves from 20% to 36% pass@1 when given a Python solution, despite negligible standalone performance on Python.This shows that target-language knowledge can make a source-language reference useful even when source-language knowledge is limited.
  • Zero-shot code translation: Translation quality depends on the source-target language pair, because syntactically similar languages can confuse the model.Python can sometimes cause Ruby translation to produce Python code, while JavaScript performs better than Python for some Kotlin and Ruby targets.
  • Few-shot prompting: Few-shot prompting consistently improves execution accuracy, especially for out-of-domain languages, partly by reducing compilation, syntax, and parsing errors.The examples help disambiguate the target language, with little effect on in-domain evaluations.

5 RELATED WORK

Existing code-generation benchmarks are mainly match-based or concentrated on Python, while other execution-based datasets offer opportunities for multilingual conversion.

  • Existing code-generation benchmarks are primarily either match-based or focused mostly on Python.

6 DISCUSSION

The paper argues that multilingual models and parallel multilingual datasets broaden code-generation evaluation and support cross-language analysis. It also identifies unresolved boundaries around true generalization and the scope of benchmark coverage.

  • The released datasets provide resources for evaluating multiple aspects of code-generation ability.These include multilingual and monolingual models, out-of-domain performance, zero-shot translation, and multilingual code insertion.
  • Large multilingual models are presented as more effective than multiple monolingual models, with 2.7B-parameter models and above clearly outperforming all monolingual models in the controlled setting.The paper attributes this pattern to data spillover and notes that the required size may increase with more training languages.
  • Parallel datasets enable evaluation of code generation in target languages using canonical solutions from a source language.This supports research on cross-language generalization, translation factors, and modeling strategies.
  • The boundary between true generalization and generalization caused by data spillover remains unclear in out-of-domain evaluations.A constructed new language may still resemble languages seen during training through shared control-flow or data-structure features.
  • MBXP and related benchmarks measure understanding of basic tasks specified by natural-language descriptions and function signatures, using completion ability as a proxy for broader code-generation capability.

A.5 LIMITATIONS

The benchmarks are designed for basic, broadly transferable programming problems, but their converted tasks and guided prompts do not fully represent language-specific or real-world coding. Results should therefore be complemented with language-specific evaluation.

  • Dataset scope: MBXP conversion assumes each basic programming problem is relevant to every target language, although suitability varies by language.String manipulation may be more natural in Python or PHP than C++.
  • Dataset scope: MBXP scores may not match the distribution of natural language usage equally across programming languages.
  • Real-world coverage: MBXP omits language-specific functionality, such as JavaScript web development and C++ memory allocation.This limits how directly its conclusions transfer to coding performance in real-world software development.
  • Evaluation scope: The conversion applies to value-oriented tests and not complex integration tests involving specific packages or APIs.The authors suggest complementing MBXP with other language-specific evaluations when available.
  • Prompt design: Prompts strongly guide models toward a particular target language, measuring guided completion ability rather than unconstrained language-selection tendency.This format isolates desired-language generation from the tendency to choose a language in free-form prompts.
  • Prompt design: The benchmark still exhibits language-specific syntax patterns, such as Ruby list.select and Kotlin nums.filter.

C.1 SAMPLE GENERATION

Sample generation uses language-specific stopping criteria and execution-based testing, while truncation and translation errors remain practical sources of failure. Multi-function evaluation is explicitly outside the current scope.

  • Generation settings: Generation uses nucleus sampling with p = 0.95, a 1792-token input limit, and up to 256 generated tokens.Inputs exceeding the limit are truncated from the left, especially in few-shot or translation settings.
  • Stopping criteria: Stopping criteria differ by language, using closing braces for curly-brace languages and ‘end’ for Ruby.
  • Stopping criteria: Multi-function evaluation, where one generated function refers to later functions, is possible but out of scope.
  • Execution evaluation: The evaluation executes generated code natively across supported languages and computes unbiased pass@k from successful test-case execution.Examples include node file.js for JavaScript and exceptions when test cases fail.
  • Failure cases: Translation can fail on language-specific mathematical built-ins, such as Python max versus Java Math.max and differing logarithm signatures.The reported Java translation incorrectly retains max and overlooks the logarithm distinction.

E LANGUAGE “SPILLOVER” IN TRAINING DATA

The evaluation finds out-of-domain generalization and relates it to multilingual data spillover, where multiple programming languages co-occur in code files or projects. However, the training setup filters languages per file, limiting one proposed source.

  • Observed spillover: Models typically show out-of-domain generalization, which the authors hypothesize may reflect data spillover in cross-lingual code projects.
  • Spillover categories: Cross-lingual data can arise from explicit language embeddings within a file, including deep or shallow guest-language embeddings.
  • Spillover categories: A guest language may also appear inside a host language as strings, a pattern common in web code and code generators.
  • Spillover categories: Languages can coexist in separate files within one project, but this source does not apply because the models filter languages per file.
  • Spillover categories: Foreign Function Interfaces provide another cross-lingual combination through identifiers or function names in compiled bytecode.
  • Observed spillover: Pass@k trends remain consistent across k, and multilingual models clearly outperform monolingual models once model size is sufficiently large.

F.2 COMPREHENSIVE SAMPLING RESULTS

Comprehensive sampling evaluates few-shot prompting across languages and reports pass@k trends for monolingual and multilingual models at several scales. The supplied passages describe the evaluation setup and show representative language-specific prompts, but do not state figure outcomes beyond the general trend.

  • Sampling results: Figures 13–16 report pass@k trends for 125M, 672M, 2.7B, and 13B monolingual and multilingual models.Each figure compares in-domain and out-of-domain languages.
  • Few-shot prompting: Few-shot prompts prepend three correct functions from the corresponding language and provide an automatic performance gain of roughly 0.3%.The authors characterize this baseline gain as small relative to the observed gains.
  • Few-shot prompting: The few-shot examples are selected automatically from three correct samples for each evaluation language without tuning on those examples.
  • Prompt examples: The examples instantiate completion prompts in Python, JavaScript, PHP, Ruby, and Kotlin with language-specific signatures and syntax.
  • Few-shot prompting: Figure 17 reports performance differences caused by few-shot prompting using pass@1 with temperature 0.2.
  • Prompt examples: The Kotlin example uses a typed function signature and binary-search-style control flow for a right-insertion task.

G.1 EVALUATION RESULTS

Few-shot prompting substantially improves pass@1 on out-of-domain languages, including both mono-lingual and multi-lingual models.

  • 5.7% to 10.3%: few-shot prompting raises pass@1 for the 13B Python model evaluated on MBJP.
  • 5.9% to 12.2%: few-shot prompting raises pass@1 for the 13B multi-lingual model evaluated on MBJP.

H.1 TRANSLATION RESULTS FROM VARIOUS LANGUAGE SOURCES

Zero-shot translation generally improves code-generation performance, with especially large gains on out-of-domain languages and uneven effects across source languages. Larger models and target-language knowledge are associated with stronger translation results, while even mono-lingual models can translate from another language.

  • Translation improves performance in most settings over the baseline across multi-lingual and mono-lingual models.The evaluation uses Python, Java, and JavaScript as source languages, with canonical solutions for Java and JavaScript obtained through bootstrapping.
  • 5.9% to 15.9%: translation raises pass@1 for the 13B multi-lingual model on Ruby when JavaScript is the source language.
  • Source languages have unequal effects: Java is strongest for MBPHP, while JavaScript is strongest for MBRBP and MBKP in the reported settings.The best source language is not necessarily the one closest in syntax, which can sometimes confuse generation into incorrect syntax.
  • Larger models typically perform better in both normal code completion and translation settings.
  • Target-language knowledge is likely more important than source-language knowledge for translation performance.For MBJSP, translation raises pass@1 from 13.8% to 30.7% for the Python model and from 23.3% to 32.8% for the JavaScript model using Python as source.

I.1 TEST CASE ERROR VERSUS NON-ASSERTION ERROR

The analysis separates test-case assertion errors from other failures such as compilation, parsing, and runtime errors, then compares how prompting affects these categories.

  • Non-assertion errors include compilation, parsing, and runtime failures unrelated to test cases.The reported fraction uses temperature 0.2 and 30 samples per problem.
  • Few-shot prompting lowers non-assertion errors for out-of-domain languages, indicating more precise language-specific syntax generation.The effect is small on in-domain languages, where models are already fluent.
  • Figure 22 reports non-assertion-error percentages for in-domain and out-of-domain datasets, while Figure 23 reports assertion-error percentages.
  • Robustness evaluation applies paraphrase by back translation, character-case changes, and synonym substitutions to code-completion prompts.

J.2 EVALUATION RESULTS

The paper evaluates robustness and code insertion across languages. Perturbations reduce pass@1 consistently, multilingual models are somewhat more robust than mono-lingual models, and right context improves insertion accuracy.

  • Robustness: 8.02% to 9.40%: average pass@1 drops on perturbed datasets across model sizes, with the largest drop for 2.7B models.The reported averages decline from 2.26 to 2.07 for 125M, 6.40 to 5.87 for 672M, 9.20 to 8.33 for 2.7B, and 12.63 to 11.62 for larger models.
  • Robustness: 7.63% versus 9.81%: multilingual models have a smaller average pass@1 drop than mono-lingual models on in-domain perturbations.
  • Robustness: 7.98% versus 8.97%: multilingual models have a smaller average pass@1 drop than mono-lingual models on out-of-domain perturbations.
  • Code insertion: Insertion uses a left-context, mask-token, and right-context sequence, with generation stopped at an end-of-mask token or another stopping condition.
  • Code insertion: Right context significantly boosts insertion performance across all evaluated languages.Accuracy increases as more lines of right context are added, and qualitative examples show consistency with that context.
  • Code insertion: Right-context insertion can produce code consistent with surrounding code, unlike left-to-right generation that does not use the right context.

L.2 EVALUATION RESULTS

Evaluation results show that larger models perform better, multilingual models generally outperform comparable monolingual models, and few-shot prompting improves out-of-domain generation and translation.

  • 13B, 672M, and 125M models achieve Python BLEU-4 scores of 6.07, 5.59, and 3.20 in zero-shot settings, versus 34.10, 24.72, and 20.75 with few-shot prompting.
  • Multilingual models outperform monolingual models, while few-shot examples help monolingual models generalize to other languages.
  • Zero-shot examples illustrate code summarization and generation tasks across Python, JavaScript, and Java.
  • BLOOM outperforms OPT, consistent with BLOOM’s code-inclusive pretraining data across languages including PHP, Java, Python, JavaScript, and Ruby.
  • CodeGen-mono 16B improves over CodeGen-multi by 6% on JavaScript and 8% on PHP.
  • Few-shot prompting raises CodeGen-multi accuracy on Ruby from 3.5% to 16.46%, demonstrating substantial gains on an out-of-domain language.

N.2 OBSERVATIONS ON VALIDATION LOSSES VERSUS PERFORMANCE

Validation loss and execution performance favor different model configurations: monolingual models fit their data better, while sufficiently large multilingual models achieve higher pass@k.

  • Validation loss roughly follows a log-linear relationship with model size.
  • Monolingual models consistently achieve lower validation loss than multilingual models at comparable sizes.
  • The authors attribute multilingual models’ execution advantage to knowledge sharing across programming languages despite their broader fitting burden.
  • Multilingual models outperform monolingual models on execution pass@k, especially beyond 672M parameters.
  • The conversion framework creates multilingual function-completion datasets by transforming prompts and test statements without requiring canonical solutions.
  • The framework uses static analysis, type mapping, target-language object construction, assertions, and deep comparisons to generate and validate test statements.
  • The converted benchmarks support multiple languages and additional evaluations through synthetic canonical solutions generated and filtered with converted tests.

O.2 POTENTIAL USE OF TRANSCODER FOR DATASET CONSTRUCTION

Transcoder is unsuitable as the primary conversion method because it supports few language pairs and introduces type and formatting errors, motivating the proposed static-analysis framework.

  • Transcoder supports only Python, Java, and C++ in the described setup, limiting conversion to 10+ languages.
  • Transcoder introduces type-inference errors that can produce false negatives, such as predicting int when the return type should be boolean.
  • The proposed framework avoids these translation-model type errors through static analysis and deterministic object mapping.
  • Examples show translated function names remaining in snake case or changing substantially across Java and C++ outputs.
  • Nested Python lists can be incorrectly mapped to strings in C++ or flat string lists in Java.
  • Synthetic canonical solutions rely on test cases as necessary and sufficient correctness checks, an assumption that may fail when source tests are incomplete.
  • The authors leave synthetic generation of additional tests from canonical solutions for future work because canonical solutions may themselves be only partially correct.
  • Manual review found that nearly all identified issues were attributable to the source dataset rather than the conversion process.
Loading 2210.14868v3…