PaperlayerView exact revision on arXiv

Source-linked AI summary

HumanEval-XL: A Multilingual Code Generation Benchmark for Cross-lingual Natural Language Generalization

Qiwei Peng, Yekun Chai, Xuhong Li

arXiv:2402.16694v2cs.CLcs.PLcs.SE

TL;DR

Existing code-generation benchmarks mostly focus on English prompts or provide limited natural-language coverage, leaving multilingual NL-to-code generalization insufficiently assessed. HumanEval-XL supplies parallel problems spanning 23 natural languages and 12 programming languages. Its evaluation finds that current LLMs still struggle to capture equivalent meanings across languages.

  • Problem

    Existing benchmarks largely focus on English prompts or limited natural-language coverage, leaving multilingual NL-to-code generalization insufficiently evaluated.

    Source: §1 · p. 1

  • Method

    HumanEval-XL creates a parallel code-generation benchmark connecting 23 natural languages and 12 programming languages.

    Source: §1 · p. 1

  • Results

    Across different-language prompts expressing the same meaning, current LLMs struggle to capture equivalent semantics.

    Source: §4.3 · p. 5

  • Takeaways & Limitations

    The study identifies effective cross-lingual natural-language generalization as a persistent challenge and key area for future research.

    Source: §5 · p. 5

Abstract

from arXiv · show

Large language models (LLMs) have made significant progress in generating codes from textual prompts. However, existing benchmarks have mainly concentrated on translating English prompts to multilingual codes or have been constrained to very limited natural languages (NLs). These benchmarks have overlooked the vast landscape of massively multilingual NL to multilingual code, leaving a critical gap in the evaluation of multilingual LLMs. In response, we introduce HumanEval-XL, a massively multilingual code generation benchmark specifically crafted to address this deficiency. HumanEval-XL establishes connections between 23 NLs and 12 programming languages (PLs), and comprises of a collection of 22,080 prompts with an average of 8.33 test cases. By ensuring parallel data across multiple NLs and PLs, HumanEval-XL offers a comprehensive evaluation platform for multilingual LLMs, allowing the assessment of the understanding of different NLs. Our work serves as a pioneering step towards filling the void in evaluating NL generalization in the area of multilingual code generation. We make our evaluation code and data publicly available at \url{https://github.com/FloatAI/humaneval-xl}.

1. Introduction

Existing code-generation benchmarks largely center on English prompts or offer limited natural-language coverage, leaving cross-lingual NL-to-code generalization difficult to assess. HumanEval-XL provides parallel problems across 23 NLs and 12 PLs, and experiments find that model scale and code pre-training matter while cross-lingual semantic understanding remains challenging.

1. Introduction

  • Existing benchmarks focus mainly on English-centric code generation or cover only a limited range of natural languages, leaving multilingual NL-to-code generalization insufficiently assessed.The gap motivates evaluation across a broader linguistic landscape.Source: §1 · p. 1
  • HumanEval-XL connects parallel coding problems across 23 natural languages and 12 programming languages for evaluating cross-lingual generalization.The parallel alignment supports comparisons across languages.Source: §1 · p. 1
  • Experiments find that larger models improve multilingual code generation, specialized code pre-training matters, and current LLMs struggle to capture equivalent meanings across languages.The evaluation covers CodeT5+, CodeGen2, GPT-3.5, and GPT-4 at various parameter scales.Source: §1 · p. 1

2. Related work

Prior code-generation benchmarks largely emphasized English-to-Python tasks, with some extending programming-language coverage or adding a small number of natural languages. HumanEval-XL instead provides parallel coverage across 23 natural languages and 12 programming languages.

2. Related work

  • Earlier benchmarks predominantly evaluated English-centric Python generation, while some extended HumanEval to multiple programming languages [1, 3].Examples of English-Python benchmarks include HumanEval [1], MBPP [2], and APPS [3].Source: §2 · p. 1
  • ODEX covered four natural languages with Python but averaged only 1.8 test cases; HumanEval-XL spans 23 natural languages and 12 programming languages in parallel.HumanEval-XL contains 22,080 prompts averaging 8.33 test cases.Source: §2 · pp. 1–2
  • Despite multilingual LLM pre-training, existing code-generation benchmarks lacked multilingual code generation for assessing cross-lingual natural-language generalization.The cited examples include ERNIE-Code, BLOOM, GPT-3.5, and GPT-4.Source: §2 · p. 2

3. HumanEval-XL

HumanEval-XL is designed as an accessible benchmark for cross-lingual natural-language transfer across programming languages. Its construction extends English prompts through translation, back-translation, quality checks, and filtering to produce a parallel multilingual dataset.

3.1. Design Principles

  • The benchmark targets accessible evaluation of cross-lingual natural-language transfer across programming languages.Its design principles emphasize task complexity, language diversity, and permissively licensed data for research use.Source: §3.1 · p. 2
  • Code generation is selected as a challenging task, and pass@k measures success on executed test cases.The metric is used to assess code-generation proficiency.Source: §3.1 · p. 2
  • The benchmark aims for broad natural- and programming-language diversity, with parallel cases across natural languages to support unbiased performance comparisons.Parallel alignment is intended to enable fair comparisons across linguistic settings.Source: §3.1 · p. 2

3.2. Dataset Construction

  • English prompt text is extracted, translated into 23 languages with GPT-4, and back-translated into English for quality checking.This is part of the four-stage construction process.Source: §3.2 · p. 2, Figure 1 · p. 3
  • Back-translations are compared with the original using BERTScore; scores below 0.95 trigger up to three retranslation attempts before the example is discarded.Retained translations then undergo heuristic checks and sampled manual review.Source: §3.2 · p. 2, Figure 1 · p. 3
  • Heuristic checks and manual review are applied before filtering produces 80 parallel coding problems across 12 programming languages and 23 natural languages.The resulting benchmark contains 22,080 coding problems.Source: §3.2 · p. 2
  • The 23 natural languages span 11 language families, while the 12 programming languages match those in Multilingual HumanEval.The programming-language set includes Python, Java, Go, Kotlin, PHP, Ruby, Scala, JavaScript, C#, Perl, Swift, and TypeScript.Source: §3.2 · p. 3, Table 1 · p. 3

4. Experiments

The experiments compare code-specialized and general-purpose LLMs on HumanEval-XL, finding substantial cross-language performance differences and persistent challenges in NL generalization. Analyses also show performance patterns associated with model family, language-resource group, and language family.

4.1. Experimental Setup

  • The experiments use CodeT5+, CodeGen2, GPT-3.5, and GPT-4, report pass@1, and show the largest CodeT5+ and CodeGen2 variants in the main text.Full results for varied model sizes are in the appendix; the authors report that trends remain consistent across scales.Source: §4.1 · p. 3

4.2. Results

  • Figure 2 evaluates pass@1; GPT-4 consistently outperforms all other models across programming languages and natural languages except C#, while GPT-3.5 trails CodeGen2-16B in every programming language except Python.Source: Figure 2 · p. 4, §4.2 · p. 4
  • CodeGen2-3.7B and CodeGen2-16B markedly outperform CodeT5+ (2B), which struggles across most scenarios; the authors suggest architecture may explain the difference.CodeT5+ uses an encoder-decoder architecture, whereas the other models use decoder-only architectures; the authors suggest the former hinders code generation without further instruction-tuning.Source: §4.2 · p. 4
  • Python is the most effectively tackled PL, while models struggle with Scala and Go.Source: §4.2 · p. 4

4.3. Analysis and Discussion

  • On Python, same-family models’ pass@1 results across language-resource levels correlate at 0.8 for CodeGen2-3.7B versus 16B and 0.87 for GPT-3.5 versus GPT-4.Table 3 reports pass@1 averages and standard deviations across NL resource groups.Source: §4.3 · p. 5, Table 3 · p. 5
  • Resource-group pass@1 differences are small but consistent, with class 3 lower across models; several language families also generally yield lower results.The lower-result families reported are Afro-Asiatic, Indo-European (Greek), Indo-European (Iranian), and Turkic; the authors say current LLMs struggle to capture equivalent meaning across languages.Source: §4.3 · p. 5, Table 4 · p. 5

5. Conclusion

The conclusion presents HumanEval-XL as a benchmark for cross-lingual NL generalization in code generation, built through an iterative process inspired by back-translation. It emphasizes that effective cross-lingual generalization remains a challenge.

  • HumanEval-XL is an iterative, back-translation-inspired benchmark spanning 12 PLs and 23 NLs.Source: §5 · p. 5
  • The benchmark is designed to assess cross-lingual NL generalization for LLMs in code generation.Source: §5 · p. 5
  • The conclusion identifies effective cross-lingual NL generalization as a persistent challenge and future research area.Source: §5 · p. 5

A. Experiment Settings

The experiment settings cover specialized code-generation models and general-purpose GPT models at multiple scales. Sampling is standardized across evaluated models, and generation stops after the first function.

B. Comprehensive Experiment Results

The appendix presents detailed model results across programming languages for comparing model performance; its tables report pass@1 across 23 natural languages.

B. Comprehensive Experiment Results

  • Appendix results cover CodeGen2 models at 1B, 3.7B, 7B, and 16B; CodeT5+ at 220M, 770M, and 2B; and GPT-3.5 and GPT-4.The appendix describes these as full experimental results across models of varying sizes.Source: B. Comprehensive Experiment Results · p. 8
  • Tables 5–9 report pass@1 results across 23 natural languages for Python, Java, JavaScript, C#, and Go.The tables provide results for different models.Source: Table 5 · p. 9, Table 6 · p. 9, Table 7 · p. 9
  • Tables 10–15 report pass@1 results across 23 natural languages for Kotlin, Perl, PHP, Ruby, Scala, and Swift.The tables provide results for different models.Source: Table 10 · p. 10, Table 11 · p. 11, Table 12 · p. 11
  • Table 16 reports pass@1 results for different models on TypeScript across 23 natural languages.Source: Table 16 · p. 12

Related papers

Loading 2402.16694v2…