PaperlayerView exact revision on arXiv

Source-linked AI summary

CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Benchmarking on HumanEval-X

Qinkai Zheng, Xiao Xia, Xu Zou, Yuxiao Dong, Shan Wang, Yufei Xue, Zihan Wang, Lei Shen, Andi Wang, Yang Li, Teng Su, Zhilin Yang, Jie Tang

arXiv:2303.17568v2cs.LGcs.AIcs.SE

TL;DR

The paper addresses the need for multilingual code generation and functional-correctness evaluation beyond Python-only benchmarks. It presents CodeGeeX, a 13B model trained across 23 languages, together with HumanEval-X and IDE extensions; CodeGeeX consistently outperforms comparable multilingual baselines, while its extensions improve reported coding efficiency for 83.4% of surveyed users.

  • Problem

    Existing code benchmarks are largely Python-only or use string similarity, limiting systematic multilingual evaluation of functional correctness.

  • Method

    The paper trains and releases CodeGeeX, a 13B multilingual model trained on more than 850 billion tokens across 23 programming languages, and develops HumanEval-X.

  • Results

    CodeGeeX consistently outperforms open-sourced multilingual baselines of similar scale on code generation and translation, while 83.4% of surveyed users report improved coding efficiency.

  • Takeaways & Limitations

    The open model, benchmark, and extensions make multilingual code generation and translation available for research, development, and programming workflows.

  • Takeaways & Limitations

    Pass rates vary substantially across languages, so CodeGeeX does not reliably solve the same problem in different languages.

Abstract

from arXiv · show

Large pre-trained code generation models, such as OpenAI Codex, can generate syntax- and function-correct code, making the coding of programmers more productive and our pursuit of artificial general intelligence closer. In this paper, we introduce CodeGeeX, a multilingual model with 13 billion parameters for code generation. CodeGeeX is pre-trained on 850 billion tokens of 23 programming languages as of June 2022. Our extensive experiments suggest that CodeGeeX outperforms multilingual code models of similar scale for both the tasks of code generation and translation on HumanEval-X. Building upon HumanEval (Python only), we develop the HumanEval-X benchmark for evaluating multilingual models by hand-writing the solutions in C++, Java, JavaScript, and Go. In addition, we build CodeGeeX-based extensions on Visual Studio Code, JetBrains, and Cloud Studio, generating 4.7 billion tokens for tens of thousands of active users per week. Our user study demonstrates that CodeGeeX can help to increase coding efficiency for 83.4% of its users. Finally, CodeGeeX is publicly accessible and in Sep. 2022, we open-sourced its code, model weights (the version of 850B tokens), API, extensions, and HumanEval-X at https://github.com/THUDM/CodeGeeX.

1 Introduction

CodeGeeX is introduced as an open multilingual code model and evaluation ecosystem addressing automatic program writing across languages. The paper reports broad task support, a new functional-correctness benchmark, consistent multilingual baseline advantages, and substantial user adoption.

  • Motivation: Automatic program writing asks whether machines can generate executable programs from descriptions of human intent.The paper situates this question within longstanding efforts spanning deductive synthesis, program synthesis, and neural code generation.
  • CodeGeeX: CodeGeeX is a 13B-parameter model trained on more than 850 billion tokens across 23 programming languages.It supports code generation, completion, translation, and explanation, and is open-sourced for research and development.
  • Deployment: 83.4% of surveyed users report improved coding efficiency from CodeGeeX extensions across supported IDEs.The extensions serve tens of thousands of daily active users and generate 4.7 billion tokens per week.
  • HumanEval-X: HumanEval-X provides 820 hand-written problem-solution pairs across C++, Java, JavaScript, Go, and Python for generation and translation evaluation.It addresses the single-language and string-similarity limitations of existing benchmarks by supporting functional-correctness evaluation.
  • Evaluation: CodeGeeX consistently outperforms multilingual code-generation models of similar scale on code-generation and translation tasks.The comparison includes CodeGen-16B, GPT-NeoX-20B, InCode-6.7B, and GPT-J-6B.

2 The CodeGeeX Model

CodeGeeX is a 13B multilingual code generation model built with a decoder-only Transformer, trained on large-scale multilingual code data, and optimized for efficient training and inference.

  • Model and training overview: CodeGeeX is a 13B model pretrained on more than 850 billion tokens from 23 programming languages.Training used 1,536 Ascend 910 AI processors over two months.
  • Architecture: Its core architecture is a 39-layer decoder-only Transformer with multi-head self-attention, MLP layers, normalization, residual connections, and a top query layer.The model performs autoregressive programming-language modeling by predicting the next token from preceding tokens.
  • Pre-training objective: The generative pre-training objective predicts each next code token and compares the predicted probability distribution with the ground-truth token distribution using cross-entropy loss.The model outputs P(x_n+1|x_1, x_2, ..., x_n, Θ) over the vocabulary.
  • Pre-training setup: CodeGeeX processes multilingual code with GPT-2 BPE tokenization, language-specific segment tags, learnable word and positional embeddings, and a 2,048-token maximum sequence length.The final vocabulary contains 52,224 tokens, while the hidden size is 5,120.
  • Training system: Parallelism, ZeRO-2, kernel fusion, and Auto Tune optimization support large-scale training on Ascend 910 hardware.The configuration uses 8-way model parallelism and 192-way data parallelism with a global batch size of 3,072.
  • Training system: 3× higher per-processor efficiency and 224% higher overall token throughput were achieved after training-efficiency optimization.The throughput comparison is against the non-optimized implementation using 1,536 processors.

3 The HumanEval-X Benchmark

HumanEval-X extends Python-only code benchmarks into a multilingual evaluation of functional correctness for code generation and translation. It covers five languages, manually rewritten problem materials, and budget allocation across languages.

  • 3.1 HumanEval-X: A Multilingual Benchmark: 820 problem-solution pairs cover 164 problems across C++, Java, JavaScript, Go, and Python.Each problem supports both code generation and code translation.
  • 3.1 HumanEval-X: A Multilingual Benchmark: HumanEval-X addresses the lack of multilingual functional-correctness evaluation in Python-only benchmarks and string-similarity-based multilingual datasets.Its tests verify whether generated code passes language-specific test cases.
  • 3.2 HumanEval-X: Tasks: Generation uses declarations and docstrings to produce implementations, whereas translation uses source-language declarations and solutions to produce target-language solutions.The target-language declaration constrains function names and variable types during translation.
  • 3.2 HumanEval-X: Tasks: pass@k estimates the probability that at least one of k generated samples passes all test cases, using n = 200 generations per problem.The benchmark averages single-problem pass@k values across test-set problems.
  • 3.2 HumanEval-X: Tasks: Multilingual pass@k distributes a fixed generation budget across candidate languages to increase sampling diversity and solve rate.The allocation assigns n_i generations to language i under an assignment π.

4 Evaluating CodeGeeX on HumanEval-X

CodeGeeX is evaluated against multilingual baselines on HumanEval-X for code generation and translation. It achieves the best average generation performance, leads many translation pairs, and benefits from multilingual budget allocation, while errors remain primarily logical.

  • 4.2 Results of Code Generation and Translation: CodeGeeX achieves the best average code-generation performance, improving over CodeGen-Multi-16B by 0.37%∼1.67% across pass@1, pass@10, and pass@100.It outperforms smaller InCoder-6.7B and CodeGen-Multi-6B models and is competitive with CodeGen-Multi-16B.
  • 4.2 Results of Code Generation and Translation: CodeGeeX-13B-FT performs best on pass@100 for 11 of 20 translation pairs, while CodeGen-Multi-16B leads on 7.CodeGeeX favors translations into Python and C++, whereas CodeGen-Multi-16B performs better for JavaScript and Go targets.
  • 4.2 Results of Code Generation and Translation: Wrong answers are the most common error category across languages, with ratios from 0.44 to 0.75 except for Go.This pattern indicates that current code-generation models more often fail in code logic than in semantics.
  • 4.3 The Multilingual Pre-Training Helps Problem Solving: Uniform and Weighted multilingual budget allocation outperform Best Single by promoting more diverse generations, with Weighted slightly better.CodeGeeX shows a decent advantage over other baselines under both allocation strategies.
  • 4.3 The Multilingual Pre-Training Helps Problem Solving: Translation performance in opposite directions is usually negatively correlated, indicating asymmetry in multilingual models’ source- and target-language focus.The paper attributes this asymmetry partly to differing language distributions in training data.

5 The CodeGeeX Tools and Users

CodeGeeX extensions bring generation, completion, translation, and explanation into several IDEs. Tens of thousands of users generate billions of tokens weekly, and 83.4% report improved or slightly increased coding efficiency.

  • 5 The CodeGeeX Tools and Users: CodeGeeX extensions for Visual Studio Code, JetBrains, and Cloud Studio support code generation, completion, translation, explanation, and customizable prompting.The tools are intended to support programming tasks in real time.
  • 5 The CodeGeeX Tools and Users: 4.7+ billion tokens are generated per week by CodeGeeX, serving tens of thousands of users with 250+ average API calls per active user per weekday.The reported token volume had been steadily growing since release.
  • 5 The CodeGeeX Tools and Users: The user-experience survey covers 168 users across developer, engineering, student, researcher, and other programmer roles.Satisfaction is scored from 0 to 5 across ease of use, reliability, features, visuals, and speed.
  • 5 The CodeGeeX Tools and Users: 83.4% of surveyed users report that CodeGeeX improves or slightly increases their coding efficiency.The survey links stronger perceived helpfulness to mainstream languages that also appear more frequently in the training data.

6 Conclusion

The paper introduces CodeGeeX and HumanEval-X to address multilingual code generation, reporting consistent baseline outperformance while identifying unresolved challenges in multilingual capacity, generality, and few-shot learning.

  • CodeGeeX is a 13B pre-trained model for 23 languages, paired with HumanEval-X to fill the gap in multilingual code generation.
  • CodeGeeX consistently outperforms open-sourced multilingual baselines of the same scale on code generation and translation tasks.
  • Model capacity is essential for multilingual programming ability, while extracting essential programming knowledge remains a research challenge.
  • CodeGeeX solves problems across languages but has varying pass rates and lacks strong generality, limiting reliable reasoning ability.
  • Exploring few-shot prompting, including chain-of-thought prompting, is identified as a future direction for code models.

A.1 Statistics of Code Corpus

The appendix identifies the code corpus as the pre-training data source and provides a table summarizing its composition.

  • Table 8 summarizes the composition of CodeGeeX’s code corpus.
  • The code corpus is presented as the corpus used for pre-training CodeGeeX.
  • Table 8 is titled “Composition of our code corpus for pre-training.”

A.2 Tokenization of CodeGeeX

CodeGeeX tokenizes code and text into dictionary-indexed integer sequences, with special handling for whitespace lengths.

  • The tokenizer first separates a code snippet into token pieces.
  • Each token is mapped to an integer using its ID in a predefined dictionary.
  • Whitespace sequences such as 4 or 8 spaces are concatenated into special tokens, and token text can mark whitespace.
  • After tokenization, code snippets and text descriptions become vectors of integers.

A.2.1 Details of Budget Allocation Strategies

The appendix describes budget-allocation strategies for multilingual training and identifies figures covering tokenization, training loss, and HumanEval-X performance over iterations.

  • Best Single selects one language, Uniform distributes the budget evenly, and Weighted allocates budgets according to training-corpus proportions.
  • Table 9 gives detailed budget assignments for allocation strategies using a total budget of k = 100.
  • Figure 12 illustrates CodeGeeX tokenization, including whitespace markers and concatenated-whitespace special tokens.
  • Figure 13 presents CodeGeeX’s training loss, while Figure 14 presents HumanEval-X pass rate versus iteration.

A.3 Evaluation on HumanEval-X (Additional)

HumanEval-X pass rates continue improving during training, while different code-generation models show language-dependent pass-rate distributions.

  • Pass rates in HumanEval-X continue improving across languages as training iterations increase.The accompanying training loss decreases steadily.
  • Different code-generation models exhibit varied pass-rate distributions across programming languages.
  • HumanEval-X pass rates vary by programming language at both t = 0.2 and t = 0.8 with p = 0.95.Figure 15 covers InCoder-6.7B, CodeGen-Multi-6B, and CodeGen-Multi-16B.

A.4 Evaluation on Other Benchmarks

On HumanEval and several downstream benchmarks, CodeGeeX shows strong multilingual code-generation, summarization, and translation performance, although model capacity and prompting affect results. The appendix also provides qualitative multilingual generation and translation examples.

  • A.4.1 Evaluation on HumanEval: 60% pass@100 places CodeGeeX second among multilingual models on HumanEval, behind PaLMCoder-540B.Monolingual models outperform multilingual ones by a large margin, suggesting multilingual models need greater capacity to master different languages.
  • A.4.2 Evaluation on MBPP: More prompt shots harm CodeGeeX performance on MBPP, while larger-capacity PaLM and Codex outperform it by a large margin.The authors attribute the few-shot effect to smaller models having insufficient reasoning ability to benefit from few-shot prompts.
  • A.4.3 Evaluation on CodeXGLUE: 20.63 average BLEU makes CodeGeeX the best model on CodeXGLUE code summarization across six languages.After excluding Ruby, which is absent from pretraining, CodeGeeX exceeds DistillCodeT5 by 1.88 average BLEU.
  • A.4.4 Evaluation on XLCoST: CodeGeeX beats all XLCoST baselines on nearly every language pair, averaging +4.10 CodeBLEU at program level and +1.99 at snippet level.Exceptions are PHP→Python at program level, plus PHP→Python and C++→Python at snippet level.
  • A.4.5 Qualitative Examples: Examples show CodeGeeX generating solutions, translating solutions, and responding to prompts in Chinese, French, Russian, and Japanese.
Loading 2303.17568v2…