Source-linked AI summary
A Systematic Evaluation of Large Language Models of Code
Frank F. Xu, Uri Alon, Graham Neubig, Vincent J. Hellendoorn
TL;DR
The paper addresses limited access to strong code language models and uncertainty about their design choices by systematically evaluating existing systems and releasing PolyCoder. PolyCoder is an open-source multilingual code model, and it achieves lower perplexity than all compared models, including Codex, on C.
Problem
Strong code language models are inaccessible, leaving model and data design choices unclear and limiting research and adaptation.
Method
The paper systematically evaluates existing code models and releases PolyCoder models trained exclusively on code from 12 programming languages.
Results
In C, PolyCoder achieves lower perplexity than all models, including Codex.
Takeaways & Limitations
The open-source PolyCoder release provides a multilingual code model for future research and applications.
Takeaways & Limitations
GPT-NeoX-20B was excluded from HumanEval because its size exceeded the available computational budget.
Abstract
from arXiv · showhide
Large language models (LMs) of code have recently shown tremendous promise in completing code and synthesizing code from natural language descriptions. However, the current state-of-the-art code LMs (e.g., Codex (Chen et al., 2021)) are not publicly available, leaving many questions about their model and data design decisions. We aim to fill in some of these blanks through a systematic evaluation of the largest existing models: Codex, GPT-J, GPT-Neo, GPT-NeoX-20B, and CodeParrot, across various programming languages. Although Codex itself is not open-source, we find that existing open-source models do achieve close results in some programming languages, although targeted mainly for natural language modeling. We further identify an important missing piece in the form of a large open-source model trained exclusively on a multi-lingual corpus of code. We release a new model, PolyCoder, with 2.7B parameters based on the GPT-2 architecture, which was trained on 249GB of code across 12 programming languages on a single machine. In the C programming language, PolyCoder outperforms all models including Codex. Our trained models are open-source and publicly available at https://github.com/VHellendoorn/Code-LMs, which enables future research and application in this area.
1 INTRODUCTION
Code language models support code completion and natural-language-to-code synthesis, but the strongest systems are inaccessible and key design choices remain unclear. The paper systematically evaluates existing models and releases PolyCoder to address the missing open-source multilingual code model.
- Motivation: The strongest code language models are not publicly available, limiting external application, adaptation, and research on their internals.Codex exposes outputs through a non-free black-box API, while its weights and training data remain unavailable.
- Motivation: Existing models differ substantially in size and training data, but the effects of these modeling and training choices remain unclear.Public models include systems trained on mixed natural-language and code corpora, whereas CodeParrot was trained solely on Python code.
- Approach: The paper systematically compares Codex, GPT-J, GPT-Neo, GPT-NeoX, and CodeParrot across programming languages and evaluation settings.The study also examines scaling with model size and training steps and evaluates generation temperature effects.
- Contribution: PolyCoder supplies the missing large open-source model trained exclusively on code from multiple programming languages.The authors provide three models ranging from 160M to 2.7B parameters under the PolyCoder name.
- Significance: The authors position the release as a step toward democratizing large language models of code.They connect this goal to concerns that state-of-the-art training and testing are restricted to a small number of large technology companies.
2 RELATED WORK
The paper situates code modeling within three pretraining paradigms and contrasts models by architecture, training corpus, size, and availability. Existing systems include left-to-right, masked, and encoder-decoder models trained on code-only or mixed natural-language-and-code data.
- Model landscape: The paper compares models whose sizes and availability differ, including proprietary systems with unknown training datasets.Figure 1 presents existing code models according to size and open-source availability.
- Pretraining methods: Left-to-right language models predict each token from preceding tokens and are highly useful for program generation such as code completion.Examples include CodeParrot, GPT-Neo, GPT-J, Codex, GPT-NeoX, and Google’s large model.
- Pretraining methods: Masked language models predict masked text pieces from surrounding context and provide bidirectional representations for downstream tasks such as classification.CodeBERT and CuBERT are examples of this paradigm.
- Pretraining methods: Encoder-decoder models encode an input sequence and decode an output sequence conditioned on that input.Masked span prediction and denoising sequence reconstruction support sequence-to-sequence tasks such as code commenting and natural-language-to-code generation.
- Training data: Code models use either code-only corpora or mixed corpora containing natural-language text and source code.CodeParrot and CodeT5 use GitHub code, while GPT-Neo and GPT-J use the Pile, which blends natural-language sources with code.
3 EVALUATION SETTINGS
The evaluation combines extrinsic code generation with intrinsic language modeling. Models are tested on HumanEval for natural-language-to-code synthesis and on perplexity over unseen repositories for each language.
- Evaluation overview: All models are evaluated using both extrinsic and intrinsic benchmarks.The two evaluations measure downstream generation and intrinsic modeling performance, respectively.
- Extrinsic evaluation: HumanEval measures whether generated code correctly solves 164 natural-language-described programming prompts.Each prompt includes code comments, function definitions, names, and test cases.
- Intrinsic evaluation: Intrinsic evaluation computes perplexity for each programming language on an unseen set of GitHub repositories.Repositories appearing in the GitHub portion of the Pile are removed for GPT-Neo and GPT-J to reduce training-to-test leakage.
- Intrinsic evaluation: Codex is evaluated through OpenAI’s code-davinci-001 API, but its unknown training data prevents test-set leakage from being ruled out.This creates a comparability caveat for Codex’s intrinsic evaluation.
4 COMPARED MODELS
The paper compares existing code language models across training data, preprocessing, architectures, and scale, while introducing PolyCoder as an open-source multilingual code model. PolyCoder uses filtered and deduplicated GitHub code with GPT-2-based models trained at three parameter sizes.
- Existing models: The study compares Codex, GPT-Neo, GPT-J, GPT-NeoX, and CodeParrot, alongside PolyCoder as an open-source model trained on code from 12 programming languages.The compared open-source models differ in whether they train on mixed natural-language-and-code corpora or code from a single language.
- PolyCoder’s data: PolyCoder’s raw corpus contains 631GB and 38.9M files collected from popular GitHub repositories across 12 programming languages.Repositories were selected with at least 50 stars, with collection capped at about 25K repositories per language.
- PolyCoder’s data: Filtering very large and very short files plus content deduplication reduced the corpus by 38% in files and 61% in dataset size.The reductions were roughly comparable to CodeParrot’s reported 70% dataset-size reduction.
- PolyCoder’s data: The resulting corpus contains less Python code than Codex or CodeParrot while covering many different programming languages.The dataset statistics report only 16G of Python code for PolyCoder.
- PolyCoder’s training: PolyCoder uses GPT-2 architecture at 2.7B, 400M, and 160M parameters, trained for up to 150K steps on a single machine with eight Nvidia RTX 8000 GPUs.The largest model has a 2048-token context window and required about six weeks of wall time to train.
5 RESULTS
Across models and languages, performance depends on scale, data composition, and generation temperature. PolyCoder leads in C but generally trails Codex and similarly sized GPT-Neo on broader evaluations.
- PolyCoder is worse than similarly sized GPT-Neo and the smaller Codex 300M overall, but stronger than CodeParrot.PolyCoder’s code-only training and limited exposure to Python tokens may contribute to this result.
- Scaling Effect: HumanEval performance increases linearly with model size on a log scale for Codex, PolyCoder, and GPT-Neo/J, while CodeParrot appears to saturate.Codex remains significantly better than open-source models across parameter counts.
- Temperature Effect: Pass@1 favors lower temperatures, Pass@100 favors higher temperatures, and Pass@10 favors intermediate temperatures.The authors therefore tune temperature separately for different generation scenarios.
- PolyCoder outperforms Codex and all other models in C, and exceeds GPT-Neo 2.7B in C, JavaScript, Rust, Scala, and TypeScript.
- GPT-J’s 6B model significantly improves over GPT-Neo’s 2.7B model across languages, but scaling to GPT-NeoX 20B yields uneven gains.Go, Java, Rust, Scala, and TypeScript show no significant improvement with the larger model.
- Python perplexity trends correlate with HumanEval performance, suggesting perplexity can estimate downstream performance at lower cost.
6 CONCLUSION
The paper systematically evaluates code language models and finds that performance generally benefits from larger models and longer training. It releases PolyCoder, an open-source model trained exclusively on code across 12 languages, which achieves the lowest perplexity in C among the evaluated models.
- Performance generally benefits from larger models and longer training time.
- PolyCoder is an open-source language model trained exclusively on code in 12 programming languages.
- PolyCoder achieves lower perplexity than all evaluated models in C.
A SCALING EFFECT: TRAINED LONGER
Longer training improves the larger PolyCoder model consistently, while benefits are subdued for the smaller model. Pass@100 increases most for the 2.7B model but drops for the 400M model.
- Training the 2.7B model from 100K to 150K steps increases performance uniformly, with the largest improvement in Pass@100.
- For the 400M model, extending training to 100K steps produces subdued improvements and lowers Pass@100.
- Longer training may provide an additional performance boost for larger models.
B TEMPERATURE EFFECT: SMALLER MODELS
Temperature affects HumanEval performance across all three model sizes, with especially high values potentially harming larger models. The authors therefore emphasize tuning temperature by model capacity and generation scenario.
- Temperature affects HumanEval performance for models of all three sizes.
- For the 2.7B model, temperature 0.8 hurts Pass@100 performance.
- The authors attribute this decline to high temperature producing overly diverse outputs that reduce correctness.
- Temperature should be tuned separately for model capacity and generation scenario.
C DETAILED PERPLEXITY RESULTS
The paper reports language-specific perplexity results in Table 5, normalizing token counts after lexer-based tokenization for cross-model comparison. It also notes that CodeParrot performs poorly outside Python because it was trained only on Python data.
- Perplexity scores are normalized using token counts obtained by tokenizing each language with its respective Pygments lexer.
- CodeParrot performs poorly in languages other than Python because it was trained only on Python data.
- Figure 8 concerns HumanEval performance under different softmax temperatures, not the detailed perplexity results.
- Table 5 reports perplexity for different models across programming languages.