CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis

Erik Nijkamp; Bo Pang; Hiroaki Hayashi; Lifu Tu; Huan Wang; Yingbo Zhou; Silvio Savarese; Caiming Xiong

Source-linked AI summary

CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis

Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, Caiming Xiong

arXiv:2203.13474v5Mar 25, 2022cs.LGcs.CLcs.PL

TL;DR

Program synthesis seeks to generate programs from user intent, but open access to competitive large models is limited by training-resource constraints. The paper trains and releases CODEGEN models and JAXFORMER, then studies multi-turn synthesis with a dedicated benchmark. CODEGEN shows competitive HumanEval performance, while multi-turn specifications improve synthesis over single-turn specifications.

Problem
Program synthesis aims to generate programs from user intent, while limited access to competitive large models hinders broader research.
Method
The paper trains CODEGEN models on natural-language and programming-language data, releases JAXFORMER and checkpoints, and evaluates multi-turn synthesis with a dedicated benchmark.
Results
Multi-turn specifications significantly improve CODEGEN program synthesis over single-turn specifications, and CODEGEN achieves competitive or improved HumanEval performance against Codex.
Takeaways & Limitations
Progressively decomposing user intent into multiple natural-language turns provides a supported paradigm for program synthesis, alongside openly released models and training infrastructure.
Takeaways & Limitations
CODEGEN may generate biased or profane outputs and vulnerable programs; these risks are not remedied, so models should not be used before treatment.

Abstract

from arXiv · show

Program synthesis strives to generate a computer program as a solution to a given problem specification, expressed with input-output examples or natural language descriptions. The prevalence of large language models advances the state-of-the-art for program synthesis, though limited training resources and data impede open access to such models. To democratize this, we train and release a family of large language models up to 16.1B parameters, called CODEGEN, on natural language and programming language data, and open source the training library JAXFORMER. We show the utility of the trained model by demonstrating that it is competitive with the previous state-of-the-art on zero-shot Python code generation on HumanEval. We further investigate the multi-step paradigm for program synthesis, where a single program is factorized into multiple prompts specifying subproblems. To this end, we construct an open benchmark, Multi-Turn Programming Benchmark (MTPB), consisting of 115 diverse problem sets that are factorized into multi-turn prompts. Our analysis on MTPB shows that the same intent provided to CODEGEN in multi-turn fashion significantly improves program synthesis over that provided as a single turn. We make the training library JAXFORMER and model checkpoints available as open source contribution: https://github.com/salesforce/CodeGen.

1 INTRODUCTION

The paper frames program synthesis as automating intent-preserving code generation and proposes multi-turn interaction to address search-space and specification challenges. It introduces a benchmark and open-source resources for studying this paradigm.

Motivation: Program synthesis aims to automate coding by generating programs that satisfy users’ specified intent.The authors position this as a way to improve experienced programmers’ productivity and broaden programming accessibility.
Motivation: Two central challenges are the intractable program search space and the difficulty of properly specifying user intent.Domain-specific languages restrict applicability, while general-purpose languages create larger search spaces.
Motivation: Input-output examples reduce specification effort but can under-specify intent, whereas formal specifications require greater expertise and effort.The paper discusses logical expressions, pseudocode, examples, and natural-language specifications as alternatives.
Approach: The proposed multi-turn approach lets users progressively provide natural-language specifications while the system generates subprograms across multiple steps.Factorizing complex specifications may ease model understanding, reduce the search space, and improve synthesized-program quality.
Evaluation: The authors develop a multi-turn programming benchmark that measures program synthesis from natural-language specifications through pass rates on expert-written test cases.They describe it as the first benchmark enabling quantitative analysis of multi-turn program synthesis.
Contributions: The work contributes open-source model checkpoints and the JAXFORMER training library to broaden access to large program-synthesis models.The authors argue that limited access to expensive training resources hinders progress.

2 MODEL TRAINING

CODEGEN uses autoregressive transformers trained sequentially on natural-language, multilingual-code, and Python-code datasets at multiple scales. The training setup is intended to study how model and data scale affect program-synthesis capacity.

Training data: THEPILE is an 825.18 GiB English corpus whose GitHub programming data constitutes 7.6% of the dataset.Models trained primarily on this corpus are called CODEGEN-NL.
Training data: BIGQUERY supplies code in C, C++, Go, Java, JavaScript, and Python for multilingual CODEGEN models.These models are called CODEGEN-MULTI.
Training data: BIGPYTHON contains permissively licensed Python code compiled from public GitHub information for mono-lingual CODEGEN models.These models are called CODEGEN-MONO.
Models: CODEGEN models are trained as autoregressive transformers with next-token prediction on natural-language and programming-language data.Model configurations contain 350M, 2.7B, 6.1B, or 16.1B parameters.
Training data: The models are trained sequentially on THEPILE, BIGQUERY, and BIGPYTHON.CODEGEN-NL starts from THEPILE, CODEGEN-MULTI adds BIGQUERY, and CODEGEN-MONO adds BIGPYTHON.
Infrastructure: JAXFORMER supports large-scale training on TPU-v4 hardware using data and model parallelism.The library and trained model weights are released as open source.

3 SINGLE-TURN EVALUATION

CODEGEN models are evaluated on HumanEval, where performance improves with model size and additional programming-language training. Python-focused training yields competitive or improved performance against Codex, while lower prompt perplexity is associated with successful synthesis.

HumanEval evaluates 164 hand-written Python problems by functional correctness on generated function completions.
Increasing model size improves overall HumanEval performance for almost all models.
Multilingual programming-language training substantially improves CODEGEN over models trained only on THEPILE, while Python-only training further improves synthesis capacity.
CODEGEN-MONO models achieve competitive or improved performance compared with Codex on HumanEval.
Pass problems have lower prompt perplexity than non-pass problems, indicating that successful synthesis is associated with better understanding of user intent.

4 MULTI-TURN EVALUATION

The Multi-Turn Programming Benchmark evaluates programs synthesized from progressively specified subproblems. Across CODEGEN models, multi-turn specifications generally reduce perplexity and improve pass rates relative to concatenated single-turn specifications, with gains varying by difficulty and model size.

4 MULTI-TURN EVALUATION: MTPB contains 115 expert-written problems requiring models to synthesize functionally correct subprograms across multiple natural-language steps.
4.1 BENCHMARK CONSTRUCTION: Each MTPB problem combines multi-turn prompts, test inputs, and expected outputs spanning diverse programming categories.
4.1 BENCHMARK CONSTRUCTION: MTPB embeds test-case inputs into prompt templates because its free-form problems do not provide a fixed function entry point.
4.3 MULTI-STEP PROGRAMMING CAPACITY SCALES WITH MODEL SIZE AND DATA SIZE: CODEGEN MTPB performance improves with both model size and code-data size.
4.2 BETTER USER INTENT UNDERSTANDING YIELDS BETTER SYNTHESIZED PROGRAMS: Multi-turn specifications have lower average perplexity than concatenated single-turn specifications for all evaluated models.
4.2 BETTER USER INTENT UNDERSTANDING YIELDS BETTER SYNTHESIZED PROGRAMS: Multi-turn specifications improve pass rates by close to or more than 10 percentage points across model sizes.
4.3 MULTI-STEP PROGRAMMING CAPACITY SCALES WITH MODEL SIZE AND DATA SIZE: Multi-turn prompts improve performance across almost all difficulty levels and model sizes, except that larger models show little benefit on easy problems.
4.3 MULTI-STEP PROGRAMMING CAPACITY SCALES WITH MODEL SIZE AND DATA SIZE: In contrasting cases, CODEGEN-MONO 16.1B can perform worse than 2.7B because the larger model takes prompts literally and becomes inflexible.

5 RELATED WORK

Prior program-synthesis research addresses difficult search and specification problems through search restrictions, language models, and varied benchmark input forms. Existing benchmarks mostly provide single, directly relevant inputs rather than multi-turn specifications.

Program synthesis faces unresolved challenges from intractable program spaces and difficulty expressing user intent accurately.
Prior methods constrain or navigate program search using domain-specific languages, stochastic techniques, and deductive top-down search.
Transformer language models have been applied broadly across natural language, code, vision, and other domains.
Program-synthesis benchmarks use inputs including preceding code, pseudocode, docstrings with signatures, and natural-language problem descriptions.

6 CONCLUSION

The paper studies multi-step program synthesis with scalable causal language models and introduces MTPB to evaluate it. Multi-step specifications are more easily digested and produce more accurate programs, while the released code and checkpoints support further research.

CODEGEN models scale multi-step program-synthesis capacity with model size and data size.
The Multi-Turn Programming Benchmark measures synthesis from intent specified across multiple steps.
Multi-step intent specifications are digested more easily by models and lead to more accurate program synthesis.
The authors open-source training code and model checkpoints for future research and practical applications.

BROADER IMPACT AND ETHICAL CONSIDERATIONS

CODEGEN scales autoregressive language models across parameter and programming-language data sizes, while releasing JAXFORMER and model weights. The work also documents risks from profane or biased training data and potentially vulnerable generated programs.

CODEGEN may generate profane or sentiment-biased content because its Pile pretraining data contains such material.The authors report 0.1% profane language in aligned GitHub data and note possible gender and religious-group sentiment biases.
Generated programs may contain vulnerabilities and safety concerns that this work does not remedy.The authors recommend treating these risks before using the models in applications.
The models vary from 350M to 16.1B parameters and programming-language token counts to study multi-turn synthesis under scaling laws.JAXFORMER supports training on TPU-v4 hardware and is released with trained model weights.

A.1 DATASETS

The CODEGEN family uses autoregressive transformers trained on sequentially curated natural-language and programming-language corpora, with model specifications summarized across several scales. Dataset preparation applies standardized filtering, deduplication, tokenization, shuffling, and concatenation.

The training corpora undergo filtering, deduplication, tokenization, shuffling, and concatenation before model training.Sequences are concatenated to fill a 2,048-token context, and exact duplicates are removed using SHA-256 hashes.
CODEGEN-NL trains on THEPILE, CODEGEN-MULTI continues from it on BIGQUERY, and CODEGEN-MONO continues from CODEGEN-MULTI on BIGPYTHON.
The models are autoregressive transformers trained with next-token prediction in sizes ranging from 350M to 16.1B parameters.The smaller configurations support comparison with GPT-NEO and GPT-J.

A.3 TRAINING

Training uses JAXFORMER to coordinate data and model parallelism across TPU-v4 hardware, with sharding schemes selected by model size. Evaluation estimates pass@k from sampled programs using an unbiased, numerically stable procedure.

JAXFORMER uses JAX pjit() to evaluate computations in parallel with equivalent semantics across a logical device mesh.The implementation targets data and model parallelism on TPU-v4 hardware.
Models up to 6B parameters use intra-TPU-VM sharding, whereas larger models use inter-TPU-VM sharding across TPU-v4 boards.The distinction follows whether model and optimizer parameters fit within one board’s combined HBM memory.
The training configuration uses Adam with (β1, β2, ϵ) = (0.9, 0.999, 1e−08), gradient clipping at 1.0, warm-up, and cosine annealing.The authors report limited capacity to optimize these hyperparameters further.
Pass@k is estimated from n = 200 samples with k ≤100 using the unbiased estimator proposed by Chen et al.A numerically stable NumPy implementation is used because direct computation is unstable.

C TYPE-RELAXED EQUIVALENCE CHECK FOR MTPB EVALUATION

MTPB evaluation relaxes output-type comparisons and organizes benchmark problems by named categories, while perplexity is computed from conditional prompt probabilities in multi-turn and concatenated single-turn settings. The supplied passages describe the evaluation machinery but do not report a figure-specific comparison.

Type-relaxed equivalence checking: MTPB equivalence checking converts arrays and pandas series to comparable formats, casts outputs to gold-standard types, and uses ε = 1e−6 for floating-point comparisons.
MTPB problems: MTPB’s listed problems span algorithmic and data-science categories, including sorted lists, subarrays, matrix operations, and condition numbers.The table caption identifies D.S. as data science and Algo. as algorithm.
Perplexity computation: For multi-turn prompts, each prompt’s probability is conditioned on the concatenated preceding prompts and synthesized subprograms.The total prompt-token count is denoted by m.
Perplexity computation: For single-turn prompts, perplexity uses the probability of the fully concatenated sequence containing all prompts and synthesized subprograms.

G ADDITIONAL BENCHMARK RESULTS

MBPP evaluation shows consistent performance gains across CODEGEN’s NL, Multi, and Mono versions, with the largest models approaching or surpassing relevant open baselines.

CODEGEN performance consistently improves across the NL, Multi, and Mono versions on sanitized MBPP.All models were sampled with n = 100 and temperature = 0.8.
CODEGEN-MONO 16.1B approaches the results from code-cushman-001 on MBPP.
CODEGEN-MONO 6.1B significantly outperformed INCODER 6B on MBPP.

H GENERATED SAMPLES

Generated samples illustrate model-specific errors and improvements in multi-turn code generation, while additional analyses examine relationships among program length, prompt length, and pass rate.

CODEGEN-MONO 16.1B sometimes assumes a variable’s type despite an explicit string-initialization instruction, causing an error when later code iterates over the variable.
CODEGEN-MONO 16.1B generates a function definition and uses return in the final turn, although return is valid only within a function.
CODEGEN-MONO 2.7B misinterprets “swap” as part of an assigned value, whereas CODEGEN-MONO 16.1B correctly assigns two variables in parallel.
CODEGEN-MONO 2.7B checks only the first character because it fails to implement num_in_str, while CODEGEN-MONO 16.1B scans all characters using a function.
Additional analyses report generated-program length versus pass rate in Figures 7–9 and prompt length versus generated-program length in Figure 10.The accompanying text identifies these as relationships analyzed across the generated samples.