Source-linked AI summary
CERT: Continual Pre-Training on Sketches for Library-Oriented Code Generation
Daoguang Zan, Bei Chen, Dejian Yang, Zeqi Lin, Minsu Kim, Bei Guan, Yongji Wang, Weizhu Chen, Jian-Guang Lou
TL;DR
Library-oriented code generation lacks affordable paired data covering the many third-party libraries programmers reuse. CERT continually pre-trains a sketcher and generator on unlabelled code, and its benchmarks and experiments show superior library-oriented generation performance, including a 13.43% pass@1 improvement over PYCODEGPT on NumpyEval.
Problem
Library-oriented generation needs to work across many third-party libraries, but collecting sufficient labelled text-code pairs for them is extremely costly.
Method
CERT continually pre-trains a sketcher to predict anonymized code sketches and a generator to fill their details, using unlabelled library-oriented code.
Results
CERT consistently outperforms baselines, with a 13.43% pass@1 improvement over PYCODEGPT on NumpyEval.
Takeaways & Limitations
CERT's sketch-based continual pre-training improves library-oriented code generation across problems with varying numbers of API calls.
Abstract
from arXiv · showhide
Code generation is a longstanding challenge, aiming to generate a code snippet based on a natural language description. Usually, expensive text-code paired data is essential for training a code generation model. Recently, thanks to the success of pre-training techniques, large language models are trained on large-scale unlabelled code corpora and perform well in code generation. In this paper, we investigate how to leverage an unlabelled code corpus to train a model for library-oriented code generation. Since it is a common practice for programmers to reuse third-party libraries, in which case the text-code paired data are harder to obtain due to the huge number of libraries. We observe that library-oriented code snippets are more likely to share similar code sketches. Hence, we present CERT with two steps: a sketcher generates the sketch, then a generator fills the details in the sketch. Both the sketcher and the generator are continually pre-trained upon a base model using unlabelled data. Furthermore, we craft two benchmarks named PandasEval and NumpyEval to evaluate library-oriented code generation. Experimental results demonstrate the impressive performance of CERT. For example, it surpasses the base model by an absolute 15.67% improvement in terms of pass@1 on PandasEval. Our work is available at https://github.com/microsoft/PyCodeGPT.
1 Introduction
Library-oriented code generation must work without costly text-code annotations covering many third-party libraries. CERT addresses this by using shared code sketches to structure continual pre-training on unlabelled code.
- Motivation: Large-scale code pre-training can reduce reliance on paired data, with Codex achieving 28.8% pass@1 on standalone Python problems.
- Motivation: Library-oriented generation targets code using third-party APIs, but covering many libraries with labelled text-code pairs is extremely costly.More than 40% of Python-tagged StackOverflow questions also have at least one library tag.
- Sketch insight: Library-oriented snippets are more likely than standalone snippets to share sketches formed by anonymizing user-defined terms.The sketch captures code structure while abstracting variables, methods, constants, and related details.
- CERT: CERT predicts a sketch first and then generates complete code from the sketch and original context, with both modules continually pre-trained on unlabelled code.The approach is designed specifically for library-oriented generation without pairwise labelled data.
2 Task Formulation
The task generates target code from a context containing natural-language descriptions and code statements. Library-oriented instances differ from standalone generation because their targets contain third-party library API calls.
- Task definition: The model maps context x, containing code and natural-language tokens, to target code y that solves the programming problem.
- Task definition: Context includes elements such as imports, function headers, variable definitions, and code comments describing the problem.
- Task distinction: Library-oriented targets contain library API calls, whereas standalone targets are expected to avoid third-party libraries.
- Task distinction: Unlike fine-tuning, the proposed approach requires continual pre-training on unlabelled code corpora rather than carefully labelled context-target pairs.
3 Methodology
CERT builds a sketcher and generator on a code-pretrained base model. The sketcher abstracts library code into sketches, while the generator learns to complete code from sketch prompts.
- CERT architecture: CERT decomposes generation into a sketcher MS that predicts s and a generator MG that produces y from [s; x].The sketcher samples 200 candidate sketches and selects the most frequent one; complete or empty sketches trigger special direct paths.
- Base models: The models are built on PYCODEGPT or CODEGEN through continual pre-training on library-specific code subcorpora.The library-specific subcorpus is extracted from the broader pre-training corpus.
- Sketcher: Sketching anonymizes user-defined code terms using predefined symbols, with variants covering constants, names, or both.The default CERT setting anonymizes constants; CERT-N anonymizes names, and CERT-NC anonymizes both.
- Generator: Generator training cross-merges sketch blocks with corresponding original-code blocks to mimic using sketches as prompts.Natural-language comments are removed from sketching files before blocks are merged.
4 Benchmark Construction
PandasEval and NumpyEval provide Python benchmarks for library-oriented generation using authentic, manually polished programming problems derived from StackOverflow. Each benchmark contains 101 problems with executable tests.
- Benchmark design: PandasEval and NumpyEval address the limited evaluation of library-oriented code generation for Pandas and NumPy.The benchmarks are intended to be diverse, authentic, high quality, moderately difficult, and unseen during pre-training.
- Data collection: Problems are sourced from highly voted StackOverflow posts with accepted answers, then manually organized, polished, and checked by experienced programmers.
- Evaluation: Each benchmark contains 101 programming problems, and every problem includes test cases for evaluation.
- Evaluation: Function-based problems receive 20 test cases, while non-function problems receive one test case checking predicted-variable correctness.Overall, 64% of PandasEval problems and 30% of NumpyEval problems have 20 test cases.
5 Experiments
CERT is evaluated on library-oriented code generation using pass@k across PandasEval and NumpyEval, with analyses of API complexity, sketch quality, and case studies. It consistently outperforms baselines and benefits from library-specific training data and sketch-based generation.
- Base Model: PYCODEGPT achieves 8.33% pass@1 on HumanEval and outperforms several comparable or larger baseline models.The compared models include AlphaCode, CodeClippy, CodeParrot, and GPT-Neo.
- Main Results: CERT consistently outperforms the evaluated baselines on PandasEval and NumpyEval, including 13.43% pass@1 improvement for PYCODEGPT-CERT on NumpyEval.CODEGEN-CERT achieves a 12.69% pass@1 improvement over CODEGEN on NumpyEval.
- API Complexity: PYCODEGPT-CERT steadily improves over PYCODEGPT across groups divided by the number of API calls.The result indicates gains across library-oriented problems with varying difficulty.
- Closer Analysis: The default sketching configuration performs best, while anonymizing both constants and names can make sketches too abstract.The default configuration anonymizes user-defined constants; CERT-N and CERT-NC represent alternative operations.
- Closer Analysis: CERT trained on library-oriented files outperforms CERTg trained for general code generation, indicating the importance of library-specific training data.The comparison uses extracted Pandas- and NumPy-oriented files for CERT training.
- Case Study: Case studies show that the generator can produce correct code even when the sketcher predicts an imperfect sketch.The sketch acts as a prompt rather than requiring perfect correctness, giving the generator robustness.
6 Related Work
Related work includes large pre-trained code models and sketch-based code-generation methods. CERT differs by targeting library-oriented generation while continually pre-training on unlabelled code instead of fine-tuning on labelled text-code pairs.
- Pre-trained Code Models: Existing decoder and encoder-decoder code models are trained on unlabelled code and can perform code generation directly, but focus on standalone code.Examples include CodeT5, CodeGPT, PLBART, PolyCoder, CODEGEN, AlphaCode, and Codex.
- Sketch-based Methods: Prior sketch-based methods require labelled text-code pairs for fine-tuning, whereas CERT continually pre-trains on unlabelled code corpora.The comparison includes Coarse-to-Fine, BAYOU, SKETCHADAPT, and PLOTCODER.
- Task Scope: CERT focuses on library-oriented code generation, addressing a setting distinct from the standalone-code emphasis of related pre-trained models.The paper also introduces benchmarks for evaluating this setting.
7 Conclusion
The paper concludes that CERT combines sketching with continual pre-training to improve library-oriented code generation and evaluates the approach with PandasEval and NumpyEval. It identifies private libraries with limited data as future work.
- Approach: CERT combines a sketcher and generator that are continually pre-trained on unlabelled code corpora.The sketcher predicts code sketches, while the generator completes the code.
- Benchmarks: PandasEval and NumpyEval are crafted to evaluate library-oriented code generation.The benchmarks target Python libraries.
- Conclusion: Experimental results and analysis support the effectiveness of CERT for library-oriented code generation.The conclusion presents this as the paper’s overall empirical finding.
- Future Work: Future work will investigate code generation for private libraries with fewer data.This scope boundary concerns libraries with limited available training data.
A PYCODEGPT: A Democratizing Code Generation Model in Python
The paper introduces PYCODEGPT, a publicly available Python code pre-trained model intended to reproduce Codex at medium scale. It addresses the limited availability of leading proprietary code models.
- Motivation: PYCODEGPT is proposed as a publicly available pre-trained model for Python code generation.It is intended to support research and applications that are hindered by unavailable state-of-the-art models.
- Model Goal: The model aims to reproduce Codex with a medium-sized architecture.The motivation is Codex’s strong performance combined with its lack of public availability.
A.1 Data Construction
The data pipeline collects Python-related repositories from GitHub and filters source files for quality, syntax, and meaningful content before training.
- The pipeline crawls 7.6M GitHub repository pages and obtains 1.2M Python-related repository URLs after filtering.
- Files must contain at least 5 lines, average lines no longer than 100 characters, maximum lines no longer than 1000 characters, and an alphanumeric rate of at least 0.98.
- The process removes automatically generated files, meaningless files, license descriptions, and low-alphanumeric-rate comments.
- Python syntax checking combines the built-in ast module with pattern matching for files containing more than two typical Python keywords.
A.2 Model Training
Model training uses GPT-Neo as the base and adapts its tokenization and sampling to Python source-code quality and distribution.
- GPT-Neo serves as the base model, while PYCODEGPT uses a new byte-level BPE tokenizer trained from scratch for Python source code.
- The tokenizer is needed because GPT-Neo’s original tokenizer is less effective for encoding Python source code than natural text.
- Resampling makes high-quality files appear more often while ensuring every file appears at least once during training.
- File quality is assessed using repository star count and unit test function rate, which reduces the weight of test files containing many user-defined constants.
A.3 Experiments
Experiments evaluate PYCODEGPT on HumanEval and CodeXGLUE, with results showing gains over comparable open-source baselines on code generation and completion metrics.
- PYCODEGPT is evaluated on HumanEval and the CodeXGLUE PY150 code-completion task.
- 4.53% on pass@1, 6.79% on pass@10, and 6.35% on pass@100 are PYCODEGPT’s improvements over CodeParrot 110M on HumanEval.
- PYCODEGPT achieves comparable performance to Codex 85M on HumanEval while outperforming CodeParrot 110M across the reported pass@k metrics.
- 3.02% higher token-level completion accuracy is achieved by PYCODEGPT over CodeParrot on CodeXGLUE PY150.
- On line-level completion, PYCODEGPT achieves 2.67% higher exact-match accuracy and 2.02% higher edit-similarity accuracy than CodeParrot.