PaperlayerView exact revision on arXiv

Source-linked AI summary

CodeT5+: Open Code Large Language Models for Code Understanding and Generation

Yue Wang, Hung Le, Akhilesh Deepak Gotmare, Nghi D. Q. Bui, Junnan Li, Steven C. H. Hoi

arXiv:2305.07922v2cs.CLcs.LGcs.PL

TL;DR

Existing code LLMs are constrained by inflexible architectures and limited pretraining objectives that do not align with all downstream tasks. CodeT5+ addresses these issues with modular architectures and diverse unimodal and bimodal objectives, achieving strong results across more than 20 benchmarks, including new HumanEval results for its instruction-tuned 16B model.

  • Problem

    Existing code LLMs use inflexible or unified architectures and limited pretraining objectives, causing weaknesses on some downstream tasks.

  • Method

    CodeT5+ flexibly combines encoder, decoder, and cross-modal modules trained with span denoising, causal LM, contrastive learning, and text-code matching objectives, using efficient initialization from frozen LLMs and instruction tuning.

  • Results

    CodeT5+ shows substantial gains across more than 20 code benchmarks, and instruction-tuned CodeT5+ 16B achieves 35.0% pass@1 and 54.5% pass@10 on HumanEval.

  • Takeaways & Limitations

    The results support CodeT5+ as an open code LLM family applicable across understanding, generation, completion, math programming, and text-to-code retrieval tasks.

  • Takeaways & Limitations

    Comparisons with billion-parameter LLMs are difficult to ensure fair because those models may overlap with downstream data and are often evaluated zero-shot.

Abstract

from arXiv · show

Large language models (LLMs) pretrained on vast source code have achieved prominent progress in code intelligence. However, existing code LLMs have two main limitations in terms of architecture and pretraining tasks. First, they often adopt a specific architecture (encoder-only or decoder-only) or rely on a unified encoder-decoder network for different downstream tasks. The former paradigm is limited by inflexibility in applications while in the latter, the model is treated as a single system for all tasks, leading to suboptimal performance on a subset of tasks. Secondly, they often employ a limited set of pretraining objectives which might not be relevant to some downstream tasks and hence result in substantial performance degrade. To address these limitations, we propose ``CodeT5+'', a family of encoder-decoder LLMs for code in which component modules can be flexibly combined to suit a wide range of downstream code tasks. Such flexibility is enabled by our proposed mixture of pretraining objectives to mitigate the pretrain-finetune discrepancy. These objectives cover span denoising, contrastive learning, text-code matching, and causal LM pretraining tasks, on both unimodal and bimodal multilingual code corpora. Furthermore, we propose to initialize CodeT5+ with frozen off-the-shelf LLMs without training from scratch to efficiently scale up our models, and explore instruction-tuning to align with natural language instructions. We extensively evaluate CodeT5+ on over 20 code-related benchmarks in different settings, including zero-shot, finetuning, and instruction-tuning. We observe state-of-the-art (SoTA) model performance on various code-related tasks, such as code generation and completion, math programming, and text-to-code retrieval tasks. Particularly, our instruction-tuned CodeT5+ 16B achieves new SoTA results on HumanEval code generation task against other open code LLMs.

1 Introduction

CodeT5+ addresses limitations in existing code LLM architectures and pretraining objectives by flexibly combining encoder, decoder, and cross-modal components. It achieves strong results across diverse code tasks and evaluation settings.

  • Motivation: Existing encoder-only and decoder-only code LLMs specialize in different task types, while unified encoder-decoder models can remain suboptimal on certain tasks.Encoder-only models favor understanding; decoder-only models favor generation but are less ideal for retrieval and detection.
  • Motivation: Limited pretraining objectives can create a discrepancy between pretraining and finetuning, such as span denoising being less aligned with autoregressive code generation.
  • Approach: CodeT5+ flexibly operates in encoder-only, decoder-only, and encoder-decoder modes using span denoising, causal LM, contrastive learning, and text-code matching objectives.The objectives use unimodal code and bimodal text-code data.
  • Approach: CodeT5+ scales efficiently by initializing components from off-the-shelf code LLMs, freezing the deep decoder, and training only the shallow encoder and cross-attention layers.Instruction tuning is also explored to align models with natural language instructions.
  • Results: Over 20 benchmarks show substantial gains across retrieval, completion, retrieval-augmented generation, math programming, and HumanEval code generation.Instruction-tuned CodeT5+ 16B reaches 35.0% pass@1 and 54.5% pass@10 on HumanEval against other open code LLMs.

2 Related Work

Prior code LLMs typically use specialized or unified architectures and limited pretraining objectives, creating task-specific weaknesses. CodeT5+ instead combines flexible modules, diverse objectives, frozen pretrained LLM components, and instruction tuning.

  • Code LLM architectures: Code LLMs are commonly organized as encoder-only, decoder-only, or encoder-decoder architectures with different strengths across understanding and generation tasks.
  • Pretraining objectives: Span denoising alone is poorly matched to autoregressive tasks such as next-line completion because it recovers short spans rather than whole programs.
  • Code LLM architectures: UniXcoder supports multiple tasks through a single encoder, but inter-task interference can degrade sequence-to-sequence performance, especially code generation.
  • Scope: Infilling-style code completion with context after the cursor is explicitly outside this work’s focus.
  • CodeT5+: CodeT5+ dynamically activates component modules for different tasks and does not rely on code-specific features.
  • Efficient training: CodeT5+ uses a shallow encoder and deep decoder, freezes most decoder parameters, and combines this scheme with instruction tuning using synthetic prompts.

3 CodeT5+: Open Code Large Language Models

CodeT5+ combines staged unimodal and bimodal pretraining with a modular encoder-decoder architecture that supports multiple downstream modes. Its objectives cover code reconstruction, autoregressive generation, and text-code alignment while enabling efficient scaling from pretrained LLMs.

  • Architecture: CodeT5+ operates in encoder-only, decoder-only, and encoder-decoder modes through a mixture of objectives on unimodal code and bimodal code-text data.
  • Unimodal pretraining: Stage one jointly trains span denoising and causal LM objectives on large-scale unimodal code to recover spans, partial programs, and complete programs.
  • Unimodal pretraining: Span denoising masks 15% of input tokens and trains the decoder to reconstruct the masked spans.
  • Unimodal pretraining: The two causal LM variants train sequence-to-sequence continuation and decoder-only full-program generation, providing autoregressive supervision at different levels.
  • Bimodal pretraining: Stage two uses function-level text-code pairs with contrastive learning, matching, and cross-modal causal LM objectives.
  • Bimodal pretraining: Text-code causal LM supports both text-to-code and code-to-text generation through a dual multimodal conversion objective.
  • Efficient scaling: Compute-efficient scaling initializes the encoder and decoder from pretrained LLMs, adds cross-attention layers, and trains only a small subset of parameters.
  • Downstream modes: After pretraining, CodeT5+ supports sequence-to-sequence generation, retrieval-augmented generation, decoder-only tasks, and understanding tasks.

4 Pretraining and Instruction Tuning

CodeT5+ combines curated multilingual code and text-code data with staged pretraining, scalable initialization, and synthetic instruction tuning. The setup supports representation learning, text-code adaptation, and alignment with natural-language instructions across varied code tasks.

  • 4.1 Pretraining Dataset: Nine programming languages are selected from GitHub Code and filtered to permissively licensed files containing 50–2000 tokens.The dataset also removes overlap with CodeSearchNet and evaluated downstream tasks.
  • 4.1 Pretraining Dataset: The curated GitHub dataset is much larger at the file level than CodeSearchNet’s bimodal function-level data.This scale supports rich representations during the first pretraining stage.
  • 4.3 Instruction Tuning: Instruction data includes novel tasks such as text-to-SQL generation and Python code optimization.Figure 4 presents examples used to align CodeT5+ with natural-language instructions.
  • 4.1 Pretraining Dataset: CodeT5+ uses CodeSearchNet’s bimodal subset in a later stage to adapt the model to text-code retrieval and generation.The first stage uses unimodal code, while the second stage focuses on text-code tasks.
  • 4.2 Pretraining Setup: CodeT5+ 220M and 770M are trained from scratch, whereas the 2B, 6B, and 16B models initialize decoders and encoders from CodeGen-mono checkpoints.The larger models use CodeGen-mono decoders at matching scales and 350M encoders.
  • 4.2 Pretraining Setup: Pretraining begins with 10k span-denoising steps followed by 100k steps of equally weighted joint training on two causal language-modeling tasks.The setup uses linear learning-rate decay, a peak learning rate of 2e-4, and task-specific batch sizes.
  • 4.3 Instruction Tuning: The instruction corpus contains over 20k independently generated code examples and is used for up to three training epochs.Pretrained LLMs generate task instructions, inputs, and expected outputs, while evaluation remains zero-shot on downstream tasks.
  • 4.3 Instruction Tuning: LM-generated instruction data is imperfect, including a wrong output in the illustrated code-optimization example, but is treated as noise.The authors report that models still benefit from the majority of the synthetic dataset.

5 Experiments

CodeT5+ is evaluated across diverse code understanding and generation settings, with strong results across math programming, code completion, retrieval, summarization, and retrieval-augmented generation. The experiments also examine model scaling, architectural modes, and objective contributions.

  • Text-to-Code Generation: CodeT5+ 16B achieves 35.0% pass@1 and 54.5% pass@10 on HumanEval, establishing new state-of-the-art results against open code LLMs.With enhancement generation, performance reaches 42.9% pass@1 and 67.8% pass@10, although gaps remain against closed-source models.
  • Math Programming: CodeT5+ 770M achieves 87.4 pass@80 on MathQA-Python and 73.8 pass@100 on GSM8K-Python, outperforming many much larger models.It achieves the best finetuning results against larger models on GSM8K-Python and remains behind Minerva.
  • Math Programming: CodeT5+ is more robust to increasing reasoning-step complexity on MathQA-Python than CodeT5, whose performance deteriorates more sharply.The comparison analyzes problem difficulty using the number of reasoning steps required for each solution.
  • Code Summarization: Encoder-decoder CodeT5+ generally outperforms encoder-only, decoder-only, and UniLM-style models on code summarization.The authors connect this result to better encoding of code contexts and more accurate code summaries.
  • Code Completion: In decoder-only mode, CodeT5+ 220M surpasses UniXcoder, while CodeT5+ 770M sets new state-of-the-art results on both code-completion metrics.CodeT5+ 220M improves over same-sized CodeT5 by +6.5 EM on PY150 and +10.4 EM on JavaCorpus.
  • Text-to-Code Retrieval: CodeT5+ 770M surpasses UniXcoder by more than 3 absolute MRR points on all three retrieval tasks across eight datasets.The authors attribute retrieval gains to text-code contrastive learning and matching objectives, particularly fine-grained text-code matching.
  • Ablation Study: Removing causal LM causes major drops in code completion and math programming, while removing text-code matching reduces retrieval performance by 2.6 average MRR.The ablation covers three representative tasks over ten datasets and also finds retrieval benefits from joint causal-LM training.
  • Unified Retrieval-Augmented Generation: CodeT5+ performs better across retrieval-augmented generation categories and can operate end-to-end as both retriever and generator.A qualitative example shows retrieved code context guiding a more correct prediction, whereas generation alone produces an incorrect implementation.

6 Conclusion

CodeT5+ combines a flexible encoder-decoder architecture with mixed pretraining objectives and efficient initialization to support diverse code intelligence tasks. Across more than 20 datasets, it shows strong performance, including new zero-shot HumanEval results from instruction-tuned CodeT5+ 16B.

  • CodeT5+ can operate in encoder-only, decoder-only, and encoder-decoder modes for code understanding and generation tasks.
  • Its pretraining mixture spans span denoising, causal language modeling, contrastive learning, and text-code matching on unimodal and bimodal data.
  • CodeT5+ uses frozen off-the-shelf LLMs to initialize model components and scale efficiently without pretraining from scratch.
  • The model is further instruction-tuned to align with natural language instructions and evaluated across more than 20 code-intelligence datasets.
  • 35.0% pass@1 and 54.5% pass@10 establish new zero-shot HumanEval results for instruction-tuned CodeT5+ 16B against other open code LLMs.

A Ethics Statement

The paper highlights ethical risks in deploying code LLMs, including toxic generated language, security vulnerabilities, and intellectual-property concerns. It also describes bimodal pretraining objectives for text-code representations, matching, and generation.

  • Ethics: Generated code summaries or comments may contain toxic or insensitive language, motivating techniques for non-toxic natural-language generation.Suggested approaches include reinforcement learning, weighted decoding, and safety-specific control tokens.
  • Ethics: Code generation may introduce security vulnerabilities, so expert review and rigorous security assessments are recommended before adoption.
  • Ethics: Code retrieval systems should attribute retrieved sources to respect authors’ rights and improve transparency, traceability, and collaboration.
  • Bimodal Pretraining: Text-code contrastive learning maps [CLS] embeddings into normalized 256-d representations and compares text-to-code and code-to-text similarities.The method uses softmax-normalized similarities and cross-entropy against one-hot positive-pair labels.
  • Bimodal Pretraining: Text-code matching uses the decoder’s fused [EOS] representation to classify pairs as matched or unmatched, alongside text-to-code and code-to-text causal generation losses.

C Additional Experimental Results

The appendix reports additional experiments on defect and clone detection, retrieval-augmented generation, and qualitative math-programming results. It also examines how the number of retrieved code samples affects generation.

  • Additional Experiments: Additional experiments cover defect detection and clone detection from the CodeXGLUE benchmark.
  • Additional Experiments: The appendix analyzes top-k retrieval effects in retrieval-augmented code generation using Exact Match, BLEU-4, and CodeBLEU.
  • Additional Experiments: The appendix includes additional qualitative results for math programming tasks.

C.1 Code Defect Detection and Clone Detection from CodeXGLUE

CodeT5+ is evaluated on CodeXGLUE defect-detection and clone-detection tasks using accuracy and F1 score. It reaches a new defect-detection accuracy result, while clone-detection gains are comparable and saturated.

  • Defect Detection: 66.7% accuracy is reported as a new SoTA result for code defect detection.
  • Task Definitions: Defect detection predicts whether code is vulnerable to software systems.
  • Clone Detection: Clone detection measures whether two code snippets share common functionality, with CodeT5+ achieving results comparable to SoTA models.The reported performance increase appears saturated because gaps between multiple baselines are close.

C.2 Analysis on the Effects of Top-k Retrievals in Retrieval-augmented Code Generation

Retrieval-augmented code generation improves as more code samples are retrieved, but gains saturate at k=5 under the model’s maximum sequence length. CodeT5+ still outperforms a top-10 retrieval baseline with only one retrieved sample.

  • Top-k Effects: Increasing the number of retrieved code samples boosts retrieval-augmented generation performance until gains saturate at k=5.
  • Top-k Effects: The saturation is attributed to the maximum sequence length of 600, which may not accommodate many retrieved code samples.
  • Comparison: CodeT5+ significantly outperforms the prior SoTA top-10 retrieval baseline even with only a top-1 retrieved code sample.

C.3 Qualitative Results in Math Programming tasks

CodeT5+ generates programs that solve math problems across difficulty levels, including multi-step problems and solutions using external libraries such as math.

  • CodeT5+ generates decent Python programs for math problems ranging from simple operations to complex problems with multiple reasoning steps.
  • The model can leverage external libraries such as math when synthesizing solutions.
  • Figures 8 and 9 present model predictions on GSM8K-Python and MathQA-Python, respectively.

D Downstream Task Finetuning Details

The downstream evaluation covers multilingual text-to-code retrieval and code summarization tasks using established benchmarks, with detailed finetuning and contrastive-learning configurations.

  • Text-to-code retrieval: Text-to-code retrieval finds the code sample most relevant to a natural-language query from a collection of candidates.
  • Text-to-code retrieval: CodeSearchNet covers six programming languages, while CosQA uses Bing search logs and AdvTest obfuscates Python variable names.
  • Training configuration: Training uses maximum sequence lengths of 350 code tokens and 64 text tokens, a 2e-5 learning rate, 10 epochs, eight A100s, and batch size 64.
  • Code summarization: Code summarization generates a natural-language summary of a code snippet across Ruby, JavaScript, Go, Python, Java, and PHP.

D.3 Code Defect Detection

This section defines several downstream code tasks and describes retrieval-augmented generation, math-programming benchmarks, and their finetuning settings.

  • Code Defect Detection: Defect detection classifies whether a code sample contains vulnerability points or not.
  • Clone Detection: Clone detection identifies whether two code samples have the same functionality or semantics.
  • Code Completion: Line-level code completion generates the remaining code from a partial source sequence using PY150 and JavaCorpus benchmarks.
  • Math Programming: Math programming evaluates solving natural-language math problems by generating programs that return final answers on MathQA-Python and GSM8K.
  • Retrieval-Augmented Code Generation: Retrieval-augmented generation retrieves similar code candidates and augments the generator input before producing target code.
Loading 2305.07922v2…