PaperlayerView exact revision on arXiv

Source-linked AI summary

CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation

Yue Wang, Weishi Wang, Shafiq Joty, Steven C. H. Hoi

arXiv:2109.00859v1cs.CLcs.PL

TL;DR

Existing code pre-training methods are limited by architecture choices that favor either understanding or generation and by treating code too much like natural language. CodeT5 uses a unified encoder-decoder model with identifier-aware objectives and bimodal dual generation, and it outperforms prior work across most CodeXGLUE tasks while showing stronger code comprehension. Its generated code can still be incorrect or vulnerable, so outputs require expert correctness and security checking.

  • Problem

    Existing methods use encoder-only or decoder-only pre-training and often ignore code token types, limiting their fit across code understanding and generation tasks.

  • Method

    CodeT5 combines a unified encoder-decoder framework with identifier tagging and recovery objectives, bimodal NL-PL dual generation, and multi-task fine-tuning.

  • Results

    CodeT5 significantly outperforms prior work in most CodeXGLUE tasks and shows better code comprehension across various programming languages.

  • Takeaways & Limitations

    Identifier-aware pre-training and bimodal dual generation improve CodeT5’s use of code semantics and NL-PL alignment within its evaluated scope.

  • Takeaways & Limitations

    Generated outputs may contain incorrect or vulnerable code and should be checked by domain experts for correctness and security.

Abstract

from arXiv · show

Pre-trained models for Natural Languages (NL) like BERT and GPT have been recently shown to transfer well to Programming Languages (PL) and largely benefit a broad set of code-related tasks. Despite their success, most current methods either rely on an encoder-only (or decoder-only) pre-training that is suboptimal for generation (resp. understanding) tasks or process the code snippet in the same way as NL, neglecting the special characteristics of PL such as token types. We present CodeT5, a unified pre-trained encoder-decoder Transformer model that better leverages the code semantics conveyed from the developer-assigned identifiers. Our model employs a unified framework to seamlessly support both code understanding and generation tasks and allows for multi-task learning. Besides, we propose a novel identifier-aware pre-training task that enables the model to distinguish which code tokens are identifiers and to recover them when they are masked. Furthermore, we propose to exploit the user-written code comments with a bimodal dual generation task for better NL-PL alignment. Comprehensive experiments show that CodeT5 significantly outperforms prior methods on understanding tasks such as code defect detection and clone detection, and generation tasks across various directions including PL-NL, NL-PL, and PL-PL. Further analysis reveals that our model can better capture semantic information from code. Our code and pre-trained models are released at https: //github.com/salesforce/CodeT5 .

1 Introduction

CodeT5 addresses limitations in prior code pre-training by unifying understanding and generation in an encoder-decoder model that incorporates code-specific identifier information. It also uses code-comment pairs for NL-PL alignment and achieves state-of-the-art results across CodeXGLUE.

  • Motivation: Prior models often favor either understanding or generation and process code like natural language, overlooking structural information important for code semantics.Encoder-only models are suboptimal for generation, decoder-only models for understanding, and added decoders may not benefit from pre-training.
  • Approach: Its identifier-aware objective distinguishes identifier tokens and recovers masked identifiers to leverage developer-assigned code semantics.Identifiers such as function names and variables often convey informative semantic cues.
  • Approach: CodeT5 jointly trains NL→PL and PL→NL generation on code-comment pairs to improve NL-PL alignment.The bimodal data comes from source code and accompanying comments that are widely available in repositories.
  • Approach: CodeT5 is a unified pre-trained encoder-decoder model supporting code understanding, generation, and multi-task learning.The model builds on T5’s denoising sequence-to-sequence architecture and uses task control codes for multiple tasks.
  • Results: CodeT5 yields state-of-the-art results on all fourteen CodeXGLUE subtasks and better captures code semantics.The evaluation covers understanding tasks such as defect and clone detection and generation tasks across multiple directions.

2 Related Work

Earlier programming-language pre-training adapted NLP architectures to code, but many methods supported limited task types or ignored code-specific characteristics. CodeT5 extends encoder-decoder pre-training to a broader task set while explicitly modeling identifiers.

  • Natural-language pre-training: NLP pre-training models include encoder-only, decoder-only, and encoder-decoder architectures with different task orientations.The related work contrasts BERT- and GPT-style models with encoder-decoder systems such as T5.
  • Programming-language pre-training: Early programming-language models applied masked language modeling, replaced-token detection, autoregressive completion, translation, or UniLM objectives to source code.CuBERT, CodeBERT, GPT-based models, UniLM, and Transcoder represent these directions.
  • Programming-language pre-training: CodeT5 explores T5-based encoder-decoder pre-training for programming language and supports a more comprehensive set of tasks.This distinguishes it from work focused on narrower generation subsets or code completion.
  • Programming-language pre-training: Prior T5- and BART-based systems support some understanding or generation tasks but generally process code like natural language and ignore code-specific characteristics.The paper positions identifier modeling as a response to this limitation.
  • Programming-language pre-training: Other code models incorporate data flow, deobfuscation, or structural token distances, whereas CodeT5 focuses on identifier tagging and prediction within a sequence-to-sequence model.The paper treats identifiers as code elements that reserve rich semantic information.

3 CodeT5

CodeT5 is a unified encoder-decoder model that combines identifier-aware pre-training with bimodal NL-PL generation to support code understanding and generation. It adapts the pre-trained model to downstream tasks through task-specific or multi-task learning.

  • Input encoding: CodeT5 accepts either PL-only inputs or concatenated NL-PL inputs, with the NL segment empty when code has no accompanying description.For bimodal inputs, natural-language and code tokens are separated with [SEP] delimiters.
  • Pre-training: Identifier tagging labels each code token as an identifier or non-identifier using node types extracted from the code’s Abstract Syntax Tree.The resulting binary labels are used in a sequence-labeling objective over the programming-language segment.
  • Pre-training: CodeT5 extends T5-style denoising Seq2Seq pre-training with identifier tagging and masked identifier prediction for code-specific information.The model targets developer-assigned identifiers and uses these tasks to incorporate token-type information into the encoder-decoder framework.
  • Pre-training: Masked identifier prediction masks all identifiers and autoregressively recovers their names from obfuscated code.A unique sentinel token represents each specific identifier, requiring the model to link repeated occurrences and comprehend the surrounding code.
  • Pre-training: Bimodal dual generation trains bidirectional NL→PL and PL→NL conversion to reduce the mismatch between pre-training and downstream generation.The objective uses naturally paired code and natural-language descriptions to improve cross-modal alignment.
  • Fine-tuning: After pre-training, CodeT5 supports task-specific transfer learning and multi-task learning across code generation and understanding tasks.Generation uses the Seq2Seq framework, while understanding tasks generate labels or predict them from decoder representations; multi-task training uses task-control codes and balanced sampling.

4 Experimental Setup

CodeT5 is evaluated across six CodeXGLUE task families spanning code understanding and generation, using shared benchmark splits and task-specific metrics. The experiments use identifier-aware pre-training, a code-specific tokenizer, and comparisons with encoder-only, decoder-only, and encoder-decoder baselines.

  • Pre-training data: CodeT5 is pre-trained on CodeSearchNet’s unimodal and bimodal data across six programming languages, plus additional C/C# data from open-source repositories.
  • Code-specific tokenizer: The code-specific tokenizer reduces downstream tokenized code sequence length by 30% - 45% compared with T5’s default tokenizer.
  • Downstream tasks: The benchmark covers code summarization, code generation, code translation, code refinement, defect detection, and clone detection.
  • Metrics: Evaluation uses smoothed BLEU-4 for summarization, BLEU-4, exact match, and CodeBLEU for generation, and BLEU-4 or exact match for code-to-code tasks.
  • Downstream tasks: CodeT5 supports six tasks and fourteen CodeXGLUE sub-tasks with a unified encoder-decoder model.
  • Comparison models: The comparison includes encoder-only, decoder-only, and encoder-decoder models, including CodeBERT, GraphCodeBERT, GPT-2 adaptations, and PLBART.

5 Results and Analysis

CodeT5 achieves strong results across CodeXGLUE understanding and generation tasks, with identifier-aware pre-training and bimodal dual generation contributing differently by task type. Analyses show benefits for code semantics, while also exposing metric and task-length limitations.

  • Generation: CodeT5-base improves code summarization by over 1.2 absolute points over PLBART on the overall score.
  • Generation: CodeT5-base improves CodeBLEU by around 4.7 points over PLBART on code generation.
  • Metric and task analysis: BLEU can undervalue semantically correct translations, while longer code-refinement inputs are harder to fix and can inflate copy-baseline BLEU.
  • Code-to-code generation: CodeT5-base improves medium code-refinement exact match by over 4.8 points, reaching 13.96 versus GraphCodeBERT’s 9.10.
  • Understanding: CodeT5-base improves defect-detection accuracy by 2.6 points over PLBART, while clone-detection results are comparable to GraphCodeBERT and PLBART.
  • Ablations: Bimodal dual generation consistently helps summarization and generation but does not help, and can slightly hurt, PL-PL generation and understanding tasks.
  • Identifier-aware pre-training: Removing masked span prediction, identifier tagging, or masked identifier prediction generally reduces performance across the evaluated tasks.

6 Conclusion

CodeT5 is a unified encoder-decoder model for code understanding and generation that incorporates code token types, identifier-aware pre-training, and bimodal dual generation. It supports multi-task learning and outperforms prior work across most CodeXGLUE tasks while showing stronger code comprehension.

  • Contributions: CodeT5 incorporates code token type information through an identifier-aware pre-training objective and bimodal dual generation using code and comments.The model uses identifiers to better capture code semantics and learns NL-PL alignment from naturally available code-comment pairs.
  • Contributions: The unified encoder-decoder architecture supports both code understanding and generation tasks and allows multi-task learning.
  • Results: CodeT5 significantly outperforms prior work on most CodeXGLUE tasks.The experiments cover a wide range of code intelligence tasks and report state-of-the-art performance on many benchmark tasks.
  • Results: Further analysis indicates that CodeT5 has better code comprehension capability across various programming languages.

Broader Impact and Ethical Consideration

The paper discusses broader benefits and risks of applying CodeT5 to software intelligence. Its open-source training data and coding-assistance use raise concerns about embedded social biases, computational cost, automation bias, sensitive information, and vulnerable code generation.

  • Potential Benefits: Software intelligence aims to improve development productivity, programming quality, and developers’ ability to reduce repetitive workloads.
  • Risks: Training data from open-source code and comments may embed social biases, including stereotypes related to race and gender.The paper suggests filtration or modulation of generated outputs as possible mitigation interventions.
  • Risks: CodeT5 pre-training requires non-trivial computational resources despite efforts to reduce unnecessary computation costs.The paper reports that CodeT5-base has 220M parameters and that its training produced around 49.25 kg CO2, offset by the provider.
  • Mitigation: The authors release pre-trained models publicly to help the code intelligence community avoid repeated training.
  • Risks: Generated coding assistance can encourage overreliance on outputs that appear correct but conflict with developer intent.The paper recommends treating outputs as references requiring expert correctness and security checks.
  • Risks: Training on public code repositories may leave sensitive information in the model and may enable generation of vulnerable code when misused.The authors note that data cleaning may not completely remove sensitive information.
Loading 2109.00859v1…