PaperlayerView exact revision on arXiv

Source-linked AI summary

CoTexT: Multi-task Learning with Code-Text Transformer

Long Phan, Hieu Tran, Daniel Le, Hieu Nguyen, James Anibal, Alec Peltekian, Yanfang Ye

arXiv:2105.08645v4cs.AIcs.PL

TL;DR

CoTexT addresses how to learn shared representations across natural and programming languages for code intelligence. It uses a T5-based encoder-decoder trained with self-supervision on multilingual unimodal and bimodal code data, then evaluates four CodeXGLUE tasks. The paper reports state-of-the-art results across Code Summarization, Code Generation, Defect Detection, and Code Refinement.

  • Problem

    CoTexT targets the need for pre-trained representations that support both natural-language and programming-language understanding and generation.

  • Method

    CoTexT adapts a T5 encoder-decoder and pre-trains it on large multilingual programming-language corpora using unimodal and bimodal data.

  • Results

    CoTexT achieves state-of-the-art results on CodeSummarization, CodeGeneration, Defect Detection, and Code Refinement in CodeXGLUE.

  • Takeaways & Limitations

    The results support CoTexT as a versatile model for diverse natural-language and programming-language code-intelligence tasks.

Abstract

from arXiv · show

We present CoTexT, a pre-trained, transformer-based encoder-decoder model that learns the representative context between natural language (NL) and programming language (PL). Using self-supervision, CoTexT is pre-trained on large programming language corpora to learn a general understanding of language and code. CoTexT supports downstream NL-PL tasks such as code summarizing/documentation, code generation, defect detection, and code debugging. We train CoTexT on different combinations of available PL corpus including both "bimodal" and "unimodal" data. Here, bimodal data is the combination of text and corresponding code snippets, whereas unimodal data is merely code snippets. We first evaluate CoTexT with multi-task learning: we perform Code Summarization on 6 different programming languages and Code Refinement on both small and medium size featured in the CodeXGLUE dataset. We further conduct extensive experiments to investigate CoTexT on other tasks within the CodeXGlue dataset, including Code Generation and Defect Detection. We consistently achieve SOTA results in these tasks, demonstrating the versatility of our models.

1 Introduction

CoTexT is introduced as a T5-style pre-trained encoder-decoder for natural and programming languages. It is trained on multilingual code corpora and evaluated across CodeXGLUE tasks, where the authors report state-of-the-art results.

  • Model and scope: CoTexT is a pre-trained encoder-decoder model for natural language and programming languages, adapted to the T5 framework.The model uses attention mechanisms and is evaluated through multi-task learning across multiple programming languages and related tasks.
  • Training data: The authors train CoTexT on multilingual programming-language corpora using different combinations of unimodal and bimodal data.The evaluated languages include Java, Python, JavaScript, and Ruby; bimodal data pairs text with code, while unimodal data contains code only.
  • Evaluation: CoTexT is fine-tuned on CodeXGLUE CodeSummarization, CodeGeneration, Defect Detection, and Code Refinement tasks.Code Refinement is evaluated on both small and medium datasets.
  • Results: The paper reports state-of-the-art results for the four evaluated CodeXGLUE tasks.The contribution includes three CoTexT versions and publicly released pre-trained checkpoints and source code.

2 Related Work

Related work adapts pre-trained language modeling to specialized domains and code. CodeBERT, GraphCode-BERT, and PLBART extend this direction through paired NL-PL data, code structure, or sequence-to-sequence denoising.

  • Domain adaptation: Domain-adapted models such as BioBERT and SciBERT improve performance on biomedical and computer-science text compared with general BERT.These models are further trained on domain-specific corpora.
  • Code pre-training: CodeBERT learns general-purpose natural-language and programming-language representations from bimodal NL-PL pairs.Its training strategy combines natural-language text with corresponding code.
  • Code pre-training: GraphCode-BERT extends CodeBERT by using data flow during pre-training to capture semantic-level code structure.The distinction is between syntactic-level structure and semantic-level structure.
  • Sequence-to-sequence models: PLBART is a pre-trained sequence-to-sequence model that uses denoising autoencoding for NL-PL understanding and generation.The paper positions PLBART as a recent related model for both understanding and generation tasks.

3 CoTexT

CoTexT uses a T5-based sequence-to-sequence design for code and text, with special handling for code tokens and self-supervised training over bimodal and unimodal corpora. Task prefixes and masked-span objectives support multiple downstream directions.

  • 3.1 Vocabulary: CoTexT uses SentencePiece vocabulary modeling but encodes code-specific out-of-vocabulary tokens into natural-language representations.This treatment is applied during both self-supervised and supervised training to preserve representative code context.
  • 3.2 Pre-training data: Bimodal pre-training combines code with corresponding natural-language text, whereas unimodal pre-training uses code sequences alone.The corpora include CodeSearchNet and GitHub repository data, with multiple programming languages represented.
  • 3.3 Input/Output Representations: During self-supervision, CoTexT masks input spans and forms target sequences from the corresponding sentinel-token representation.The model uses input and target sequences for both pre-training and supervised training.
  • 3.4 Model training: CoTexT uses a sequence-to-sequence encoder-decoder initialized from the 220-million-parameter Base T5 model.The reported training setup uses a 0.001 learning rate, 1024-token input and target lengths, and batch size 128.
  • Multi-task learning: Task-specific prefixes identify the intended task, including programming-language prefixes for CodeSummarization.The same maximum-likelihood teacher-forcing objective supports text-code and code-text generation.

4 Experiments

The experiments use CodeXGLUE, a benchmark spanning code intelligence tasks and evaluation formats. CoTexT is evaluated on selected summarization, generation, refinement, and defect-detection tasks.

  • Benchmark: CodeXGLUE provides code intelligence tasks, model evaluation, and leaderboard comparison across classification and generation settings.Its ten tasks include code-text, text-code, code-code, and text-text scenarios.
  • Evaluated tasks: The study focuses on Code Summarization, Code Generation, Code Refinement, and Defect Detection within CodeXGLUE.The evaluation datasets are summarized in Table 3.

4.1 CodeXGLUE

CodeXGLUE is a benchmark for code understanding and generation that provides diverse tasks, evaluation, and leaderboard comparison.

  • CodeXGLUE facilitates machine learning studies on code understanding and code generation problems.
  • The benchmark includes 10 code intelligence tasks spanning code-text, text-code, code-code, and text-text scenarios.
  • CoTexT focuses on Code Summarization, Code Generation, Code Refinement, and Defect Detection within CodeXGLUE.

4.2 Evaluation Tasks

The evaluation covers generation, repair, and vulnerability classification tasks across multiple programming languages and dataset settings.

  • Evaluation Setup: The evaluation uses TPU v2-8 with settings from the original T5 model, while task-specific input and target lengths are summarized in Table 2.
  • Code Summarization: Code Summarization generates a natural-language description for code snippets across Python, Java, Javascript, PHP, Ruby, and Go.
  • Code Generation: Text-to-Code Generation produces Java functions from natural-language descriptions, code environments, and Javadoc-style method comments.
  • Code Refinement: Code Refinement automatically corrects bugs in Java functions using SMALL functions under 50 tokens and MEDIUM functions containing 50–100 tokens.
  • Defect Detection: Defect Detection classifies whether C code snippets contain vulnerabilities such as resource leaks or denial-of-service risks.

4.3 Experimental Setup

The experimental setup compares CoTexT with established code-language models using task-specific metrics and CodeXGLUE dataset statistics.

  • Baselines: The baselines include CodeGPT, CodeGPT-adapted, CodeBERT, and PLBART, representing autoregressive, masked-language, and sequence-to-sequence approaches.
  • Baselines: CodeGPT is trained from scratch on CodeSearchNet, whereas CodeGPT-adapted begins from a GPT-2 checkpoint.
  • Baselines: CodeBERT combines masked language modeling with replaced token detection, while PLBART uses token masking, deletion, and infilling.
  • Performance Metrics: Smooth BLEU-4 evaluates Code Summarization, and corpus-level BLEU evaluates the remaining tasks.
  • Performance Metrics: CodeBLEU measures code quality using syntactic and semantic features derived from abstract syntax trees and data-flow structure.
  • Performance Metrics: Accuracy is defined as the ratio of generated sequences matching the reference to the total number of observations.

5 Results

CoTexT is evaluated across multi-task and single-task CodeXGLUE settings, covering code summarization, refinement, generation, and defect detection. It achieves strong or state-of-the-art results across these tasks, with especially notable gains in code refinement and defect detection.

  • Multi-Task Learning: Code Summarization uses T5-based multi-task learning across Ruby, JavaScript, Go, Python, Java, and PHP.The results are reported in Table 5.
  • Code Summarization: Base T5 achieves higher overall BLEU-4 results than other related CodeXGLUE leaderboard models, highlighting the importance of domain-specific pretraining.
  • Code Summarization: CoTexT achieves SOTA on overall, Python-specific, Java-specific, and Go-specific Code Summarization scores while remaining competitive on other languages.The authors attribute this pattern to larger Python and Java training sets.
  • Multi-Task Learning: Code Refinement uses task-specific prefixes for both small and medium test sets in the multi-task setup.
  • Code Refinement: CoTexT improves significantly over natural-language-only base T5 across Code Refinement metrics, achieving SOTA on all small-test metrics and medium-test accuracy.
  • Single-Task Learning: CoTexT achieves SOTA on Java Code Generation across Exact Match, BLEU, and CodeBLEU, with only slight per-metric gains over competing models.CoTexT and CodeGPT-adapted both achieve 20.10 for Exact Match.
  • Single-Task Learning: CoTexT outperforms PLBART by 3.44% on Defect Detection and achieves SOTA despite C being absent from its training data.The result is attributed to understanding similar programming languages.

6 Conclusion

The paper introduces CoTexT as a pretrained representation for natural and programming languages. Using the T5 framework and large programming-language corpora, it achieves state-of-the-art results on four CodeXGLUE code-intelligence tasks.

  • CoTexT is a pretrained language representation for both programming language and natural language, focused on text-code and code-text understanding and generation.
  • Pretraining on a large programming-language corpus is effective across Code Summarization, Code Generation, Code Refinement, and Code Detection.
  • CoTexT achieves state-of-the-art results on four CodeXGLUE code-intelligence tasks.
  • Future work will test CoTexT on broader programming-language and natural-language generation tasks, including autocompletion and code translation.
Loading 2105.08645v4…