Source-linked AI summary
CodeBERT: A Pre-Trained Model for Programming and Natural Languages
Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, Ming Zhou
TL;DR
Connecting natural language with programming language requires representations that support both understanding and generation tasks. CodeBERT combines bimodal and unimodal training with a hybrid objective, achieving state-of-the-art results on code search and code documentation generation and outperforming prior models on NL-PL probing.
Problem
The paper addresses the need for general-purpose representations connecting natural language and programming language across understanding and generation tasks.
Method
CodeBERT jointly trains on bimodal NL-PL pairs and unimodal code using a Transformer and hybrid masked-language-modeling and replaced-token-detection objectives.
Results
CodeBERT achieves state-of-the-art performance on natural language code search and code documentation generation, and outperforms RoBERTa and a code-only model on fixed-parameter NL-PL probing.
Takeaways & Limitations
CodeBERT provides a strong pretrained foundation for NL-PL understanding, generation, and zero-shot probing across programming languages.
Takeaways & Limitations
CodeBERT’s objectives mainly target NL-PL understanding, leaving generation-related learning objectives as a potential improvement direction.
Abstract
from arXiv · showhide
We present CodeBERT, a bimodal pre-trained model for programming language (PL) and nat-ural language (NL). CodeBERT learns general-purpose representations that support downstream NL-PL applications such as natural language codesearch, code documentation generation, etc. We develop CodeBERT with Transformer-based neural architecture, and train it with a hybrid objective function that incorporates the pre-training task of replaced token detection, which is to detect plausible alternatives sampled from generators. This enables us to utilize both bimodal data of NL-PL pairs and unimodal data, where the former provides input tokens for model training while the latter helps to learn better generators. We evaluate CodeBERT on two NL-PL applications by fine-tuning model parameters. Results show that CodeBERT achieves state-of-the-art performance on both natural language code search and code documentation generation tasks. Furthermore, to investigate what type of knowledge is learned in CodeBERT, we construct a dataset for NL-PL probing, and evaluate in a zero-shot setting where parameters of pre-trained models are fixed. Results show that CodeBERT performs better than previous pre-trained models on NL-PL probing.
1 Introduction
CodeBERT is a bimodal pre-trained model for natural and programming languages that captures their semantic connection and supports NL-PL understanding and generation tasks. It is trained across six programming languages and evaluated on code search, code-to-text generation, and probing.
- Model: CodeBERT is a bimodal pre-trained model for multiple programming languages that captures semantic connections between natural and programming language.It produces general-purpose representations for NL-PL understanding and generation tasks.
- Training: Training uses GitHub repositories spanning six programming languages, with bimodal data pairing code and function-level natural-language documentation.One model is learned for all six languages without explicit input-language markers.
- Evaluation: CodeBERT is effective on natural-language code search and code-to-text generation tasks.The introduction identifies code search and code documentation generation as representative downstream applications.
- Probing: The work creates the first dataset for investigating the probing ability of code-based pre-trained models.The dataset supports probing the knowledge learned by such models.
2 Background
Large pre-trained models have substantially improved NLP through self-supervised learning on large-scale plain text, while multimodal pre-training extends this paradigm to aligned inputs from different modalities. CodeBERT is distinguished from prior code pre-training by its cross-modal training, use of bimodal and unimodal data, and coverage of six programming languages.
- Pre-trained models: Large pre-trained models have brought dramatic empirical improvements across almost every NLP task through self-supervised learning on large-scale plain texts.Self-supervised annotations are automatically collected from raw data without manual annotation.
- Pre-training objectives: Language modeling and its variations dominate pre-training objectives, including GPT’s next-word prediction from preceding context.Masked language modeling is motivated by using both preceding and following contexts for better general-purpose contextual representations.
- Multimodal pre-training: Multimodal pre-trained models learn implicit alignment between different input modalities, typically from paired language-image or language-video data.ViLBERT learns from image-caption data by reconstructing masked image-region categories or masked words from observed inputs.
- Code pre-training: CodeBERT differs from concurrent Python-code pre-training by using cross-modal training, combining bimodal NL-PL with unimodal PL/NL data, and covering six programming languages.The concurrent work uses masked language modeling and next sentence prediction on Python source code, with each sentence defined as a logical code line.
3 CodeBERT
CodeBERT uses a RoBERTa-base Transformer architecture with 125M parameters and represents both individual NL/code tokens and whole input sequences. It is pretrained on bimodal and unimodal data using MLM and RTD objectives, then adapted to downstream NL-PL tasks through task-specific representations or encoder initialization.
- Architecture: CodeBERT uses the same multi-layer bidirectional Transformer architecture as RoBERTa-base, with 125M total parameters.The architecture follows BERT and RoBERTa.
- Representations: Its pre-training input concatenates natural-language and code segments separated by special tokens, producing contextual token vectors and an aggregated [CLS] representation.The [CLS] representation supports classification or ranking.
- Training Data: Training combines 2.1M bimodal NL-code datapoints with 6.4M unimodal codes across Python, Java, JavaScript, PHP, Ruby, and Go.Bimodal datapoints are functions paired with documentation, while unimodal codes are functions without paired documentation.
- Pre-training Objectives: Masked language modeling predicts masked tokens from NL-code pairs, with 15% of input tokens masked during training.The MLM objective is applied to both natural-language and programming-language tokens.
- Pre-training Objectives: Replaced token detection classifies every input position as original or replaced, enabling training with both bimodal and unimodal data.NL and PL n-gram generators with bidirectional contexts are learned from corresponding unimodal data.
- Downstream Use: For downstream tasks, CodeBERT’s [CLS] representation measures query-code relevance for code search, while its encoder initializes an encoder-decoder model for code-to-text generation.These settings use the same concatenated input format for code search and an encoder-decoder framework for generation.
4 Experiment
The experiments evaluate CodeBERT through fine-tuned code search and generation tasks, zero-shot NL-PL probing, and generalization to an unseen programming language. CodeBERT outperforms key baselines across these evaluations, including achieving state-of-the-art documentation generation performance.
- Experiment design: The evaluation covers fine-tuned natural-language code search, zero-shot NL-PL probing with fixed parameters, code documentation generation, and an unseen programming language.The experiments use CodeSearchNet and assess both downstream performance and knowledge encoded without parameter updates.
- 4.1 Natural language code search: CodeBERT consistently outperforms RoBERTa and code-only pre-trained models on natural-language code search, while RoBERTa initialization further improves performance.Code search is evaluated using macro-average Mean Reciprocal Rank over 999 distractor codes.
- 4.2 NL-PL probing: CodeBERT performs better than baselines on almost all programming languages in both NL and PL probing, with bidirectional contexts outperforming preceding-only contexts.The probing task masks a code or word token and requires recovering it among distractor candidates; CodeBERT also correctly predicts both masked tokens in the reported Python case study.
- 4.3 Code documentation generation: CodeBERT models pre-trained with RTD and MLM outperform RoBERTa for code-to-documentation generation and achieve a 1.3 BLEU score gain overall.The passage reports state-of-the-art performance for this generation task.
- 4.4 Unseen programming language: 22.36 BLEU score is achieved on an unseen programming language, improving by 2.55 points over RoBERTa but remaining slightly below code2seq.The comparison attributes code2seq’s advantage partly to its use of compositional paths in the abstract syntax tree, which CodeBERT does not use.
5 Conclusion
CodeBERT is presented as a large bimodal pre-trained model for natural and programming languages, achieving state-of-the-art downstream performance and outperforming prior models on fixed-parameter NL-PL probing. The paper identifies improvements to generators, generation objectives, AST integration, and domain/language adaptation as future directions.
- Contributions: CodeBERT is presented as the first large bimodal pre-trained model for natural language and programming language.It is trained on both bimodal and unimodal data.
- Contributions: Fine-tuning CodeBERT achieves state-of-the-art performance on natural language code search and code-to-documentation generation.
- Probing: With parameters fixed, CodeBERT performs better than RoBERTa and a continuously trained codes-only model on the probing task.The probing task is treated as a cloze-style answer-selection problem with distractors for both NL and PL.
- Future work: Future work includes better generators, generation-related objectives, AST integration, broader NL-PL tasks, more programming languages, and flexible domain/language adaptation.These directions target replaced token detection, code-documentation generation, and generalization across domains and languages.
A Data Statistic · B Train Details
The section identifies the CodeSearchNet Corpus as the basis for natural-language code-search data statistics, covering training, validation, and testing splits across six programming languages. No train-detail content is provided in the supplied passages.
- A Data Statistic: Table 6 reports training, validation, and testing data statistics for six programming languages.The supplied paragraph directs readers to Table 6 for these split statistics.
- A Data Statistic: The supplied passages do not specify the numerical values in Table 6.Only the table’s purpose and coverage are provided here.
- A Data Statistic: The reported statistics concern the CodeSearchNet Corpus for natural language code search.This is the scope stated in Table 6’s caption.
B.1 Pre-training · B.2 CodeSearch · B.3 Code Summarization on Six Programming Languages
CodeBERT’s experiments use specified pre-training and fine-tuning configurations for CodeSearch and code summarization. The summarization decoder and optimization setup are fixed across six programming languages, with development-set selection or early stopping.
- B.1 Pre-training: Pre-training uses batch size 2,048, learning rate 5e-4, Adam, 10K warmup steps, maximum length 512, and 100K training steps.
- B.1 Pre-training: Training 1,000 batches takes 600 minutes with MLM and 120 minutes with RTD.
- B.2 CodeSearch: CodeSearch fine-tuning uses learning rate 1e-5, batch size 64, maximum sequence length 200, and up to 8 epochs with Adam.The model performing best on the development set is evaluated on the test set.
- B.3 Code Summarization on Six Programming Languages: Code summarization uses maximum input and inference lengths of 256 and 64, respectively, with Adam, learning rate 5e-5, and batch size 64.Hyperparameters are tuned and early stopping is performed on the development set.
B.4 Code Summarization on C#
For C# code summarization, the comparison uses a 2-layer attention-based GRU decoder and selects hyperparameters by grid search based on development-set performance.
- The comparison uses a 2-layer GRU decoder with an attention mechanism, following state-of-the-art methods that use RNN decoders.
- The grid search varies batch size over {32, 64} and learning rate over {2e-5, 5e-5}, reporting settings that perform best on the development set.
C Learning Curve of CodeSearch
CodeBERT performs better during the early fine-tuning stage on the Python and Java learning curves, indicating that it provides good initialization for downstream tasks.
- C Learning Curve of CodeSearch: CodeBERT performs better at the early fine-tuning stage, providing good initialization for downstream tasks.This result is shown in Figure 4’s learning curves.
- C Learning Curve of CodeSearch: Figure 4 compares learning curves for different pre-trained models on Python and Java.
D Late Fusion
CodeBERT is evaluated as a unified encoder for late-fusion natural-language code search by separately encoding natural language and code, then ranking codes through dot-product similarity. It outperforms RoBERTa and code-only pre-training, while remaining comparable to standard fusion and offering greater efficiency for online systems.
- Late-fusion setup: CodeBERT separately encodes natural language and code, then uses dot-product similarity to retrieve nearest codes in a shared vector space.This evaluates CodeBERT as a unified encoder in the late-fusion setting.
- Late-fusion setup: Fine-tuning maximizes the ground-truth dot product while minimizing dot products with distractor codes.The objective directly trains the model to distinguish the correct code from alternatives.
- Evaluation: The late-fusion experiment is conducted on two programming languages with relatively small amounts of data.Results are reported in Table 7 for natural-language code search by late fusion.
- Results: CodeBERT performs better than RoBERTa and the model pre-trained with codes only.The comparison concerns natural-language code search under the late-fusion setting.
- Results: Late fusion performs comparably to the standard approach while being more efficient and suitable for online systems.The separate encoding and similarity calculation support more efficient deployment.
E Case Study
The case study qualitatively examines CodeBERT on code search and code documentation generation. Python search results are highly relevant, and CodeBERT outperforms all baselines in documentation generation examples.
- E Case Study: The case study qualitatively evaluates CodeBERT on code search and code documentation generation tasks.The analysis presents illustrative cases rather than additional quantitative evaluations.
- E Case Study: The Python code search example reports highly relevant results for the query, showing only the top2 results because of limited space.The results were searched from 1,156,085 Python code data.
- E Case Study: CodeBERT performs better than all baselines in Java and Python code documentation generation examples.The examples are shown in Figure 6 for Java and Figure 7 for Python.