Source-linked AI summary
CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation
Shuai Lu, Daya Guo, Shuo Ren, Junjie Huang, Alexey Svyatkovskiy, Ambrosio Blanco, Colin Clement, Dawn Drain, Daxin Jiang, Duyu Tang, Ge Li, Lidong Zhou, Linjun Shou, Long Zhou, Michele Tufano, Ming Gong, Ming Zhou, Nan Duan, Neel Sundaresan, Shao Kun Deng, Shengyu Fu, Shujie Liu
TL;DR
CodeXGLUE addresses the need for benchmarks supporting machine learning research in program understanding and generation. It provides 10 tasks across 14 datasets, an evaluation platform, and baseline systems, while remaining limited in coverage of several software-engineering tasks.
Problem
Benchmarks influence applied AI research, while growing developer populations increase the importance of AI-based code intelligence for improving development productivity.
Method
The paper introduces CodeXGLUE, combining 14 datasets, 10 programming-language understanding and generation tasks, an evaluation and comparison platform, and baseline systems.
Results
CodeXGLUE provides a benchmark spanning diverse program understanding and generation tasks, including datasets selected or created to support data-driven method development and evaluation.
Takeaways & Limitations
The benchmark is intended to support model development and evaluation across various program understanding and generation problems and to increase software-developer productivity.
Takeaways & Limitations
CodeXGLUE does not cover many machine-learning software-engineering tasks with sufficient data, including idiom mining, bug localization, test-case generation, and program synthesis.
Abstract
from arXiv · showhide
Benchmark datasets have a significant impact on accelerating research in programming language tasks. In this paper, we introduce CodeXGLUE, a benchmark dataset to foster machine learning research for program understanding and generation. CodeXGLUE includes a collection of 10 tasks across 14 datasets and a platform for model evaluation and comparison. CodeXGLUE also features three baseline systems, including the BERT-style, GPT-style, and Encoder-Decoder models, to make it easy for researchers to use the platform. The availability of such data and baselines can help the development and validation of new methods that can be applied to various program understanding and generation problems.
1 INTRODUCTION
CodeXGLUE addresses the lack of a broad benchmark suite for code intelligence by combining diverse datasets, tasks, evaluation infrastructure, and baseline models.
- The benchmark is motivated by the growing importance of AI-based code intelligence as the developer population expands.The cited passage estimates 23.9 million professional developers in 2019 and 28.7 million in 2024.
- Pretrained models improved code understanding and generation, but lacked a benchmark suite covering a wide range of tasks.
- CodeXGLUE provides 14 datasets, 10 programming-language understanding and generation tasks, and a platform for model evaluation and comparison.
- The benchmark spans code-code, text-code, code-text, and text-text tasks, including clone detection, completion, translation, search, summarization, and documentation translation.
- Eight datasets were previously proposed, while newly introduced datasets cover cloze testing, line-level completion, code translation, code search, and documentation translation.
- CodeXGLUE supplies BERT-style, GPT-style, and Encoder-Decoder baselines for understanding, completion or generation, and sequence-to-sequence tasks.
2 TASKS OVERVIEW
CodeXGLUE covers diverse program understanding and generation tasks spanning code, text, and their relationships.
- Clone detection measures semantic similarity through binary classification or retrieval of similar code.
- Defect detection identifies whether source code contains exploitable defects such as resource leaks, use-after-free vulnerabilities, or denial-of-service attacks.
- Cloze testing predicts masked code tokens either from a vocabulary or by distinguishing between “max” and “min”.
- Code completion predicts following tokens from context, with token-level and line-level subtasks.
- Code translation converts code from one programming language to another.
- Code search evaluates semantic relatedness between text queries and code through retrieval and query-code relevance prediction.
- Code repair automatically refines code by fixing bugs, while text-to-code generation produces code from natural-language descriptions.
- Code summarization generates natural-language comments for code, and documentation translation translates code documentation between natural languages.
3 DATASETS
The datasets combine established benchmarks with newly constructed resources designed to support data-driven development and evaluation across code intelligence tasks.
- CodeXGLUE selects or creates datasets whose task definitions are clear and whose volume supports data-driven machine-learning development and evaluation.
- Clone detection uses BigCloneBench for binary semantic classification and POJ-104 for retrieving programs solving the same problem.
- BigCloneBench contains over 6,000,000 true clone pairs and 260,000 false clone pairs from 10 functionalities.
- POJ-104 contains 104 problems with 500 student-written C/C++ programs per problem and supports retrieval of programs solving the same problem.
- The cloze test evaluates masked-token prediction and max-versus-min semantic understanding across six programming languages.
- CT-all selects 930 target tokens after Byte Pair Encoding and handcrafted removal of meaningless tokens.
- CT-maxmin evaluates whether models distinguish the semantic difference between “max” and “min”.
3.4 Code completion
CodeXGLUE extends token-level completion resources with line-level test sets in Python and Java, alongside parallel Java–C# code for translation.
- PY150 and the Github Java Corpus provide Python and Java data for token-level code completion, while new test sets evaluate unfinished-line completion.
- PY150 contains 150,000 Python source files collected from Github.
- The PY150 line-level test set contains 10,000 randomly selected examples with at least 15% of each file available as context.
- PY150 line-level completion uses an explicit ⟨EOL⟩ marker, with average input and output lengths of 489.11 and 6.56 tokens.
- The Github Java Corpus uses Java statement delimiters rather than ⟨EOL⟩, with average input and output lengths of 350.62 and 10.49 tokens.
- The code-translation dataset contains equivalent-functionality Java–C# pairs mined from several open-source projects.
- Paired functions are conservatively identified using matching or similar signatures, class names, and directory structures.
- The translation corpus contains 11,800 paired functions or methods, with 500 development pairs and 1,000 test pairs.
3.6 Code search
Code search covers retrieval of relevant code for natural-language queries and classification of whether a function answers a query, using CodeSearchNet AdvTest and WebQueryTest.
- Code search subtasks: Code search includes retrieval of the most relevant code for a natural-language query and binary classification of whether code answers that query.The two subtasks use CodeSearchNet AdvTest and WebQueryTest, respectively.
- CodeSearchNet AdvTest: CodeSearchNet AdvTest pairs Python functions with documentation paragraphs used as queries.The dataset is drawn from the CodeSearchNet corpus and filters examples based on parsing and documentation quality.
- CodeSearchNet AdvTest: 251,820 / 9,604 / 19,210 examples remain for CodeSearchNet AdvTest training, validation, and testing after filtering.The filtering removes unparsable code, unusually short or long documentation, special tokens, empty documentation, and non-English documentation.
- CodeSearchNet AdvTest: MRR drops from 0.809 to 0.419 for RoBERTa and from 0.869 to 0.507 for CodeBERT after function and variable names are normalized.These scores are reported for Python code search on CodeSearchNet.
- WebQueryTest: WebQueryTest addresses the mismatch between conventional documentation-based queries and real user search queries.It uses real web queries and evaluates whether a paired code function can answer each query.
3.7 Code repair
The code repair task aims to automatically fix buggy Java functions by mapping them to corresponding fixed functions, using a dataset derived from bug-fixing commits.
- Task and dataset: Code repair aims to fix bugs in code automatically.The task uses buggy Java functions as sources and corresponding fixed functions as targets.
- Task and dataset: The repair dataset extracts pre-commit buggy code and post-commit fixed code from Java-file GitHub commits associated with bug-fixing messages.The construction uses GitHub Archive, Google BigQuery, and the GitHub Compare API.
- Related generation dataset: CONCODE contains 100,000 training examples and 4,000 validation and testing examples for generating Java class member functions.Each example combines natural-language descriptions, code environments, and code snippets.
- Related summarization dataset: CodeSearchNet provides six programming languages for code summarization, with documentation drawn from publicly available non-fork GitHub repositories.The dataset is filtered to remove content unrelated to functions, including external links and HTML image tags.
3.10 Documentation translation
Documentation translation automatically translates code documentation between natural languages, focusing on low-resource language pairs with scarce parallel data.
- Task and data: Documentation translation targets code documentation translation between natural languages such as English and Chinese.The data is crawled from Microsoft Documentation and includes software and code description documents in different languages.
- Task and data: The dataset introduces multilingual translation tasks for English paired with Latvian, Danish, Norwegian, and Chinese.The stated focus is on language pairs where parallel data is scarce.
- Data filtering: Filtering removes pairs with identical source and target sentences, fewer than three words on either side, or a source-target length ratio above three.Pairs are also filtered using a word-alignment ratio threshold.
- Data filtering: Pairs with a fast_align word-alignment ratio below 0.6 are removed.This rule is part of the dataset filtering process.
- Dataset size: 43K, 19K, 44K, and 50K training sentence pairs are provided for English-Latvian, English-Danish, English-Norwegian, and English-Chinese, respectively.Each language pair also has 1K development and 1K test sentence pairs.
4 BASELINE SYSTEMS
CodeXGLUE provides BERT-style, GPT-style, and Encoder-Decoder baselines spanning program understanding and sequence-to-sequence generation tasks.
- Baseline overview: The baseline suite includes CodeBERT for program understanding, CodeGPT for completion and generation, and an Encoder-Decoder framework for sequence-to-sequence generation.The three pipelines are illustrated in Figure 6.
- CodeBERT: CodeBERT is a 12-layer Transformer encoder with 768-dimensional hidden states and 12 attention heads.It is pretrained with masked language modeling and replaced token detection on CodeSearchNet data.
- CodeGPT: CodeGPT is a 12-layer Transformer decoder with GPT-2 architecture and training objectives, pretrained on Python and Java corpora.The corpora come from CodeSearchNet, including 1.1M Python functions and 1.6M Java methods.
- CodeGPT: CodeGPT uses both from-scratch and domain-adaptive models for each programming language.The domain-adaptive model starts from GPT-2 and continues training on code, while the other learns its vocabulary and parameters from scratch.
- Encoder-Decoder: The Encoder-Decoder framework initializes its encoder with CodeBERT and uses a randomly initialized 6-layer Transformer decoder.It is provided for code repair, code translation, code summarization, and documentation translation.
5 EXPERIMENT
CodeXGLUE evaluates baseline systems across 10 programming-language understanding and generation tasks, reporting task-specific accuracy and training or inference costs. Pretrained models achieve strong results across several tasks, while structural information remains a stated avenue for improvement in some settings.
- The experiments report baseline accuracy across 10 tasks and also measure model training and inference time.
- Clone Detection: CodeBERT achieves an overall score of 90.4 on clone detection, outperforming RoBERTa despite not leveraging code structure.
- Defect Detection: CodeBERT achieves a 62.1 accuracy score for defect detection, obtaining state-of-the-art performance, although its improvement over TextCNN is limited.
- CodeGPT-adapted achieves state-of-the-art code completion with an overall score of 71.28 and text-to-code generation with a CodeBLEU score of 35.98.
- Code Summarization: CodeBERT gains 1.3% BLEU over RoBERTa for code summarization and achieves state-of-the-art performance on six programming languages.
- Documentation Translation: Pretraining improves documentation translation by 13.49 BLEU points over the Transformer Baseline, from 52.67 to 66.16 overall.
6 RELATED WORK
CodeXGLUE is positioned among diversified benchmark datasets designed to support evaluation across applications, while acknowledging that several software-engineering tasks remain outside its coverage.
- The paper separately reports results for code repair, code translation, code summarization, and documentation translation.
- Diversified benchmarks such as ImageNet, GLUE, XTREME, and XGLUE support testing model generalization across broad application areas.
- CodeXGLUE is presented as the first diversified benchmark dataset applicable to various code intelligence problems.
- CodeXGLUE does not cover several data-supported software-engineering tasks, including idiom mining, bug localization, test-case generation, and program synthesis.
7 CONCLUSION
The paper aims to support models for diverse program understanding and generation problems and plans to expand CodeXGLUE's languages, tasks, and pretrained-model research.
- CodeXGLUE seeks to support models applicable to various program understanding and generation problems, with the goal of increasing developer productivity.
- Future work includes adding programming languages and downstream tasks while exploring new structures for advanced pretrained models.