Source-linked AI summary
CodeNet: A Large-Scale AI for Code Dataset for Learning a Diversity of Coding Tasks
Ruchir Puri, David S. Kung, Geert Janssen, Wei Zhang, Giacomo Domeniconi, Vladimir Zolotov, Julian Dolby, Jie Chen, Mihir Choudhury, Lindsey Decker, Veronika Thost, Luca Buratti, Saurabh Pujar, Shyam Ramji, Ulrich Finkler, Susan Malaika, Frederick Reiss
TL;DR
Modernizing aging software and improving development productivity motivates AI-for-Code research, which still needs broad, high-quality data. The paper presents CodeNet, a large, diverse, richly annotated dataset with tools and evaluations spanning multiple coding tasks. Its scale and annotations are intended to support research in code similarity, classification, translation, and performance improvement.
Problem
Aging enterprise codebases create difficult modernization and maintenance tasks, motivating better tools and datasets for AI-assisted software development.
Method
The paper constructs CodeNet, a curated dataset with extensive metadata, annotations, preprocessing tools, and customizable access for source-code research.
Results
CodeNet supports benchmarking diverse tasks including code similarity, classification, translation, and code-performance improvement, with reference experiments in classification and similarity.
Takeaways & Limitations
CodeNet's scale, diversity, and rich annotations provide research opportunities across AI for Code and software engineering.
Takeaways & Limitations
CodeNet may challenge techniques that rely heavily on comments because samples may be sparsely commented and comments may use multiple languages.
Abstract
from arXiv · showhide
Over the last several decades, software has been woven into the fabric of every aspect of our society. As software development surges and code infrastructure of enterprise applications ages, it is now more critical than ever to increase software development productivity and modernize legacy applications. Advances in deep learning and machine learning algorithms have enabled numerous breakthroughs, motivating researchers to leverage AI techniques to improve software development efficiency. Thus, the fast-emerging research area of AI for Code has garnered new interest and gathered momentum. In this paper, we present a large-scale dataset CodeNet, consisting of over 14 million code samples and about 500 million lines of code in 55 different programming languages, which is aimed at teaching AI to code. In addition to its large scale, CodeNet has a rich set of high-quality annotations to benchmark and help accelerate research in AI techniques for a variety of critical coding tasks, including code similarity and classification, code translation between a large variety of programming languages, and code performance (runtime and memory) improvement techniques. Additionally, CodeNet provides sample input and output test sets for 98.5% of the code samples, which can be used as an oracle for determining code correctness and potentially guide reinforcement learning for code quality improvements. As a usability feature, we provide several pre-processing tools in CodeNet to transform source code into representations that can be readily used as inputs into machine learning models. Results of code classification and code similarity experiments using the CodeNet dataset are provided as a reference. We hope that the scale, diversity and rich, high-quality annotations of CodeNet will offer unprecedented research opportunities at the intersection of AI and Software Engineering.
1 Introduction
AI is increasingly being applied to software engineering, but the quality and usefulness of AI-generated code remain open questions. CodeNet is presented as a large, diverse, high-quality dataset intended to accelerate AI-for-Code research and support developer productivity.
- Motivation: AI-based tools can manipulate and generate code, but their ability to do so with high quality remains an open question.Recent deep-learning models sometimes generate surprisingly plausible code after training on vast datasets.
- Motivation: AI could augment traditional coding tools by refining heuristics, prioritizing among sound answers, and handling incomplete or invalid code more robustly.These capabilities could expand the scope of software-development tool support.
- Problem: Enterprise developers must maintain aging codebases by modernizing them, fixing bugs, improving performance, adapting requirements, strengthening security, and meeting regulations.These tasks are difficult and create a need for tools that improve developer productivity.
- Contribution: CodeNet is introduced as a first-of-its-kind dataset in scale, diversity, and quality for accelerating algorithmic advances in AI for Code.Its design follows the broader role of best-of-breed datasets in enabling advances in deep-learning research.
2 The CodeNet Dataset
CodeNet is a large, curated collection of programming submissions with extensive metadata and documented tools for dataset access and code transformation. It spans many problems, languages, submission outcomes, and representations, while its educational focus limits suitability for enterprise-oriented code research.
- Usability features: The dataset provides documented preprocessing and derived resources, including near-duplicate information and token sequences for machine-learning inputs.Its directory structure separates source data, metadata, problem descriptions, and derived information.
- Dataset construction: CodeNet is derived from AIZU and AtCoder online judges, whose automated review provides submission outcomes and associated metadata.Problem descriptions and metadata are available through REST APIs.
- Scale and diversity: CodeNet contains 13,916,868 submissions across 4053 problems and 55 programming languages.Most submissions use C++, Python, Java, C, Ruby, or C#; C++ accounts for 57% of submissions.
- Scale and diversity: The dataset includes pedagogical problems ranging from elementary exercises to advanced algorithmic tasks, submitted by beginners and experienced coders.Submissions include both correct and error-containing solutions, with outcomes labeled.
- Code samples: Each code sample is a single source file that includes test-case input and computed-result output, usually containing one function.More complex problem submissions may contain several functions, and standard file extensions identify languages.
- Metadata: Metadata uses dataset-level and problem-level hierarchies to support queries and selections across problems, languages, and source files.Data and metadata are separated, and the dataset-level metadata records problem origins, CPU limits, and memory limits.
- Limitations: CodeNet may challenge comment-dependent techniques because samples are not extensively commented and comments may use multiple languages.This limitation specifically affects methods relying on a preponderance of comments in code.
- Limitations: The dataset contains solutions to high-school and beginning-college programming problems, making it unsuitable for enterprise APIs and advanced design patterns.This scope boundary limits its use for enterprise-oriented code research.
3 Related Datasets
The section reviews source-code datasets for tasks including clone detection, vulnerability detection, translation, search, generation, and summarization, then highlights limitations of POJ-104 and GCJ.
- Source-code datasets target tasks such as clone detection, vulnerability detection, code completion, repair, translation, search, generation, and summarization.
- POJ-104: POJ-104 contains 52,000 C and C++ submissions for 104 pedagogical programming problems and supports code classification and similarity research.
- POJ-104: POJ-104 does not distinguish C from C++, omits judging metadata, lacks problem statements, and provides no execution information.
- POJ-104: Some POJ-104 problems are identical and some submissions are near duplicates, although such cases are relatively uncommon compared with other datasets.
- GCJ: GCJ spans 2008–2020 competitions and multiple languages, while its GCJ-297 benchmark has 297 problems and approximately 280K submissions with imbalanced problem counts.
- GCJ: GCJ is larger and more language-diverse than POJ-104 but lacks metadata and information about identical problems and near duplicates.
4 CodeNet Differentiation
CodeNet differentiates itself through scale, rich annotations, and duplicate-aware cleaning, addressing dataset properties needed for broad and reliable source-code research.
- CodeNet is designed around desired high-quality dataset properties and is compared with related datasets in Table 5.
- Large scale: CodeNet contains approximately 10 times more code samples than GCJ, while its C++ benchmark is approximately 10 times larger than POJ-104.
- Rich annotation: CodeNet includes correctness outcomes, error categories, problem statements, sample inputs, and sample outputs that are absent from GCJ and POJ-104.
- Clean samples: CodeNet identifies near-duplicate code and clusters identical problems, making this information available for dataset use while it is absent from GCJ and POJ-104.
5 Construction of CodeNet
CodeNet combines submissions from AIZU and AtCoder, standardizes raw files, detects code and problem duplicates, and filters samples for customized benchmarks.
- CodeNet aggregates problems, submissions, and metadata from AIZU and AtCoder, using APIs for AIZU and web scraping for AtCoder.
- Raw data from different sources is standardized with UTF-8 encoding, removed byte-order marks, and Unix-style line endings.
- Cleansing: Near duplicates are scored with set and multiset Jaccard indices, using thresholds of 0.9 and 0.8 respectively after excluding comments and preprocessor directives.
- Cleansing: Identical problem descriptions are found with fdupes, while graphs connect potential duplicates whose near-duplicate code-pair counts exceed a threshold; descriptions are manually verified.
- Benchmarks: CodeNet benchmarks in C++, Python, and Java select unique problems and samples, with C++ benchmark sizes chosen to make the task challenging.
6 Code Representation and Tools
CodeNet supplies preprocessing tools that convert source code into machine-learning representations, including token streams, simplified parse trees, and code graphs.
- CodeNet provides preprocessing tools that transform source code into representations suitable as machine-learning inputs.
- Tokenizer: Fast C tokenizers are offered for C, C++, Java, Python, and JavaScript, while the parse-tree generator produces token streams for four languages.
- Simplified Parse Tree (SPT): Simplified parse trees remove single-child internal nodes from ANTLR4 parse trees while retaining essential structure and use an extensible JSON graph schema.
- Simplified Parse Tree (SPT): SPT nodes encode node type, applicable token or parsing-rule type, and reserved-word status.
- Code graphs: Code graphs use WALA to generate system dependence graphs and inter-procedural control-flow graphs expressing program relationships.
7 CodeNet Challenge
CodeNet’s launch gained rapid community attention, and the authors plan challenges to sustain adoption and innovation in AI for code.
- Within three months, CodeNet’s GitHub repository received 1000 stars and over 119 forks.
- The authors envision CodeNet as an umbrella for curating AI-for-code datasets and driving widespread adoption.
- The planned CodeNet challenges aim to build on the project’s momentum and create excitement in the AI community.
8 Experiments with the CodeNet Dataset
The paper establishes baseline CodeNet experiments across classification and evaluates representations ranging from token statistics to sequences, transformers, and graphs. Token order substantially improves classification, while C-BERT and GNNs provide stronger or competitive performance with language-dependent differences.
- Experiments: The experiments cover code classification, similarity, generalization, and token inference using four CodeNet-derived benchmark datasets.
- Classification setup: Classification assigns each programming problem to a class and evaluates models using training, validation, and held-out test splits.
- Models: The study compares bag-of-tokens, token-sequence CNNs, C-BERT, SPT-based GNNs, and code-graph GNNs.
- Classification results: Approximately 90% accuracy is reached across all benchmarks by CNNs using token sequences, substantially exceeding the simpler bag-of-tokens representation.
- Classification results: Approximately 97% accuracy is achieved by C-BERT on both Java and Python benchmarks, while its performance is less impressive on the C++ benchmarks.
- Classification results: GNNs are consistently competitive, with code graphs slightly improving over SPTs on Java but performing worse on Python.
- Bag-of-tokens baseline: 64.50±0.13% test accuracy is obtained on C++1400 with the bag-of-tokens MLP, versus 0.071% for random guessing.
- Token-sequence CNN: 93.71±0.18% test accuracy is obtained on C++1400 with the token-sequence CNN, significantly exceeding the bag-of-tokens approach.
8.2 Code Similarity
The similarity experiments define similarity functionally, compare token-based and graph-based models, and show that richer learned representations outperform primitive token-frequency features. Results vary with graph structure, node attributes, and benchmark composition.
- Task definition: Two code samples are considered similar when they solve the same problem, rather than merely resembling each other textually.
- Methods: The study compares concatenated bag-of-tokens vectors, a Siamese CNN model over token sequences, and SPT-based approaches including AROMA and MISIM.
- Token-based results: The Siamese token-sequence model significantly outperforms bag of tokens, whose similarity accuracy is modest on balanced binary classification.
- SPT-based results: MISIM outperforms rule-based AROMA because MISIM learns feature extraction through supervised neural training.
- SPT-based results: Across Java250 test sets, GMN outperforms MISIM using only SPT structure, and adding MISIM node attributes further improves GMN’s score.
- Generalization: Figure 6 reports a 12% higher POJ-104 test score for a model trained on C++1000 than for one trained on GCJ-297, despite GCJ-297’s 10% validation advantage.
- Token-based results: The bag-of-tokens model achieves rather modest accuracy below 87% across all benchmark datasets.
Siamese Network with Token Sequence
The paper evaluates Siamese and graph-based code-similarity models, masked language modeling, and cross-dataset generalization using CodeNet benchmarks.
- Similarity architecture: A Siamese network processes two source-code token sequences with shared weights and compares their outputs using absolute difference.Identical inputs produce identical outputs, guaranteeing detection of similarity for identical samples.
- Similarity results: 96.56±0.07% test accuracy is achieved on the C++1400 similarity benchmark despite removing identifiers, comments, and many keywords from token sequences.The authors report similar accuracy for other benchmarks.
- SPT-based experiments: SPT-based experiments separate training, validation, and test sets by programming problem, using 50%, 25%, and 25% of problems respectively.Similarity scores are reported as mean and standard deviation of MAP@R across five random seeds.
- Graph models: GMN experiments compare structural similarity from SPT graphs with models that additionally use node attributes.The attribute-enhanced model replaces the node encoder with an embedding layer and explores feature dimensions from 64 to 128.
- Evaluation limitation: MAP@R evaluation is computationally expensive for GMN models, so Table 14 reports results on smaller sampled test sets.Computing embeddings for every SPT pair in the test set creates the bottleneck.
- Masked language modeling: A masked language model randomly masks one source token and predicts the original token as its label, achieving top-1 accuracy of 0.9104.The experiment uses fixed-length tokenized inputs and evaluates top-1 and top-5 predictions.
- Cross-dataset generalization: On POJ-104, the model trained on C++1000 achieves a 12% higher score than the model trained on GCJ-297, despite GCJ-297’s 10% higher validation score.The paper attributes this pattern mainly to GCJ-297’s submission imbalance and CodeNet cleaning and deduplication.
9 Further Uses of CodeNet
CodeNet’s metadata, acceptance labels, test inputs, and language diversity support multiple software-engineering tasks beyond the demonstrated benchmarks.
- Search and repair: CodeNet’s problem-submission relationships support type-4 similarity applications such as code search and clone detection.Acceptance labels also enable extracting buggy and fixed code pairs for code repair.
- Performance analysis: Input-bearing samples allow execution-based extraction of CPU runtime and memory-footprint information for performance-related research.These measurements extend CodeNet beyond purely syntactic or structural code analysis.
- Program translation: CodeNet’s programs in many languages support neural program translation for porting legacy codebases to modern languages.The stated practical goals are increased accessibility and lower maintenance costs.
10 Conclusion
The paper presents CodeNet as a large, diverse, high-quality dataset for advancing AI for Code across several coding tasks. Its annotations and breadth are intended to create research opportunities at the intersection of AI and software engineering.
- Contribution: CodeNet is presented as a very large-scale, diverse, high-quality dataset for accelerating algorithmic advances in AI for Code.The paper emphasizes both dataset scale and diversity.
- Supported tasks: The dataset supports benchmarking code similarity, classification, translation across programming languages, and code performance improvement.These tasks connect the dataset to code recommendation and related software-engineering applications.
- Outlook: The authors expect CodeNet’s scale, diversity, and rich annotations to offer research opportunities across AI and Software Engineering.This is stated as the paper’s concluding outlook.