Source-linked AI summary
A parallel corpus of Python functions and documentation strings for automated code documentation and code generation
Antonio Valerio Miceli Barone, Rico Sennrich
TL;DR
Existing corpora limit progress on practical code documentation and code generation because they are noisy, small, or structurally easy. The paper introduces a diverse Python corpus with docstrings, evaluates NMT baselines, and finds the dataset substantially more challenging than earlier corpora.
Problem
Existing code-description corpora are limited by noise, small size, and structural properties that make code documentation and generation appear unusually easy.
Method
The paper releases over a hundred thousand diverse Python functions paired with programmer-written docstrings and trains NMT models in both directions for baseline evaluation.
Results
The dataset is substantially more challenging than previously published Python corpora, while backtranslation improves baseline performance by 0.5−0.6 BLEU points.
Takeaways & Limitations
The released corpora, metadata, scripts, and baselines support research on models for realistic code documentation and generation, including project-level settings.
Takeaways & Limitations
Existing corpora do not fully represent the complexity of code documentation and generation because some use pseudo-code or repetitive code structures.
Abstract
from arXiv · showhide
Automated documentation of programming source code and automated code generation from natural language are challenging tasks of both practical and scientific interest. Progress in these areas has been limited by the low availability of parallel corpora of code and natural language descriptions, which tend to be small and constrained to specific domains. In this work we introduce a large and diverse parallel corpus of a hundred thousands Python functions with their documentation strings ("docstrings") generated by scraping open source repositories on GitHub. We describe baseline results for the code documentation and code generation tasks obtained by neural machine translation. We also experiment with data augmentation techniques to further increase the amount of training data. We release our datasets and processing scripts in order to stimulate research in these areas.
1 Introduction
The paper addresses code documentation and code generation with a new corpus designed to better represent the complexity of programming tasks. Existing corpora are often small, domain-specific, noisy, or structurally easy, motivating diverse Python functions paired with programmers’ docstrings.
- Motivation: Code documentation, code generation, and natural-language code search are practically and scientifically important joint language-programming tasks.The paper links their practical interest to programmer productivity and their scientific interest to connections between language, computation, and reasoning.
- Existing corpora: Existing corpora inadequately represent task difficulty because they are limited by noise, size, or structural properties.Prior datasets include small human-annotated corpora, restricted-domain collections, and examples whose descriptions align easily with repetitive or pseudo-code-like programs.
- Research gap: The paper identifies a need for a more challenging corpus that better reflects code and documentation as they occur in the wild.The authors specifically argue that existing low-noise corpora can be unusually easy and may have limited practical usefulness.
- Our proposal: The proposed resource contains over a hundred thousand diverse Python code fragments with descriptions written by their programmers.The corpus uses docstrings as natural-language descriptions and is collected from publicly available open-source repositories on GitHub.
- Our proposal: The study trains and evaluates neural machine translation baselines for code documentation and code generation and releases related datasets and outputs.It also releases a code-only corpus with synthetic docstrings, extraction scripts, baseline configurations, and validation and test outputs.
2 Dataset
The dataset is built by extracting top-level Python functions from GitHub, separating declarations, docstrings, and bodies, and preprocessing them into parallel and code-only corpora. The resulting main corpus contains 150,370 function triples, with defined training, validation, and test partitions.
- Extraction and preparation: The extraction pipeline separates each top-level function into its declaration, docstring, and remaining body.Functions with docstrings enter the parallel corpus; functions without docstrings enter a code-only corpus for later synthetic annotation.
- Extraction and preparation: The data are cleaned by removing comments, normalizing syntax, removing irrelevant spaces and newlines, and filtering docstring lines.The code preprocessing is functionally reversible except when code accesses its own docstring or source-code string.
- Dataset description: The corpus statistics are summarized by counts of examples, tokens, and code lines, plus tokens-per-example statistics.These summaries are reported in Tables 1 and 2.
- Dataset description: 150,370 function declaration–docstring–body triples comprise the main parallel corpus.This is the direct output of the extraction process before the reported train/validation/test partitioning.
3 Baseline results
Neural machine translation models provide lower-bound baselines for both code documentation and code generation on the new dataset. Backtranslation improves the base model modestly, while both tasks remain substantially harder than results reported on earlier Python corpora.
- Setup: The baseline uses neural machine translation in both directions without programming-language syntax knowledge.The stated goal is to assess lower-bound performance rather than compete with syntax-aware techniques.
- Setup: Moses tokenization followed by Byte-Pair Encoding is used to reduce data sparsity for function declarations and task inputs.The datasets include function declarations as inputs for both documentation and generation.
- Results: Backtranslation improves the base model by 0.5 −0.6 BLEU points.The improvement is obtained by combining synthetic docstrings with the main parallel corpus for code generation.
- Results: Both tasks appear very challenging compared with earlier Python-corpus results in the 60 −85 BLEU range.The authors note that BLEU is only a shallow approximation, but regard the large difference as evidence that this dataset is more difficult.
- Results: The dataset is intended to represent the complexity of code documentation and code generation as performed by human programmers.The authors frame the baseline results as a reference point for future models.
4 Conclusions
The work addresses limitations in existing code–natural-language corpora by introducing a large, diverse Python corpus and reporting neural machine translation baselines. The corpus is intended to better represent the complexity of human code documentation and generation and support future research and applications.
- Existing corpora do not adequately represent code documentation and generation because of noise, limited size, and restrictive structural properties.
- The authors introduce a large, diverse corpus of Python functions paired with docstrings scraped from public repositories.
- Neural machine translation baselines indicate that the new dataset is more challenging than previously published corpora.
- The corpus may support practical tools such as documentation and code stubs or context-aware autocomplete, while stimulating models that address programming complexity.