Source-linked AI summary
Recommendations for Datasets for Source Code Summarization
Alexander LeClair, Collin McMillan
TL;DR
Code-summarization research lacks standardized datasets, limiting comparable and reproducible evaluations. The paper proposes dataset-building guidelines, examines splitting choices, and releases a Java dataset; function-based splits yield a false BLEU boost over project-based splits.
Problem
Code-summarization researchers lack standardized datasets, and repository processing differences make evaluation results difficult to compare and reproduce.
Source: §1 · p. 1Method
Results
23.02 BLEU with function splits versus 17.41 BLEU with project splits, a nearly one-third increase the authors call a false boost because test-project functions enter training.
Source: §4.1 · p. 4Takeaways & Limitations
The authors conclude that function-based splitting should be avoided because it places functions from test projects in training, an unrealistic evaluation scenario.
Source: §4.1 · p. 4Recommendations may need adaptation for IDE comment generation, where same-project code-comment pairs can be useful.
Source: §5 · p. 5
Abstract
from arXiv · showhide
Source Code Summarization is the task of writing short, natural language descriptions of source code. The main use for these descriptions is in software documentation e.g. the one-sentence Java method descriptions in JavaDocs. Code summarization is rapidly becoming a popular research problem, but progress is restrained due to a lack of suitable datasets. In addition, a lack of community standards for creating datasets leads to confusing and unreproducible research results -- we observe swings in performance of more than 33% due only to changes in dataset design. In this paper, we make recommendations for these standards from experimental results. We release a dataset based on prior work of over 2.1m pairs of Java methods and one sentence method descriptions from over 28k Java projects. We describe the dataset and point out key differences from natural language data, to guide and support future researchers.
1 Introduction
Source-code summarization research lacks standardized datasets, making results difficult to compare and reproduce. The paper proposes dataset-building guidelines and releases a large Java methods-and-descriptions dataset.
1 Introduction
- Source-code summarization produces short natural-language descriptions of code, commonly for software documentation such as Java method summaries in JavaDocs.Many techniques adapt methods originally developed for NLP problems.Source: §1 · p. 1
- Without standardized datasets, researchers process repositories differently, making evaluation conditions inconsistent and earlier results difficult to reproduce.Standardized datasets can let researchers compare techniques under the same test conditions and follow community best practices.Source: §1 · p. 1
- Dataset-creation differences can mislead researchers by overstating or understating technique performance, making reported results difficult to interpret and replicate.The paper gives examples of large differences between reported and replicated scores.Source: §1 · p. 1
- The paper proposes guidelines for building code-summarization datasets and releases over 2.1 million Java method-description pairs from over 28,000 projects.The dataset is intended to support future researchers.Source: §1 · p. 1
2 Related Work
Related work has shifted from heuristic and template-based methods toward data-driven approaches, but the field lacks standard datasets. Different dataset sizes and preprocessing choices accompany substantially divergent reported results.
2 Related Work
- The literature moved from heuristic and template-based summarization methods toward data-driven techniques, while the field remained small and new.Related applications include commit-message and method-name generation, pseudocode generation, and code search.Source: §2 · p. 2
- No standard code-summarization datasets had been published, and two Java CODENN studies reported BLEU-4 scores of 6.3 and 25.3 on different datasets.The authors describe the differing results as difficult to interpret across studies.Source: §2 · p. 2
- Dataset sizes and tokenization choices differ, including contrasting approaches to replacing or removing out-of-vocabulary comment tokens.The cited datasets contained 2.1 million and 69,708 Java method-comment pairs.Source: §2 · p. 2
3 Dataset Preparations
The section bases its dataset on earlier Java method-comment data, examines processing choices, and characterizes differences between code and natural-language summaries.
3 Dataset Preparations
- LeClair et al. dataset derives from Sourcerer and contains 2.1m Java method-comment pairs after initial filtering.Source: §3 · p. 2
3.1 Research Questions
- The study quantifies dataset-processing choices; RQ1 contrasts method-level and project-level splits, within a scope of Java methods paired with JavaDoc descriptions.Source: §3.1 · p. 2
3.3 Dataset Characteristics
- 22% of words occur once across Java methods versus 35% in book summaries; 34% occur in one Java method and 20% in two.Source: Figure 1 · p. 3, Figure 2 · p. 3
3.2 Methodology
- For RQ2, four random project splits retaining automatically generated code were compared with four project splits where it was removed.Source: §3.2 · p. 3
3.3 Dataset Characteristics
- Repeated code words are concentrated within methods rather than across them, and this overlap inflates BLEU scores when datasets are split by function.Source: §3.3 · p. 3
- Java methods are longer than comments, while 70% of summary-comment words are absent from code, requiring models to map implementation details to high-level concepts.Source: §3.3 · pp. 3–4
4 Experimental Results & Conclusion
Function-level splits produced a large, misleading BLEU boost over project-level splits, while retaining automatically generated code also raised BLEU and increased variation. The authors advise against function-level splitting and recommend precautions to remove generated code.
4.1 RQ1: Splitting Strategy
- Over 30% of comment words also occur in methods, while about 11% of code words occur in comments; methods average around 30 words versus 10 for comments.Source: Figure 3 · p. 4
4.2 RQ2: Removing Autogen. Code
- Retaining automatically generated code raised baseline performance to 18 BLEU and increased split-dependent variation, supporting reasonable precautions to remove it.The authors report that the boost was smaller than the split-strategy effect.Source: §4.2 · pp. 4–5
4.1 RQ1: Splitting Strategy
- 23.02 BLEU for function splits versus 17.41 for project splits—a nearly one-third boost deemed false because functions from test projects entered training; avoid function-level splitting.The comparison uses four runs for each split strategy.Source: Figure 4 · p. 4, §4.1 · p. 4
5 Discussion
The discussion presents the study’s split-strategy findings and dataset as contributions, while noting that IDE settings may call for different data and that simulations should preserve revision chronology.
5 Discussion
- The paper contributes evidence on project-versus-function splitting and a cleaned, tokenized dataset of 2.1m Java method-description pairs with train, validation, and test splits.Source: §5 · p. 5
- For IDE generation, same-project pairs may be useful because IDEs can access live source and edit history rather than relying on post-hoc repositories.Source: §5 · p. 5
- An IDE simulation should train only on revisions preceding its test point: for revision 75, use revisions 1–74, not revisions 76 onward.Source: §5 · p. 5
6 Downloadable Dataset
The appendix offers raw, filtered, and tokenized versions of the Java method-comment data, with the tokenized version also providing standard data splits and a reshuffling script.
6 Downloadable Dataset
- The Raw Dataset contains SQL records for file names, comments, and method line ranges; the unpreprocessed Filtered Dataset has 2.1m pairs after removing generated files and methods without comments.Source: §6 · p. 5
- The Tokenized Dataset applies the LeClair and McMillan (2019) scheme to filtered strings and includes train, validation, and test sets plus a reshuffling script.Source: §6 · p. 5