Source-linked AI summary
The Adverse Effects of Code Duplication in Machine Learning Models of Code
Miltiadis Allamanis
TL;DR
Big Code models rely on large code corpora, but near-duplicate files can overlap training and test data, leaving the impact of duplication underexamined. This paper analyzes how duplication biases source-code model evaluation, measures the effect across datasets and models, and provides tools and practices for mitigation. Reported performance can be up to 50% better than performance observed by users, while the paper also shows that the appropriate treatment of duplicates depends on the target application.
Problem
Big Code research lacks adequate investigation of how duplicated files affect machine learning models and their evaluation.
Method
The paper analyzes duplication theoretically and empirically across code corpora, models, and evaluation settings, while releasing detection tools and best practices.
Results
Up to 50% worse performance is observed by users than in reported results across ten code corpora and machine learning models.
Takeaways & Limitations
Removing exact and near duplicates supports more accurate model comparisons and better machine-learning-based tools for programmers.
Takeaways & Limitations
The paper assumes the target application’s true data distribution contains no duplicates, although some applications may include duplicated code.
Abstract
from arXiv · showhide
The field of big code relies on mining large corpora of code to perform some learning task. A significant threat to this approach has been recently identified by Lopes et al. (2017) who found a large amount of near-duplicate code on GitHub. However, the impact of code duplication has not been noticed by researchers devising machine learning models for source code. In this work, we explore the effects of code duplication on machine learning models showing that reported performance metrics are sometimes inflated by up to 100% when testing on duplicated code corpora compared to the performance on de-duplicated corpora which more accurately represent how machine learning models of code are used by software engineers. We present a duplication index for widely used datasets, list best practices for collecting code corpora and evaluating machine learning models on them. Finally, we release tools to help the community avoid this problem in future research.
1 Introduction
Big Code research depends on large code corpora, but near-duplicate files can overlap training and test sets. This paper characterizes the resulting evaluation distortions and offers tools and practices to address them.
- Problem: Near-duplicate files are widespread in large code corpora because engineers often copy files across projects.The issue was first observed in GitHub-scale corpora and concerns file-level near-clones.
- Problem: Identical or highly similar files can appear in both training and test sets, compromising model evaluation.This overlap is the core mechanism through which duplication affects reported performance.
- Findings: Reported performance can exceed performance observed by users by up to 50% across ten code corpora and machine learning models.The paper measures this gap empirically across datasets and models.
- Contributions: The paper provides an application-driven deduplication principle, theoretical analysis, and an empirical demonstration of duplication effects.These contributions address when deduplication is needed and how duplication biases evaluation.
- Contributions: An open-source detector, dataset duplication index, and best practices are released to help researchers mitigate the problem.The detector supports C#, Java, Python, and JavaScript.
2 Code Duplication & Machine Learning
Code duplication changes how source-code models are trained and evaluated because repeated files alter dataset distributions and can cross training-test boundaries. The paper formalizes this bias, relates it to practical applications, and identifies when deduplication is appropriate.
- Definitions: Code duplication means that a large code snippet appears multiple times with no or small differences within a corpus.The paper distinguishes duplicates from the broader set of code clones.
- Applications: Source-code models aim to learn statistical properties of coding practice for tools such as code completion, type prediction, and code summarization.These tools generally provide recommendations on new or unseen code.
- Applications: The relevant true data distribution depends on the target application, so researchers should identify the data their component will encounter.Duplicates are generally excluded for token-level completion but may belong to distributions involving copy-pasted libraries or functions.
- Assumptions: The paper assumes the target distribution contains no duplicates and considers duplication only across files, excluding smaller clones spanning a few lines.These assumptions define the paper’s scope for analyzing duplication effects.
- Duplication bias: Training-test splitting can create in-train, in-test, and cross-set duplicates, each affecting estimates of model behavior differently.Cross-set duplicates can make memorization appear to be generalization.
- Duplication bias: As the duplication factor d increases, the effect of duplication bias β increases in the rewritten estimate ˆf = (1 − d)f̄ + dβ.Repeated samples overweight some training examples and skew reported test metrics.
3 Measuring Duplication
The paper measures file-level near-duplicate code across public machine-learning datasets using token-based detection, then reports duplication statistics and validates detection precision. Results show substantial duplication in many datasets, with rates depending on dataset construction and splitting strategy.
- Detecting near-duplicates: The authors adapt SourcererCC for file-level near-duplicate detection and release the tool under a permissive license.The method tokenizes files and uses identifier/literal fingerprints with Jaccard-similarity thresholds t0 = 0.8 and t1 = 0.7.
- Detecting near-duplicates: The detector identifies duplicates when both identifier/literal fingerprint similarities exceed t0 = 0.8 and t1 = 0.7, excluding files with fewer than 20 identifier tokens.The approach uses a set and multiset of identifier and literal tokens and assumes similar files share tokens.
- Duplication statistics: 10 publicly available machine-learning datasets are analyzed for file-level duplication after prior efforts removed many exact clones.Near-duplicates can remain despite differences in whitespace, comments, or other small code modifications.
- Duplication statistics: 68.7% of Concode methods are duplicates, although function-level sampling may produce higher duplication rates than full-file datasets.Smaller datasets collected from curated project sets generally suffer less duplication, and cross-set rates decrease when splitting across projects.
- Validation and observations: 100 random duplicate pairs from Javascript-150k and 100 from Java-Large received perfect precision in manual inspection.The evaluation supports the detector’s precision on these two datasets, while qualitative inspection found versions, configuration-like files, exact clones, and boilerplate code.
4 Impact on Machine Learning Models
The study evaluates how duplication biases machine-learning training and testing, finding that its effects vary across models, tasks, metrics, and model capacity. Across diverse evaluations, deduplication generally lowers measured performance, especially for identifier-related predictions, while larger models can exploit duplicated code more strongly.
- Evaluation terminology: Code duplication can bias evaluation because existing work often trains and tests on datasets containing duplicated files, rather than measuring performance on duplicate-free data.The study distinguishes biased and unbiased training and testing to expose this difference.
- Neural language-model results: Identifier-related metrics are more severely affected by duplication than metrics for non-identifier tokens.The paper notes that duplication makes otherwise sparse identifiers appear more frequently.
- Model capacity: Larger-capacity models tend to be more heavily affected because additional capacity can be used to memorize duplicated code.The paper cautions that model comparisons under duplication may reflect memorization of cross-set duplicates rather than real performance.
- Other models and tasks: 11.1% lower BLEU was reported for Seq2Seq documentation prediction after removing duplicates.This result used the Python Docstrings v1 dataset.
- Other models and tasks: Removing duplicates reduced measured performance across all evaluated methods, although the magnitude differed across models and tasks.The paper attributes the differences to model characteristics and task-specific sensitivity to duplication.
5 Mitigating Duplication: Best Practices
The paper recommends aligning deduplication and evaluation with the target application's data distribution. It also identifies model capacity, temporal evolution, and deployment conditions that require careful interpretation of results.
- Model Capacity: Large-capacity models are more affected by duplication, which can make comparisons unfair to smaller-capacity models.Naive memorization baselines such as k-nearest neighbors can help reveal duplication when they perform unusually well.
- Understanding the True Data Distribution: Deduplication should follow the target application's expected data distribution and may need file-level or finer-grained treatment.The appropriate granularity depends on the use case.
- Data Collection: Batch-collected code corpora should use near-duplicate detection because removing exact matches and forks alone is insufficient.Splitting data across projects can help, but duplication may remain.
- Alternative Mitigations: Removing duplicates is often simplest, but down-weighting duplicated samples can preserve useful near-duplicate variation while giving each duplicate group one sample's weight.This adjustment can apply to both the loss function and performance metrics.
- Other Considerations: Temporal evaluation and deployment-specific training do not eliminate duplication concerns, because code copied into new files can contaminate datasets and offline tests may diverge from online use.Online evaluation is more accurate when feasible, while offline evaluation should reproduce online conditions.
- Conclusions: Removing exact and near duplicates supports more accurate model comparisons and better machine-learning tools for programmers.The paper also highlights opportunities to exploit or build models robust to near-duplicates.