PaperlayerView exact revision on arXiv

Source-linked AI summary

Are Large Language Models Memorizing Bug Benchmarks?

Daniel Ramos, Claudia Mamede, Kush Jain, Paulo Canelas, Catarina Gamboa, Claire Le Goues

arXiv:2411.13323v3cs.SEcs.AIcs.LG

TL;DR

Public bug benchmarks may overlap with LLM training data, but the extent of leakage in software-engineering benchmarks has received limited study. The paper measures benchmark membership and model familiarity and reproduction signals, finding stronger leakage signals for older benchmarks, especially Defects4J, and lower risk for newer datasets and models. It concludes that benchmark leakage risk should be monitored and that its findings primarily concern base models.

  • Problem

    Although public bug benchmarks are used to assess LLMs, limited evidence quantifies whether training-data leakage compromises evaluation of true model capability.

    Source: §I · p. 1

  • Method

    The study assesses potential leakage through TheStack repository membership, NLL, and 5-gram accuracy on bug benchmarks and likely newer code.

    Source: §I · p. 2

  • Results

    Older benchmarks, especially Defects4J, show higher memorization signals, while newer benchmarks and newer models show lower leakage risk or less memorization.

    Source: §IV · p. 6, §VII · p. 7

  • Takeaways & Limitations

    Researchers can use BugsInPy and BugsCpp with relatively low leakage risk for now, while regularly reassessing membership, NLL, and 5-gram accuracy as training datasets evolve.

    Source: §IV · p. 6

    The findings primarily address leakage in base models; the effects of fine-tuning strategies are left for future empirical assessment.

    Source: §V · p. 7

Abstract

from arXiv · show

Large Language Models (LLMs) have become integral to various software engineering tasks, including code generation, bug detection, and repair. To evaluate model performance in these domains, numerous bug benchmarks containing real-world bugs from software projects have been developed. However, a growing concern within the software engineering community is that these benchmarks may not reliably reflect true LLM performance due to the risk of data leakage. Despite this concern, limited research has been conducted to quantify the impact of potential leakage. In this paper, we systematically evaluate popular LLMs to assess their susceptibility to data leakage from widely used bug benchmarks. To identify potential leakage, we use multiple metrics, including a study of benchmark membership within commonly used training datasets, as well as analyses of negative log-likelihood and n-gram accuracy. Our findings show that certain models, in particular codegen-multi, exhibit significant evidence of memorization in widely used benchmarks like Defects4J, while newer models trained on larger datasets like LLaMa 3.1 exhibit limited signs of leakage. These results highlight the need for careful benchmark selection and the adoption of robust metrics to adequately assess models capabilities.

I. INTRODUCTION

Real-world bug benchmarks support realistic evaluation of software-engineering tasks, but their public availability raises concern that leakage may inflate measured LLM capability. The paper quantifies this risk using dataset membership, NLL, and 5-gram accuracy, finding stronger memorization signals for older models, especially codegen-multi, and for Defects4J.

I. INTRODUCTION

  • Benchmark leakage is a concern because training on information unavailable at prediction can inflate performance metrics and misrepresent model effectiveness,.Publicly accessible benchmark programs and bug solutions may have entered LLM training data.Source: §I · p. 1
  • A codegen-multi prompt from Defects4J bug 39 reproduced the remainder of the benchmark solution verbatim, including comments, suggesting memorization.The model received the function through line 11 and generated the following lines to match the benchmark solution.Source: §I · p. 1, Figure 1 · p. 2
  • Prior work quantified leakage in NLP and ML, but the authors found no comparable assessment of popular software-engineering bug datasets and note that training-data opacity complicates detection.Dataset inclusion alone does not establish memorization, particularly among billions of training samples.Source: §I · p. 1
  • The study assesses potential leakage through TheStack membership, NLL, and 5-gram accuracy on bug benchmarks and recent repositories mined from GitHub.The 2024 repository dataset serves as a likely novel-code comparison.Source: §I · p. 2
  • Older models, especially codegen-multi, show stronger memorization signals on bug benchmarks, while newer models such as LLaMa 3.1 show less; Defects4J has the highest potential memorization across models and metrics.The authors identify high 5-gram accuracy and low NLL as signals of greater memorization.Source: §I · p. 2

II. METHODOLOGY

The methodology combines collection of bug benchmarks and recent repositories with model selection and leakage evaluation. The new repository set is intended to provide likely-unseen comparison data.

II. METHODOLOGY

  • The methodology has three components: data collection, model selection, and evaluation.The overview describes evaluation of NLL and n-gram accuracy on base models, plus benchmark membership analysis in TheStack.Source: §II · p. 2, Figure 2 · p. 3

A. Data Collection & Filtering.

  • The benchmark set includes BugsCpp, Defects4J, and BugsInPy, alongside newer GitBug-Java and SWEBench-Lite.SWEBench-Lite was chosen instead of SWEBench because of computational constraints.Source: A. Data Collection & Filtering · p. 2
  • For benchmark bugs, the study gathered patched ground-truth files, removed files with over 85% overlap, and retained the oldest file when bugs shared one file.Only fixed files were collected because these correspond to solutions a model may have memorized.Source: A. Data Collection & Filtering · p. 2
  • The researchers curated 3,214 Java and Python repositories from 2024, using a 100-star threshold and MinHash with LSH to filter overlap before sampling 250 files per language.Repositories from 2022–2023 were also used to filter 2024 repositories containing duplicated data.Source: A. Data Collection & Filtering · p. 2

B. Model Selection.

The study evaluates open-source pretrained base models and detects potential leakage with repository membership and file-level familiarity and reproduction metrics. Membership is checked in TheStack; NLL and n-gram accuracy are computed on files.

B. Model Selection.

  • The evaluation uses open-source base models, excluding instruction-tuned models because NLL requires sequence probabilities and n-gram accuracy may suit conversational models less well.The model set spans codegen-multi, LLaMa 3.1, Gemma 2, StarCoder, and Mistral families.Source: B. Model Selection · p. 3

C. Leakage Detection

  • Leakage detection measures repository-level membership in TheStack and file-level NLL and n-gram accuracy on patched benchmark files or sampled new-repository files.TheStack membership is assessed using repository presence because the study lacks direct access to all evaluated models’ training datasets.Source: C. Leakage Detection · p. 3
  • Lower NLL indicates a sequence appears more natural to a model and may reflect familiarity, while higher n-gram accuracy indicates closer exact reproduction of reference code.A 5-gram score of 1.0 means the model can produce a sequence verbatim.Source: C. Leakage Detection · p. 3

III. RESULTS

The results assess benchmark leakage in base models through membership in TheStack, NLL, and 5-gram accuracy, with regression analysis of model characteristics. Repository membership can indicate project familiarity without proving exposure to a particular fixed file.

III. RESULTS

  • The results examine TheStack membership, NLL, 5-gram accuracy, and regression relationships between model characteristics and the two file-level metrics.The membership analysis compares three versions of TheStack.Source: §III · p. 4

A. Membership

  • GitBug-Java has the lowest repository membership in TheStack, while Defects4J and SWEBench-Lite have high proportions of repositories represented.The membership table excludes the new 2024 Java and Python data because TheStack includes data only through 2023.Source: A. Membership · p. 4
  • Repository membership does not establish that a model saw a specific fixed file, though it may indicate familiarity with the project’s source code.That familiarity may contribute to higher-quality patches or results, so membership is a consideration when assessing generalization.Source: A. Membership · p. 4

B. Negative Log Likelihood

NLL patterns suggest Defects4J is especially familiar to codegen-multi, while newer LLaMa 3.1 models show more consistent NLL across datasets. Recent repositories generally have higher NLL, though their distribution may differ from training data.

B. Negative Log Likelihood

  • 0.15 NLL on Defects4J for codegen-multi is the lowest reported here, and the authors say it suggests memorization of its solutions.Gemma 2 27B also has low Defects4J NLL (0.38) relative to other benchmarks and repositories.Source: B. Negative Log Likelihood · p. 4
  • Codegen-multi’s SWEBench-Lite NLL is 0.38 despite the benchmark’s recent publication, consistent with its projects predating publication and fixes dating as early as 2017.The authors report that this aligns with their membership analysis.Source: B. Negative Log Likelihood · p. 4
  • New repositories have higher NLL than Defects4J, BugsCpp, BugsInPy, and SWEBench-Lite, but distribution differences may also contribute to the gap.The paper notes that evaluation benchmarks draw from prominent projects that may have appeared repeatedly in pretraining.Source: B. Negative Log Likelihood · p. 4
  • 5.63× lower NLL on Defects4J than on new Java repositories, and 3.82× lower than on GitBug-Java, indicate greater codegen-multi familiarity with Defects4J.The heatmap encodes each cell as the column dataset’s NLL divided by the row dataset’s NLL; darker cells indicate higher ratios.Source: B. Negative Log Likelihood · p. 4, Figure 4 · p. 5
  • LLaMa 3.1 70B’s Defects4J-to-GitBug-Java NLL ratio is 1.27, and LLaMa 3.1 models show relatively consistent NLL across benchmarks.The LLaMa 3.1 family was trained on 30× more tokens than codegen-multi.Source: B. Negative Log Likelihood · pp. 4–5

C. 5-Gram Accuracy

Defects4J has the highest 5-gram accuracy across model families, with codegen-multi particularly high; comparisons with recent repositories and other metrics qualify what these matches indicate.

C. 5-Gram Accuracy

  • Figure 5 presents 5-gram accuracy for selected models, with the appendix containing full tables and other models reported as consistent with the displayed results.Interpretation across models can differ because of vocabulary and tokenization differences.Source: C. 5-Gram Accuracy · p. 5
  • 82% 5-gram accuracy on Defects4J for codegen-multi strongly suggests it likely memorized much of the benchmark’s solutions.Defects4J has the highest 5-gram accuracy across all model families; the codegen-multi and Gemma 2 differences from new repositories are 34 and 14 percentage points, respectively.Source: C. 5-Gram Accuracy · p. 5
  • 47% for new Java repositories and 48% for new Python repositories are the average 5-gram accuracies, which the authors suggest may reflect common coding patterns.Codegen-multi also has notably high 5-gram accuracy on SWEBench-Lite, despite its recent publication, because it incorporates older data.Source: C. 5-Gram Accuracy · p. 5
  • LLaMa 3.1 8B and 70B have consistent 5-gram accuracy across benchmarks, but this metric alone cannot conclusively establish absence of memorization.A BugsInPy patch’s “__init__(self,” and “__init__(\nself,” differ in 5-gram matching despite equivalent content, so the authors recommend considering NLL, 5-gram accuracy, and membership together.Source: C. 5-Gram Accuracy · p. 5

D. How Do Model Characteristics Influence Risk of Leakage?

Mixed-effects linear regressions relate model size and pretraining budget to average NLL and 5-gram accuracy, accounting for dataset and tokenizer variability. Both predictors are associated with lower NLL and higher 5-gram accuracy, with examples across LLaMa 3.1 and Gemma 2 families.

D. How Do Model Characteristics Influence Risk of Leakage?

  • Each additional 1B parameters predicts a 0.002-unit decrease in average NLL, while each additional 1T training tokens predicts a 0.014-unit decrease.The reference prediction is 0.744 at 6B parameters and 1T tokens; the model accounts for dataset and tokenizer as random effects.Source: D. How Do Model Characteristics… · p. 5
  • Each additional 1B parameters predicts a 0.001-unit increase in 5-gram accuracy, while each additional training-budget unit predicts a 0.006 increase.The reference prediction is 0.465; all reported coefficients have statistical significance.Source: D. How Do Model Characteristics… · p. 6
  • LLaMa 3.1 70B and Gemma 2 27B show higher 5-gram accuracy and lower NLL than their smaller family counterparts, alongside larger parameter counts and, for Gemma, a larger training budget.LLaMa 3.1 8B and 70B share a 15T-token budget; Gemma 2 2B used 3T tokens and 27B used 12T.Source: D. How Do Model Characteristics… · p. 6
  • The authors infer that models with more parameters tend to show higher n-gram accuracy and may memorize more; LLaMa 3.1 70B predicts strings, class names, and if-statements in contexts where they may not be immediately apparent.The regression predictors are parameter count and training-token budget, used as a proxy for unique token count; exact unique-token counts were often unavailable.Source: D. How Do Model Characteristics… · pp. 5–6

IV. DISCUSSION

Defects4J shows the strongest signs of potential memorization, while BugsInPy and BugsCpp appear lower risk by comparison. The discussion recommends supplementing evaluations with recent benchmarks and monitoring leakage indicators as pretraining data evolves.

IV. DISCUSSION

  • 82% 5-gram accuracy on Defects4J for codegen-multi compares with 64% for both CodeLlama 7B and Gemma-2 27B.Source: §IV · p. 6
  • 80% of Defects4J is represented in TheStack, making elimination of this leakage in future models likely nearly impossible.Codegen-multi also shows similarly low NLL on SWEBench-Lite.Source: §IV · p. 6
  • BugsInPy and BugsCpp have lower leakage risk in almost all models, with metrics not significantly different from newer GitBug-Java.The paper recommends regularly reassessing membership, 5-gram accuracy, and NLL against more recent data.Source: §IV · p. 6
  • LLaMa 3.1 appears to memorize fewer benchmark solutions, though the 70B model still produced solution files with little context.Source: §IV · p. 6
  • Supplementing evaluations with recent benchmarks such as GitBug-Java may provide more reliable assessments because recent bugs and patches are less likely to be in pretraining data.Source: §IV · p. 6

V. LIMITATIONS AND THREATS

The authors acknowledge limits in filtering and matching newly collected repositories to benchmark data. They also qualify leakage measurements because train-split exposure can resemble leakage, and their findings primarily concern base models.

Data Collection

  • New-data filtering may not exclude legacy code, and the authors cannot guarantee repository distributions match the benchmarks; a >100-star filter was used to argue comparable project quality.Source: §V · p. 6

Train + Test Splits

  • Because patch and buggy files differ little, train-split exposure alone could yield high 5-gram accuracy and low NLL; verbatim patched outputs also occur, so analysis tracks trends rather than absolute values.The authors note that benchmark patch files may be close to files in the train split, even when only buggy files leaked.Source: §V · p. 7

Forgetting

  • The findings mainly concern base models; full-scale fine-tuning may reduce leakage through forgetting, while frozen-weight adapters and larger models may raise leakage risks, which remain for future work.Source: §V · p. 7

VI. RELATED WORK

The related work frames leakage measurement as an open challenge and applies established metrics to bug benchmarks. The conclusion reports higher memorization signals in older benchmarks and recommends considering both benchmark and model choice.

VI. RELATED WORK

  • LLMs are used in automated program repair, vulnerability detection, and locating buggy lines, while training on public code repositories raises concerns about remembered benchmark solutions and inflated performance.Source: §VI · p. 7
  • Leakage measurement remains an open challenge; common approaches compare benchmark loss, generated-code n-gram overlap, or performance on original and transformed benchmarks.Source: §VI · p. 7
  • The study applies established leakage metrics to widely recognized software-engineering bug benchmarks and examines recent and earlier models.Its model examples include LLaMa 3.1 and codegen-multi.Source: §VI · p. 7

VII. CONCLUSION

  • Older benchmarks, especially Defects4J, show higher memorization signals across models, particularly codegen-multi 6B; GitBug-Java, BugsInPy, and BugsCpp show lower risk.The newer datasets have NLL and 5-gram accuracy similar to new 2024 repositories.Source: §VII · p. 7
  • Newer models with higher training budgets show significantly lower data-leakage risks.Source: §VII · p. 8
  • Researchers should consider the evaluated model and pair older benchmarks such as Defects4J with newer ones such as GitBug-Java.The conclusion presents this as a way to improve model evaluation and address potential leakage issues.Source: §VII · p. 8

APPENDIX A

The appendix captions identify results and diagnostics for NLL, 5-gram accuracy, NLL ratios, and linear-regression residuals.

APPENDIX A

Loading 2411.13323v3…