PaperlayerView exact revision on arXiv

Source-linked AI summary

BenchChecker: Assessing the Credibility of Bug-Fixing Benchmarks for LLMs

Di Wu, Xu He, Shu Wang, Kun Sun

arXiv:2610.19447v1cs.SE

TL;DR

Bug-fixing benchmark results may be inflated when evaluated patches occur in LLM training data, while undisclosed datasets and text-only access complicate contamination checks. BenchChecker combines repository- and patch-presence tests using textual outputs, and its evaluations show both robust detection and substantial medium-task resolution-rate drops after contaminated samples are filtered.

  • Problem

    Bug-fixing benchmark patches may appear in model training data, but undisclosed training datasets and closed-source models' text-only outputs complicate direct contamination checks.

    Source: §1 · p. 1

  • Method

    BenchChecker uses a two-stage forensic process: it probes code prefixes for repository presence, then checks patch persistence against repository history.

    Source: §1 · p. 1, §3.2 · p. 4, §3.3 · p. 4

  • Results

    Across multiple benchmarks, BenchChecker achieves approximately 0.9 AUC, while filtering contaminated samples reduces medium-task resolution rates by more than 20%.

    Source: §9 · p. 15

  • Takeaways & Limitations

    The findings indicate that current LLM evaluations significantly overestimate performance when contaminated samples remain in benchmark test sets.

    Source: §9 · p. 15

    The exact Tc,start is practically unavailable, so some contaminated samples may be false negatives; the authors estimate these cases at approximately 10% of samples.

    Source: §3.1 · p. 3

Abstract

from arXiv · show

While Large Language Models (LLMs) show promise for Automatic Program Repair (APR), existing benchmarks suffer from data contamination, in which test samples appear in training sets, artificially inflating performance. To address this, we propose BenchChecker, a framework compatible with both open-source and closed-source models that supports more credible evaluation by detecting such contamination. Grounded in the observation that persistent patches likely remain in recent repository snapshots used for training, BenchChecker employs a forensics-based two-stage detection process using only textual model outputs together with public repository history. The first stage conducts a repository presence test by probing the model with code prefixes to infer training inclusion; the second stage performs a patch presence test to verify patch persistence via similarity alignment and test-suite verification. Evaluations using StarCoder-15B, StarCoder2-15B, and CodeGen2-16B trained on The Stack v2 demonstrate that BenchChecker achieves robust detection with AUC scores around 0.9 across six benchmarks. Furthermore, applying BenchChecker to SWE-bench Verified and Multi-SWE-bench reveals that filtering contaminated samples reduces the reported resolution rates of most LLMs by more than 20% relative to the original rates on medium-difficulty tasks.

1 Introduction

The introduction frames benchmark contamination as a threat to credible bug-fixing evaluation and presents BenchChecker, a two-stage, text-only framework for detecting it. Its evaluations report robust detection and substantial drops in medium-task resolution rates after contaminated samples are filtered.

1 Introduction

  • For six commonly used benchmarks, over 85% of source repositories appear in The Stack v2 and more than 50% of test patches persist in those repositories.The authors connect this persistence to recent repository snapshots used in training.Source: §1 · p. 1
  • Contamination is difficult to verify because training datasets are undisclosed, while closed-source models expose text rather than logits required by common membership-inference methods.The introduction notes that benchmark patches in training data may let models rely on memorization rather than reasoning or generalization.Source: §1 · p. 1
  • BenchChecker first tests repository presence by probing code prefixes, then checks patch persistence through code similarity and, when needed, test-suite verification.The first stage selects snippets for repetitiveness, distinctiveness, and context-independence; the second treats patches passing all tests as equivalent implementations.Source: §1 · pp. 1–2
  • Across six benchmarks, BenchChecker achieves AUC around 0.9 with a false positive rate below 0.1, outperforming logit-based baselines.The controlled evaluation uses StarCoder-15B, StarCoder2-15B, and CodeGen2-16B trained on The Stack v2, where sample membership is known.Source: §1 · p. 2
  • After filtering contamination, top-10 resolution rates fall by more than 20% on medium tasks; contamination-induced degradation declines from 40% in 2023 to 5% in 2025.The authors interpret the later decline as evidence that newer LLMs demonstrate substantially stronger generalization capabilities.Source: §1 · p. 2
  • BenchChecker is presented as the first bug-fixing benchmark contamination framework that uses only textual outputs and supports both open-source and closed-source models.The authors also report that their framework outperforms existing logit-dependent baselines across multiple benchmarks.Source: §1 · p. 2

2 Background

This section defines bug-fixing benchmarks and explains how training-data collection practices can introduce contamination risks.

2 Background

  • A bug-fixing benchmark is a curated dataset and evaluation protocol for measuring how effectively a model repairs software defects.It typically includes real repository bugs and developer patches, tests for fixing and regression avoidance, and metadata such as language and commit details.Source: §2 · p. 2
  • Popular benchmarks such as SWE-bench and Multi-SWE-bench use leaderboards to compare state-of-the-art LLM resolution rates.The benchmark protocol includes fail-to-pass tests and pass-to-pass tests.Source: §2 · p. 2
  • Training datasets collect recent repository snapshots to maintain freshness, but those snapshots may already contain benchmark patches.The passage names The Stack and The Stack v2 as datasets collecting recent default-branch snapshots.Source: §2 · p. 2

3 System Design

BenchChecker uses repository history and textual model outputs to assess whether a benchmark patch was present in training data. Its two-stage design tests repository inclusion and then patch persistence in the cutoff snapshot.

3.1 Motivation

  • The cutoff-snapshot shortcut assumes patches present in the latest pre-cutoff snapshot persisted continuously; a patch removed before collection ends can be a false negative.The paper reports such cases comprise approximately 10% of samples, partly because the exact collection start time is unavailable.Source: §3.1 · p. 3

3.2 System Overview

  • BenchChecker separates repository inclusion from patch persistence, using a repository-presence test followed by a patch-presence test.The first stage identifies patterns, instantiates query–target pairs, and records successful predictions; the second aligns patch blocks and verifies them through similarity or testing.Source: §3.2 · p. 4

3.4 Stage II: Patch Presence Test

  • Stage II preprocesses and localizes patch blocks in the cutoff snapshot, then checks similarity and uses the benchmark test suite when matching is inconclusive.The fallback replaces patch blocks with aligned snapshot code and classifies the sample as contaminated if the modified diff applies and passes all tests.Source: §3.2 · p. 4, §3.4 · p. 6

3.3 Stage I: Repository Presence Test

  • Stage I probes repository code prefixes for exact target reproduction, treating multiple hits on carefully selected signatures as evidence of repository inclusion.Signatures favor repetitive snippets with distinctive targets and context-independent queries; hit counts are tested against a default hit probability, with rare, high-entropy hits weighted more heavily.Source: §3.3 · pp. 4–5

4 System Implementation

The implementation identifies repository patterns for Stage I and uses structural and semantic matching to locate patches for Stage II. It combines code-pattern mining, query deduplication, and fallback localization methods.

4.1 Repository Presence Test

  • Stage I extracts code-block clones, text duplications, and word co-occurrences, then uses templates to deduplicate repeated patterns and reduce query correlations.The implementation uses NiCad for clones, Simhash [58] for text duplication, and PrefixSpan [54] for frequent co-occurrences; SequenceMatcher extracts templates from clone and duplication patterns.Source: §4.1 · pp. 6–7
  • Stage I estimates query information by computing reference-model loss on unrevealed target sub-tokens and uses the maximum loss as the entropy score.Near-duplicate query prefixes are removed to mitigate syntactic correlations.Source: §4.1 · p. 7

4.2 Patch Presence Test

  • Stage II localizes patch blocks with AST matching and embedding-based similarity, requiring every matched logical line to exceed the similarity threshold for presence.GumTree performs subtree matching within candidate functions; jina-code-embeddings-0.5B [36] reranks candidate lines when exact matching fails.Source: §4.2 · p. 7

5 Experiment and Evaluation

Across six bug-fixing benchmarks, BenchChecker is evaluated against logit-based baselines, with experiments also examining dataset overlap, design choices, and hyperparameters. It consistently outperforms the baselines, while Java code reuse and specific patch-presence failure modes qualify its performance.

5.1 Experimental Setup

  • Experiments use StarCoder-15B, StarCoder2-15B, and CodeGen2-16B trained on The Stack v2, with AUC, TPR, and FPR evaluated against four logit-based membership-inference baselines.The six benchmarks cover Python and Java bug-fixing tasks.Source: §5.1 · pp. 7–8, §5.2 · p. 8

5.2 Dataset and Benchmark

  • About 70% of benchmark patches and over 85% of source repositories appear in The Stack v2; Category 3 samples are around 3% in Python benchmarks and 10% in Java benchmarks.The authors attribute Java’s higher Category 3 prevalence to reused outdated repository versions.Source: §5.2 · p. 8, Table 1 · p. 8

5.3 Performance Evaluation

  • BenchChecker significantly outperforms logit-based baselines across all six benchmarks and three models, exceeding 0.9 AUC on SWE-bench Verified, BugsInPy, and Multi-SWE-bench versus baselines around 0.5.In the complete evaluation, its false positive rate remains at or below 0.1.Source: Table 2 · p. 8, §5.3 · p. 9
  • Patch-presence AUC is approximately 0.9, but Java code reuse lowers performance; Stage II false negatives mainly involve patches absent from the cutoff snapshot (87.0%) or heavily refactored (13.0%).For refactored cases, similarity matching handles 77.0%, test-suite verification recovers 11.0%, and 12.0% remain challenging.Source: §5.3 · p. 9

5.4 Ablation Study

  • Removing snippet-pattern identification drops TPR to 0.428, while unconstrained targets or removing entropy constraints raise false positives; patch-stage ablations also show formatting, localization, similarity, and test verification matter.Without formatting and comment removal, AUC falls to 0.786; exact string matching instead of similarity yields AUC 0.688.Source: §5.4 · p. 10, Table 5 · p. 11

5.5 Hyperparameter Analysis

  • Optimal default hit probabilities are approximately 0.08 for Python and 0.2 for Java, while the patch similarity threshold performs best at 0.7; temperature changes AUC by at most 0.02.The authors attribute the higher Java probability to stronger structural regularity and redundancy, and use temperature 1.0 by default.Source: §5.5 · pp. 10–11, Figure 5 · p. 10

6 Empirical Study

BenchChecker is applied to leaderboard submissions to assess contamination and resolution-rate changes, with the study also examining how those changes vary by task difficulty, benchmark, model timeline, and bug-fixing patterns. Filtering contaminated samples is associated with substantial drops, especially on harder tasks, while the analysis identifies recurring localization and cross-file challenges.

6 Empirical Study

  • Stage I detects both benchmarks’ repositories as present in every analyzed closed-source model at p < 0.05, providing model-specific evidence before Stage II assesses patches.The study covers 159 submissions disclosed as of October 2025 and calibrates hit probability separately for each target model.Source: §6 · p. 11

6.1 Resolution Rate Discrepancy

  • Filtering contaminated cases produces relative resolution-rate drops exceeding 20% for every top-10 SWE-bench Verified submission on 1–4 h tasks and averaging about 40% on Multi-SWE-bench Medium tasks.Across all tasks, the top-10 drops reach 55.9% on Multi-SWE-bench and 11.8% on SWE-bench Verified.Source: §6.1 · p. 12
  • The contamination impact varies by benchmark and agent framework: Multi-SWE-bench drops reach 55.9%, while same-model frameworks differ by at most 3.3% there.On SWE-bench Verified, the top-10 Claude-4 submissions show overall relative drops from 8% to 11%.Source: §6.1 · p. 12

6.2 Resolution Rate Discrepancy in Timeline

  • SWE-bench Verified’s overall resolution-rate discrepancy declines from 42.5% in 2023 to 5.2% in 2025, and stronger reported performers show smaller drops after contamination removal.In the 15–60 min subset, the discrepancy decreases from 55.8% to nearly 0%.Source: §6.2 · p. 12

6.3 Bug-Fixing Patterns of SOTA LLMs

  • Among failures on seen patches, buggy-code localization is the most common pattern (16/30), followed by partial fixes (8/30).Models can fail to apply a memorized patch when they localize the bug to the wrong code location.Source: §6.3 · pp. 12–13
  • Among successful repairs, 49/61 involve single-file modifications, 35/61 contain bug-triggering scenarios, and 21/61 provide direct solution hints.The study reports that constrained repair scope and clear diagnostic information characterize many successful cases.Source: §6.3 · p. 13
  • Among failures on unseen patches, all 25 lack explicit solution hints and 15/25 require cross-file modifications.The authors report that models struggle with dependencies and interactions spanning files or modules.Source: §6.3 · p. 13

7 Discussion

BenchChecker’s repository-presence signal transfers to code completion, while the discussion identifies limits in lexical distinctiveness, patch persistence, and artifact coverage. For training strategies that include full repository histories, the authors suggest using Stage I to exclude repositories detected in the corpus.

Transferability to Other Tasks

  • Including a sample’s repository in training increased exact-match rates by up to 60% over unseen repositories on RepoBench.The authors cite this as evidence that repository presence indicates potential contamination beyond bug fixing.Source: §7 · p. 13

Transferability to Other Data Modalities

  • Outside code, repetitiveness can be represented by co-occurrence patterns and distinctiveness by features such as unique character names or an artist’s stylistic signatures.Context-independence may be less useful outside code because natural language and artwork generally provide weaker context-target relations than structured programming languages.Source: §7 · pp. 13–14

Limitations and Future Work

  • Stage I assesses target distinctiveness lexically rather than structurally; Stage II may underestimate contamination when patches are removed or substantially refactored before the cutoff, though such cases appear uncommon.BenchChecker also does not directly cover contamination through artifacts without comparable code, such as issue descriptions or natural-language discussions.Source: §7 · p. 14
  • For training strategies that include full repository commit histories, the authors suggest applying Stage I and excluding samples from repositories detected in the training corpus.Source: §7 · p. 14

8 Related Work

Prior contamination studies examine memorization, file-path identification, mutated patches, and cutoff effects across code and cybersecurity tasks. Related work also describes patch-presence testing as a way to verify a patch directly rather than infer it from version information.

Data Contamination Study in Bug-Fixing Benchmarks

  • Prior studies used negative log-likelihood for Defects4J memorization, tested SWE-bench file-path identification, examined original-patch recovery after mutation, and reported cutoff effects in cybersecurity tasks.The cited studies are Ramos et al. [55], Liang et al. [41], Kong et al., and recent cybersecurity benchmarks [68] [74], respectively.Source: §8 · p. 14
  • The SWE-bench Verified figures show a gold patch beside a model-generated patch for one instance and an issue description beside a patch for another.Source: Figure 8 · p. 15, Figure 9 · p. 15

Patch Presence Test

  • Patch-presence testing checks for concrete evidence that a known security patch was applied, without relying on version numbers or changelogs.Prior applications include firmware, Android/Java, C/C++ systems, and supply-chain risk assessment.Source: §8 · p. 15

9 Conclusion

BenchChecker combines repository-level and patch-level checks using textual model outputs to detect contamination in bug-fixing benchmarks. The authors report robust detection across benchmarks and resolution-rate drops exceeding 20% on medium-difficulty tasks after contaminated samples are excluded.

9 Conclusion

  • A robust AUC of approximately 0.9 across multiple benchmarks and resolution-rate drops over 20% on medium-difficulty tasks indicate that unfiltered evaluations can overstate LLM performance.Source: §9 · p. 15
  • BenchChecker detects bug-fixing benchmark contamination by combining repository-level and patch-level verification while relying only on textual outputs.Source: §9 · p. 15
  • The evaluation reports that BenchChecker consistently outperforms existing baselines across multiple benchmarks.Source: §9 · p. 15

Ethical Considerations

The ethical discussion considers how contamination findings could affect benchmark maintainers, model developers, and leaderboard users. It reports public-data safeguards against incorrect labels and frames transparency as a way to reduce reliance on inflated scores.

Ethical Considerations

  • Contamination findings may affect leaderboard rankings and user trust; the authors recommend transparent reporting rather than hiding potentially inflated scores.They also suggest maintainers can filter contaminated samples to strengthen leaderboard credibility.Source: Ethical Considerations · p. 15
  • The study uses publicly released, widely used development benchmarks, repositories, and leaderboard submissions that contain no sensitive data.Source: Ethical Considerations · p. 15
  • Because incorrect contamination labels could unfairly affect benchmarks or submissions, the authors use transparent criteria, error analysis, and maintainer engagement as safeguards.Source: Ethical Considerations · p. 15
  • The authors proceeded with publication after judging residual risk low given the safeguards and the benefit of improving benchmark credibility.They describe the study as a tool for maintainers and a way to reduce contamination effects in repository-level benchmarks.Source: Ethical Considerations · pp. 15–16

A Case Study on RepoBench

BenchChecker is applied to RepoBench v1.1 Cross-File-First to examine repository inclusion and code generation. Across three models, about 15% of repositories show strong signs of memorization despite being created after reported cutoffs.

  • The case study evaluates CodeLlama-34B, GPT-4o, and DeepSeekCoderV2-Lite on 580 repositories and 5,054 questions from RepoBench v1.1 Cross-File-First.The split contains repositories created from October 6 to December 31, 2023, with minimal overlap with reported model cutoffs.Source: A Case Study on RepoBench · p. 20
  • RepoBench’s Cross-File-First task predicts the first cross-file line before any local reference to its module or symbol.Figure 10 reports exact-match ratios for repositories detected as included or excluded.Source: A Case Study on RepoBench · p. 20, Figure 10 · p. 20
  • Approximately 15% of RepoBench repositories show strong signs of memorization across all three models despite being created after their reported cutoffs.The authors note that post-cutoff construction reduces but does not eliminate contamination risk.Source: Table 10 · p. 20, A Case Study on RepoBench · p. 20

B Pseudocode of Core Algorithms

The pseudocode summarizes four procedures: text-duplication clustering, word co-occurrence extraction, target-query entropy calculation, and patch-code localization.

Related papers

Loading 2610.19447v1…