PaperlayerView exact revision on arXiv

Source-linked AI summary

The SWE-Bench Illusion: When State-of-the-Art LLMs Remember Instead of Reason

Shanchao Liang, Spandan Garg, Roshanak Zilouchian Moghaddam

arXiv:2506.12286v4cs.AIcs.SE

TL;DR

The paper asks whether software-engineering benchmark results reflect generalizable problem-solving or memorized patterns, and investigates this with cross-benchmark analysis and diagnostic tasks. It finds performance decay within the SWE-Bench ecosystem and on external repositories, including external-repository file-path accuracy drops of up to 47 percentage points.

  • Problem

    The paper examines whether coding-benchmark gains reflect generalizable problem-solving or memorized patterns, a distinction made difficult by overlap between training corpora and benchmark repositories.

    Source: §1 · p. 1

  • Method

    The authors compare performance across similar benchmarks as a proxy for memorization and use diagnostic tasks that withhold information needed for ordinary problem-solving.

    Source: §1 · p. 2, §2.2 · p. 3

  • Results

    Across ten models, performance decays within the SWE-Bench ecosystem and on external repositories, with external-repository file-path accuracy dropping by up to 47 percentage points.

    Source: §6 · pp. 9–10

  • Takeaways & Limitations

    The findings suggest SWE-Bench Verified gains may partly reflect benchmark-specific optimization, motivating temporal controls, cross-repository validation, and cross-benchmark analysis.

    Source: §6 · p. 10

    Experiments included only o3-mini among OpenAI reasoning models because of limited resources.

    Source: §4.3 · p. 8

Abstract

from arXiv · show

As large language models (LLMs) become increasingly capable and widely adopted, benchmarks play a central role in assessing their practical utility. For example, SWE-Bench Verified has emerged as a critical benchmark for evaluating LLMs' software engineering abilities, particularly their aptitude for resolving real-world GitHub issues. Recent LLMs show impressive performance on SWE-Bench, leading to optimism about their capacity for complex coding tasks. However, current evaluation protocols may overstate these models' true capabilities. It is crucial to distinguish LLMs' generalizable problem-solving ability and other learned artifacts. In this work, we introduce two diagnostic tasks: file path identification from issue descriptions alone and ground truth function reproduction with only the current file context and issue description to probe models' underlying knowledge. We present empirical evidence that performance gains on SWE-Bench-Verified may be partially driven by memorization rather than genuine problem-solving. We show that state-of-the-art models achieve up to 76% accuracy in identifying buggy file paths using only issue descriptions, without access to repository structure. This performance is merely up to 53% on tasks from repositories not included in SWE-Bench, pointing to possible data contamination or memorization. Similar patterns are also observed for the function reproduction task, where the verbatim similarity is much higher on SWE-Bench Verified than on other similar coding benchmarks (up to 35% consecutive 5-gram accuracy on SWE-Bench Verified and Full, but only up to 18% for tasks in other benchmarks). These findings raise concerns about the validity of existing results and underscore the need for more robust, contamination-resistant benchmarks to reliably evaluate LLMs' coding abilities.

1 Introduction

The introduction asks whether benchmark gains reflect generalizable coding ability or remembered patterns, and proposes cross-benchmark comparisons and diagnostic tasks to investigate that distinction. Its results show unusually strong SWE-Bench performance alongside drops on external benchmarks, raising concerns about benchmark validity.

1 Introduction

  • The motivating gap is whether coding-benchmark gains reflect generalizable problem-solving or memorized patterns from training data.The introduction notes potential overlap because training corpora include public repositories used to construct benchmarks such as SWE-Bench.Source: §1 · p. 1
  • Three diagnostic tasks probe file-path identification, function reproduction, and prefix completion to measure distinct memorization patterns.They target issue-file associations and verbatim code reproduction at different scales.Source: §1 · pp. 1–2
  • The paper uses performance disparities across similar benchmarks as a proxy for potential training-data contamination rather than genuine coding ability.The approach assumes genuine skill should yield similar performance across comparable tasks, while memorization may produce unusually high scores on likely-overlapping tasks.Source: §1 · p. 2
  • Up to 76% file-path accuracy on SWE-Bench-Verified instances contrasts with substantially lower performance on external benchmarks, suggesting benchmark-specific memorization.The file-path task provides no contextual information that should be necessary for the task.Source: §1 · p. 2
  • The findings raise concerns about software-engineering benchmark validity and support more robust, contamination-resistant evaluation approaches.Source: §1 · p. 2

2 Approach

The approach uses three diagnostic tasks to probe memorization by withholding information that would support ordinary problem-solving. The tasks test issue-to-file identification, complete-function reproduction, and sequential code completion.

2.1 File Path Identification Task

  • Models predict a buggy file path from the issue report and repository name without repository structure, code, or metadata; exact matches to ground-truth patch paths count as correct.The study also reports filtered accuracy when issue descriptions do not explicitly mention paths.Source: §2.1 · p. 2

2.2 Function Reproduction Task

  • Models regenerate functions removed by the ground-truth patch using issue descriptions and buggy-file context, but without function specifications or signatures.The setup also withholds docstrings and context from other files, making successful reproduction a test of prior exposure to the implementation.Source: §2.2 · p. 3

2.3 Prefix Completion Task

  • Prefix completion gives models code preceding a patched snippet and compares generated continuations with ground truth for verbatim matches.The evaluated continuation spans up to N lines, where N is the number of lines modified at that location; this complements whole-function reproduction by testing sequential code recall.Source: §2.3 · p. 3

3 Experiment Setup

The experiments use three established benchmarks and specify diagnostic settings, evaluation metrics, models, and prompt templates for assessing coding behavior.

3.1.1 SWE-Bench-Verified. / 3.1.2 SWE-Bench-C#.

  • SWE-Bench-Verified comprises 500 curated tasks from 12 Python repositories, while SWE-Bench-C# follows its construction pipeline with 75 tasks from 11 C# repositories.The standard benchmark tasks require code edits based on issue descriptions; the study uses task context as specified for its diagnostics.Source: §3.1.1 · p. 3, §3.1.2 · p. 3

3.1.3 RefactorBench.

  • RefactorBench provides 39 selected tasks from a benchmark of 100 handcrafted, multi-file refactoring tasks across nine open-source Python repositories.The tasks use natural-language instructions and require patches that perform the requested refactors.Source: §3.1.3 · p. 3

3.2 Additional experiment settings / 3.2.1 Full-SWE-Bench. / 3.2.2 SWE-Bench Extra.

  • Additional evaluations cover sampled Full-SWE-Bench tasks, recent issues from SWE-Bench repositories, and tasks from repositories outside SWE-Bench.The settings test benchmark-specific and temporal patterns; the recent-issue set has 217 tasks, and the outside-repository set has 245 tasks from seven repositories.Source: §3.2 · p. 4, §3.2.1 · p. 4, §3.2.2 · p. 4

3.3 Metrics

  • The metrics include exact file-path accuracy, filtered accuracy excluding descriptions with path-like strings, frequency-aware consecutive 5-gram overlap, and instances with any exact code-hunk match.The last measure applies to prefix completion; an instance is compromised when generated code contains at least one exact ground-truth implementation hunk.Source: §3.3 · p. 5

3.4 Models

  • The study evaluates ten OpenAI and Anthropic models, using provider APIs with default sampling settings and model-family-specific generation limits.The listed limits are 2,048 tokens for chat and Claude models and 4,096 for the reasoning-focused o-series models.Source: §3.4 · p. 5

3.5.1 File-Path Identification Prompt Template.

  • The file-path prompt supplies the repository name and issue description and asks the model to reason and output one predicted buggy file path.Figure 6 presents the prompt template.Source: §3.5.1 · p. 5, Figure 6 · p. 5

3.5.2 Function Reproduction Task Prompt Template.

  • The function-reproduction prompt gives the repository, function names and file paths, and file content with patched functions removed, then requests reconstructed implementations.Figure 7 presents the prompt template.Source: §3.5.2 · p. 5, Figure 7 · p. 6

4.1 File Path Identification Accuracy

File-path accuracy declines from curated SWE-Bench Verified tasks to broader and external sets, while model-family patterns point to both instance-specific and repository-level memorization. The results suggest SWE-Bench scores may not reflect transferable coding ability alone.

4.1.1 Instance-Specific memorization.

  • The within-ecosystem decline suggests instance-specific memorization, with models disproportionately exposed to or optimized for widely circulated benchmark problems.Performance is highest on the commonly reported Verified subset and declines on broader or newly collected tasks.Source: §4.1.1 · p. 6

4.1.2 Repository-Bias memorization.

  • Accuracy below 53% on Outside-Repo Tasks, despite their popular repositories, supports repository-bias memorization of patterns specific to SWE-Bench’s twelve repositories.The authors describe overfitting to those repositories’ architectural patterns and problem distributions, rather than transferable skills.Source: §4.1.2 · pp. 6–7

4.1 File Path Identification Accuracy

  • 60–76% accuracy on SWE-Bench Verified fell to 57–71% on Full-SWE-Bench and 50–68% on SWE-Repo, while Outside-Repo Tasks remained below 53%.Figure 8 compares file-path identification accuracy across benchmarks.Source: §4.1.1 · p. 6, §4.1.2 · p. 6, Figure 8 · p. 7

4.1.3 Model-Specific Contamination Patterns.

  • OpenAI chat-model rankings vary across benchmarks, o3’s advantage over smaller reasoning models is ambiguous, and Claude accuracy generally rises by generation, with a RefactorBench exception for Sonnet 4.Across ten models and vendors, the reported hierarchy is SWE-Bench Verified > Full-SWE-Bench > SWE-Repo Tasks > SWE-Bench C#, RefactorBench, and Outside-Repo Tasks.Source: §4.1.3 · p. 7

4.1.4 Implications for Benchmark Validity.

  • The two memorization patterns mean high SWE-Bench scores are not purely indicative of generalizable coding ability and may be inflated by specialized task and repository knowledge.The authors allow that such knowledge may exist instead of, or alongside, transferable software-engineering skills.Source: §4.1.4 · p. 7

4.2 File Path Identification Accuracy on Filtered Instances

After excluding instances whose descriptions contain file paths or import statements, the same accuracy hierarchy persists: Verified tasks remain easiest, and external-repository performance still drops sharply. The filtered results support both instance-specific and repository-bias memorization.

4.2 File Path Identification Accuracy on Filtered Instances

  • The filtered results support both instance-specific patterns within SWE-Bench repositories and repository-bias memorization on external repositories.The authors say the persistence indicates performance is not due to simple text matching and suggests benchmark scores may not purely reflect transferable software-engineering skills.Source: §4.2 · p. 7
  • Filtered file-path accuracy remains highest on SWE-Bench Verified and drops on Full SWE-Bench and SWE-Repo tasks, with the external-repository decline also persisting.Figure 9 presents results after removing instances with explicit file paths or import statements from issue descriptions.Source: §4.2 · p. 7, Figure 9 · p. 8

4.3 Function Reproduction Result

Function reproduction shows higher verbatim overlap on curated SWE-Bench Verified tasks than on less-curated or external tasks. The authors interpret the pattern as evidence of memorized training sequences rather than systematic derivation from limited context.

4.3 Function Reproduction Result

  • With only function names and issue descriptions, models lack enough information to systematically derive correct implementations, so high consecutive 5-gram overlap indicates reliance on memorized sequences.The reported 5-gram overlap reaches up to 34.9% on Verified, compared with up to 18.2% outside the SWE-Bench ecosystem.Source: §4.3 · p. 8
  • The function-reproduction experiments included only o3-mini among OpenAI reasoning models because of limited resources.The authors say the findings demonstrate that Verified performance substantially reflects memorization of training sequences.Source: §4.3 · p. 8
  • 34.9% 5-gram overlap on SWE-Bench Verified fell to 28.7% on SWE Full and 18.2% on SWE Extra, matching RefactorBench’s 18.1% and nearing Outside Tasks’ 13.9%.SWE Extra comes from the same repositories as Verified, yet has overlap comparable to external benchmarks.Source: §4.3 · p. 8

4.4 Direct Memorization on SWE-Bench Verified

Prefix completion produced exact instance-level code matches across evaluated SWE-Bench Verified tasks, with rates varying across model families. The authors say these matches strengthen concerns that benchmark results reflect memorization rather than algorithmic reasoning or program comprehension.

4.4 Direct Memorization on SWE-Bench Verified

  • 11.7%–31.6% instance-level verbatim match rates were reported across eight models on SWE-Bench Verified.The Claude family’s rates rose from 12.1% to 31.6%; GPT-family rates were 17.4%–18.4%, and o3-mini had the lowest rate at 11.7%.Source: §4.4 · pp. 8–9
  • Claude-family verbatim match rates increased from 12.1% to 31.6% across generations, while GPT-family rates remained within 17.4%–18.4%.The authors describe the Claude progression as suggesting a strong correlation between model capacity and memorization.Source: §4.4 · pp. 8–9
  • Exact code reproduction from contextual prefixes without problem descriptions or bug identification suggests these instances reflect memorization rather than algorithmic reasoning or program comprehension.The authors conclude that a substantial portion of reported SWE-Bench performance gains may be attributable to memorization.Source: §4.4 · p. 9

4.5 Comparison of 5-Gram Overlap for Buggy V.S. Ground Truth Code Snippet

The function-reproduction comparison finds that models’ outputs favor patched code on some established SWE-Bench settings, while the signal disappears or reverses on the freshness-controlled SWE-Bench Extra split. The patterns vary by provider and language ecosystem, supporting the authors’ call for freshness-controlled evaluation.

4.5 Comparison of 5-Gram Overlap for Buggy V.S. Ground Truth Code Snippet

  • Δ5 subtracts overlap with the buggy function from overlap with the ground-truth function; positive values indicate greater similarity to the patched implementation, and negative values to the buggy code.Figure 11 summarizes the measure across benchmarks and model families.Source: §4.5 · p. 9
  • Anthropic’s Claude models show roughly +2 to -6 percentage points on SWE-Bench Verified, Full-SWE-Bench, and Outside-Repo Tasks, while OpenAI models are generally below +2pp on those sets.The passage associates the smaller OpenAI differentials with weaker memorization of those specific fixes.Source: §4.5 · p. 9
  • On SWE-Bench Extra, all models except Claude 3.5 Sonnet and GPT 4.1 have negative Δ5, while the authors report positive Δ5 when historical patched code is in the training mix.The authors interpret the post-cutoff split’s shift toward the buggy reference as supporting their contamination-mitigation claim.Source: §4.5 · p. 9
  • On the SWE C# subset, OpenAI’s GPT-4* family has the largest positive differentials, at approximately +3 to 5pp, exceeding Claude.The authors suggest this may reflect heavier exposure to patched C#/.NET code, or lighter deduplication of that ecosystem, in OpenAI training data.Source: §4.5 · p. 9
  • The authors identify higher likelihood of benchmark memorization on SWE-Bench Verified and SWE-Bench C#, and call for freshness-controlled benchmarks when interpreting coding performance and memorization risk.Their contrast is measurable positive Δ5 for historical patched code and a disappearing or reversing signal for code post-dating the model.Source: §4.5 · p. 9

5 Related Work

The related work describes uncertainty about whether coding-benchmark gains reflect problem-solving or pattern matching, alongside existing approaches that mutate tasks or probe model outputs. The paper positions cross-benchmark comparison as a reusable alternative that does not require training-data or model-internal access.

5.1 Coding Benchmarks and Agents

  • SWE-Bench uses real GitHub issues and patches, but public GitHub data raises exposure concerns as agent performance improves while the role of genuine problem-solving remains unclear.The passage frames this as uncertainty about the extent to which gains reflect problem-solving versus pattern matching.Source: §5.1 · p. 9

5.2 Benchmark Contamination

  • Task-mutation approaches alter benchmarks through semantic transformations, AST-level mutations, or prompt variations to expose overfitting to specific formulations.The cited examples include EvoEval [19], AST-level mutations [4], and TaskEval [16].Source: §5.2 · p. 9
  • Metric-based probing analyzes outputs without changing tasks, but hidden-state measures are unavailable for commercial models and n-gram overlap can confuse correct solutions with verbatim memorization.The passage names perplexity, Negative Log Likelihood, and n-gram accuracy as probing measures [9] [14].Source: §5.2 · p. 9
  • The paper proposes comparing performance across related benchmark variants and external evaluations as a reusable memorization proxy that requires neither training-data exposure knowledge nor model-internal access.The framework is presented as addressing limitations of prior mutation and metric-based approaches.Source: §5.2 · p. 9

6 Conclusion

The conclusion presents evidence of instance-specific and repository-bias memorization patterns across ten models, with performance decay within the SWE-Bench ecosystem and on external repositories. It argues that SWE-Bench Verified gains may partly reflect benchmark-specific optimization and calls for contamination-resistant evaluation.

6 Conclusion

  • Across ten models, the authors report instance-specific performance decay within the SWE-Bench ecosystem and repository-bias memorization, including external-repository file-path drops of up to 47 percentage points.They report these patterns across model families and vendors.Source: §6 · pp. 9–10
  • Figure 11 compares 5-gram overlap differences against ground-truth and buggy snippets across benchmarks in the function-reproduction task.A higher difference indicates greater similarity to ground-truth code.Source: Figure 11 · p. 10
  • The findings suggest SWE-Bench Verified performance improvements may partly reflect benchmark-specific optimization rather than genuine advances in coding capabilities.The conclusion points to temporal controls, cross-repository validation, and systematic cross-benchmark analysis as evaluation needs.Source: §6 · p. 10
Loading 2506.12286v4…