PaperlayerView exact revision on arXiv

Source-linked AI summary

Schrödinger's Code Repository: Have LLMs Learned SWE-bench or Memorized It?

Silin Chen, Yufei Yang, Xiaodong Gu, Yuling Shi, Chengcheng Wan, Haibing Guan

arXiv:2609.27891v1cs.SEcs.AI

TL;DR

Fixed, repeatedly encountered repository presentations leave unclear whether coding-agent performance reflects robust repository reasoning or familiarity with repository-specific cues. SchrodingerRepo dynamically instantiates transformed repository views at evaluation time; removing familiar cues lowered performance on SWE-bench Verified and SWE-QA.

  • Problem

    Fixed, repeatedly encountered repository presentations leave unclear whether strong SWE-bench performance reflects robust code reasoning or memorization of repository-specific patterns.

    Source: §I · p. 1

  • Method

    SchrodingerRepo dynamically instantiates semantically equivalent repository views at evaluation time, transforming familiar cues while preserving executable behavior.

    Source: bust software engineering… · p. 2

  • Results

    Removing familiar cues reduced performance on SWE-bench Verified and SWE-QA; on SWE-bench Verified, full transformations lowered Pass@1 by 6.0–14.4 percentage points across models.

    Source: §VIII · p. 10

  • Takeaways & Limitations

    The findings suggest current coding agents rely substantially on familiar repository-side cues, with 81.6–83.6% of extra actions spent on exploration and localization.

    Source: §VIII · p. 10

    The implementation primarily targets command-line repository interaction, and richer repository-aware tools may require additional adaptation.

    Source: §VI · p. 10

Abstract

from arXiv · show

Repository-level coding benchmarks have become the standard for evaluating coding agents, yet they inherently suffer from data leakage because they are built upon popular open-source repositories repeatedly used for training. Consequently, strong performance may reflect memorization of canonical repository cues rather than robust repository reasoning. We propose SchrodingerRepo (Schrödinger's Repository), an evaluation framework for testing coding agents under dynamically instantiated repository representations. Instead of repeatedly using a static representation of the test repository, SchrodingerRepo treats the test repository as an evaluation-time latent variable that is dynamically instantiated only when the agent enters the evaluation environment. The instantiated repository preserves the original executable behavior while eroding familiar cues such as naming conventions, file layouts, and implementation patterns through four transformation levels: problem statement reconstruction, namespace remapping, intra-file layout reordering, and functionality-preserving code rewriting. We evaluate popular LLMs on SWE-bench Verified and SWE-QA. Results show that removing familiar repository cues consistently degrades agent performance and substantially increases interaction costs across models. Further analysis reveals that the additional cost is primarily caused by increased difficulty in repository exploration and localization. These findings suggest that current coding agents may partially rely on memorized repository-side cues, highlighting the need for evaluation under dynamically instantiated repository representations.

I. INTRODUCTION · bust software engineering capabilities, or simply familiarity with leaked repository representations?

SchrodingerRepo dynamically transforms repositories while preserving executable behavior and eroding familiar cues, enabling evaluation of whether coding agents rely on canonical repository representations. On SWE-bench Verified, transformations reduce Pass@1; they also increase exploration costs, while results on temporally held-out SWE-rebench instances suggest the degradation is not simply due to greater task difficulty.

I. INTRODUCTION

  • Repository-level tasks require navigating large codebases, locating root causes, editing across files, and validating fixes; canonical benchmark scores may reflect memorized repository-specific patterns rather than code reasoning.Source: §I · p. 1

bust software engineering capabilities, or simply familiarity with leaked repository representations?

  • SchrodingerRepo makes repository representation an evaluation-time latent state, producing a deterministic distinct view per run while preserving executable behavior and eroding familiar repository-side cues.The paper presents this as the first systematic study of repository representation sensitivity in repository-level coding-agent evaluation.Source: bust software engineering… · p. 2
  • Its four transformation levels reconstruct problem statements, remap repository-owned namespaces, reorder intra-file layout, and rewrite local implementations while preserving functionality.Source: bust software engineering… · p. 2
  • The evaluation tests GPT-5.4-mini, GPT 5.1, DeepSeek-v4-Flash, and Gemini-3.1-Flash-Lite on SWE-bench Verified, March 2026 SWE-rebench, and SWE-QA.Source: bust software engineering… · p. 2
  • 6.0–14.4 percentage points: full repository transformations reduce SWE-bench Verified Pass@1 across evaluated models, with Namespace Mapping producing the largest drop among individual levels.Source: bust software engineering… · p. 2
  • 81.6–83.6% of additional actions go to exploration-oriented behaviors across evaluated agents, alongside markedly higher token consumption.Source: bust software engineering… · p. 2
  • On SWE-QA, transformed repository views reduce answer quality by up to 4.64 points and increase actions by 18.15–43.02%.Source: bust software engineering… · p. 2
  • On temporally held-out SWE-rebench instances created after the evaluated LLMs’ release, transformations preserve Pass@1 while increasing interaction cost, suggesting Verified-set degradation reflects removed familiar cues rather than intrinsically harder tasks.Source: bust software engineering… · p. 2

II. BACKGROUND · III. APPROACH · A. Overview

The section motivates evaluation under repository representations that are not fixed or visible in advance, reporting task-specific recall on SWE-bench Verified and introducing semantics-preserving transformations to disrupt familiar repository cues.

II. BACKGROUND

  • SWE-bench Verified draws tasks from public repositories whose issues, code, tests, and discussions may appear in training data, motivating a test for task-specific recall.The authors frame this motivation in relation to OpenAI’s analysis that SWE-bench Verified no longer reliably measures frontier coding capabilities.Source: §II · p. 2
  • More than 65% of evaluated instances showed clear leakage evidence, and more than 18% yielded patch/test-level recall before repository access.The study’s recall categories range from recovering affected files or symbols and core repair logic to concrete patch content or test-specific information.Source: §II · p. 2
  • Experts revealed issue semantics round by round while models lacked repository files, gold patches, and test information, checking outputs against hidden references for novel task-specific content.Experts could stop the process and request more concrete recall evidence as needed.Source: §II · p. 2

A. Overview

  • SchrodingerRepo presents each task through an evaluation-time repository view that is semantically equivalent but hidden until the agent enters the environment, preserving the issue, behavior, and test-defined correctness.The design aims to erode repository-side cues potentially memorized from public benchmark artifacts.Source: A. Overview · p. 2
  • Its four levels reconstruct problem wording, remap repository identity and symbols, and create structurally or behaviorally equivalent code variants under the original execution and testing constraints.The mapping is seeded and invertible, with semantics-preserving transformations at both the observation and working-repository layers.Source: A. Overview · p. 3
  • The invertible mapping keeps tool execution grounded in the real SWE-bench environment and translates final patches back into original repository coordinates for standard evaluation.The four transformation levels can be tested jointly for end-to-end robustness or individually to isolate each type’s effect.Source: A. Overview · p. 3

B. Level 1: Problem Statement Reconstruction · C. Level 2: Namespace Mapping

Level 1 reconstructs task statements while preserving their bug descriptions, requirements, and success criteria. Level 2 replaces repository-internal names with a consistent virtual namespace while keeping execution grounded in the original repository.

B. Level 1: Problem Statement Reconstruction

  • Level 1 creates a semantically equivalent task statement to reduce sensitivity to canonical benchmark phrasing while preserving the bug description, requirements, and success criteria.Source: B. Level 1: Problem Statement… · p. 3
  • A generator paraphrases and reorders the statement, then a verifier checks task-defining constraints and triggers refinement if any semantics are missing or weakened.Source: B. Level 1: Problem Statement… · p. 3

C. Level 2: Namespace Mapping

  • Level 2 gives agents a seeded, invertible virtual namespace for repository paths, modules, and symbols, without modifying the physical repository or executable task.Source: C. Level 2: Namespace Mapping · p. 3
  • The mapping process extracts repository identifiers from AST nodes and filters out built-ins, reserved keywords, and third-party symbols to limit remapping to repository-internal terms.Source: C. Level 2: Namespace Mapping · p. 3
  • Seed-controlled subword remapping preserves naming conventions and shared-token consistency while leaving external APIs and language constructs unchanged.Multiple mapping variants may be generated per repository, with deterministic selection based on repository identity and semantic seed.Source: C. Level 2: Namespace Mapping · p. 3
  • A bidirectional translator converts observations into the virtual namespace and translates agent actions back, so execution remains grounded in the original repository state.Source: C. Level 2: Namespace Mapping · p. 3

D. Level 3: Intra-file Layout Reordering · E. Level 4: Functionality-Preserving Rewrite · F. Level-wise Validity Checks

Levels 3 and 4 alter repository ordering or implementation form while preserving the unresolved task, and the paper validates transformations to distinguish agent ability from transformation artifacts. The sections describe how these changes are made and checked.

D. Level 3: Intra-file Layout Reordering

  • Level 3 reorders contiguous runs of top-level definitions and class methods, using non-reorderable statements as anchors while preserving repository behavior.The transformation changes local ordering without altering higher-level organization.Source: D. Level 3: Intra-file Layout Reordering · p. 3
  • Level 3 builds an AST-based definition-time dependency graph and samples valid topological orders uniformly when alternatives exist, retaining the original order otherwise.The graph encodes name-availability constraints from decorators, defaults, annotations, class bases, and class-body expressions. The resulting repository is an execution-time overlay, while evaluation uses the original SWE-bench state.Source: D. Level 3: Intra-file Layout Reordering · p. 4

E. Level 4: Functionality-Preserving Rewrite

  • Level 4 first identifies issue-relevant code and generates offline diff patches that restate its implementation in a behaviorally equivalent form rather than fixing the issue.Its objective is to change representation and local implementation patterns near the eventual fix location.Source: E. Level 4: Functionality-Preserving… · p. 4
  • Validated Level 4 rewrites become the agent’s working repository, complementing Level 3 by changing issue-relevant implementation form rather than definition order.The same unresolved task is presented in a rewritten variant for standard SWE-bench Verified evaluation.Source: E. Level 4: Functionality-Preserving… · p. 4

F. Level-wise Validity Checks

  • Human reviewers inspected 100 randomly sampled instances per level and confirmed that transformed versions preserve the task, target issue, and information needed for resolution.For Level 1, an engineer additionally reviews reconstructed statements for semantic equivalence before presentation to the downstream agent.Source: F. Level-wise Validity Checks · pp. 4–5
  • Level 2 preserves bash-command semantics by translating only namespace-bearing arguments or embedded code, recording observed mappings and restricting reverse translation to them.The translation interface leaves command heads unchanged, changing the agent-visible namespace without changing the executable task.Source: F. Level-wise Validity Checks · p. 5
  • Levels 3 and 4 retain a transformed repository only when Pass to Pass(V) = 1 and Fail to Pass(V) = 0 in the official SWE-bench execution environment.These test-suite conditions preserve already-correct behavior while ensuring the target bug remains unfixed.Source: F. Level-wise Validity Checks · p. 5

G. Patch Recovery · IV. REPOSITORY REPRESENTATION STUDY

SchrodingerRepo varies repository presentation while preserving underlying behavior, enabling a controlled study of coding-agent performance, behavior, transfer to repository QA, and difficulty on temporally held-out issues. Submissions are recovered from final repository states so evaluation remains grounded in the original repository.

G. Patch Recovery

  • Submissions are reconstructed from final repository states after transformations and agent edits, grounding evaluation in the original SWE-bench state across transformation levels.The recovery formulation uses files whose final contents differ from their original contents.Source: G. Patch Recovery · p. 5

IV. REPOSITORY REPRESENTATION STUDY

  • RQ1 compares canonical issue-resolution instances with all four transformation levels enabled jointly and evaluates each level separately to identify contributors to the robustness gap.The comparison concerns repository-side representations and coding-agent performance on repository-level issue resolution.Source: §IV · p. 5
  • RQ2 examines how eroded repository cues redistribute agent actions across navigate, search, read, probe, edit, and test.The study uses a fine-grained action taxonomy to analyze behavior on repository-level issue resolution.Source: §IV · p. 5
  • RQ3 applies the transformation framework to repository-level question answering to test whether performance and behavioral shifts extend beyond issue resolution.Source: §IV · p. 5
  • RQ4 evaluates only the full SchrodingerRepo setting on temporally held-out issues created after the evaluated LLMs’ release to test whether the effect persists when direct exposure is unlikely.Source: §IV · p. 5
  • The study varies agent-facing repository presentation while holding task, execution semantics, and evaluation criteria fixed to isolate effects on agent effectiveness, efficiency, and behavior.SchrodingerRepo is the experimental instrument for systematic manipulation of repository-side cues without changing underlying repository behavior.Source: §IV · p. 5

A. LLM Selection … V. EMPIRICAL RESULTS

The supplied passages describe agent selection, evaluation metrics, and implementation protocols. They identify a SWE-bench Verified results table but provide no result values or comparisons.

A. LLM Selection

  • The representative agent uses mini-swe-agent, a scaffold used by recent coding-agent methods [26], [27].Source: A. LLM Selection · p. 5
  • Four backends are evaluated on SWE-bench Verified; analysis uses GPT-5.4-mini and DeepSeek-v4-Flash, while Gemini-3.1-Flash-Lite is tested on 300 high-leakage-evidence instances.Source: A. LLM Selection · p. 5

B. Metrics

  • Pass@1 measures single-run solve proportion on SWE-bench Verified and SWE-rebench, while SWE-QA uses its default Average Score metric.Source: B. Metrics · p. 5
  • Average Actions, Average Input Tokens, and Average Output Tokens measure interaction length and promptside and generation-side inference cost.Source: B. Metrics · p. 5

C. Implementation Details

V. EMPIRICAL RESULTS

  • Table I is labeled as reporting results on SWE-bench Verified, but the supplied caption contains no values for comparison.No main-result comparison can be stated from the provided table passage.Source: Table I · p. 6

C. Implementation Details

  • Levels 3 and 4 are restricted to golden-patch-related files and code regions, respectively, to limit preprocessing, validation, and runtime costs.Source: C. Implementation Details · p. 6
  • Each instance receives three randomly seeded transformed views per level; agents run independently on each, and results are averaged across views.Different instances from one repository and repeated runs of the same instance typically use different semantically equivalent repository presentations.Source: C. Implementation Details · p. 6

A. RQ1: Effects of SchrodingerRepo · B. RQ2: Effect on Agent Behavior During Issue Resolution

On SWE-bench Verified, SchrodingerRepo’s full transformations lower Pass@1 across models, while namespace remapping causes the largest individual degradation. Agents spend most of their added interaction budget exploring transformed repositories, but models differ in how they explore and in the performance-cost trade-off.

A. RQ1: Effects of SchrodingerRepo

  • With all four transformations, Pass@1 fell for GPT-5.4-mini (46.8% to 35.6%, −11.2%), DeepSeek-v4-Flash (72.8% to 66.8%, −6.0%), GPT 5.1 (44.6% to 36.2%, −8.4%), and Gemini-3.1-Flash-Lite (56.7% to 42.3%, −14.4%).All drops were statistically significant at p < 0.05; interaction costs also rose markedly, with input-token use increasing by more than 2.5× in the strongest affected configurations.Source: A. RQ1: Effects of SchrodingerRepo · pp. 6–7
  • Because the issue, execution environment, and correctness criteria remain unchanged, the degradation points to sensitivity to repository representation; canonical-only evaluation may conflate robust reasoning with representation familiarity.The full setting produced the largest overall degradation, indicating that familiarity is distributed across multiple repository presentation levels, not naming structure alone.Source: A. RQ1: Effects of SchrodingerRepo · pp. 6–7
  • Level 2 Namespace Mapping reduced Pass@1 by 7.4, 6.4, and 6.0 percentage points for GPT 5.1, GPT-5.4-mini, and DeepSeek-v4-Flash, respectively, with significant drops across settings (p < 0.01).By contrast, Level 1 had minor effects: GPT 5.1 and GPT-5.4-mini were unchanged, while DeepSeek-v4-Flash decreased by 2.0 percentage points.Source: A. RQ1: Effects of SchrodingerRepo · p. 6
  • Namespace Mapping also produced the largest cost increases: actions rose 63.3%, 32.6%, and 112.4%, and input-token use rose 162.3%, 108.2%, and 218.3%, respectively, for GPT 5.1, GPT-5.4-mini, and DeepSeek-v4-Flash.Levels 3 and 4 had milder Pass@1 changes: 0.8 to 3.4 percentage points for Level 3 and 1.2 to 2.8 for Level 4, though they removed additional repository cues.Source: A. RQ1: Effects of SchrodingerRepo · pp. 6–7

B. RQ2: Effect on Agent Behavior During Issue Resolution

  • In Figure 3’s full-transformation setting, 83.6% of DeepSeek-v4-Flash’s added actions and 81.6% of GPT-5.4-mini’s went to navigate, search, read, and probe, rather than editing or testing.For DeepSeek-v4-Flash, only 16.5% went to edit and test; overall, repository exploration accounted for more than 80% of added actions for both models.Source: B. RQ2: Effect on Agent Behavior… · p. 7, Figure 3 · p. 7
  • DeepSeek-v4-Flash devoted 31.4% of added actions to probing, while GPT-5.4-mini emphasized reading (41.8%) and searching (33.1%), with only 3.7% probing; GPT-5.1 showed a similar read-search pattern.DeepSeek-v4-Flash’s next-largest shares were read (19.4%) and search (17.9%); its strategy favored runtime probing, while GPT-5.4-mini relied more on static inspection and textual retrieval.Source: B. RQ2: Effect on Agent Behavior… · p. 7
  • More exploratory agents better recover missing repository context and preserve more original Pass@1, but at greater interaction and token cost; more conservative agents use fewer actions yet suffer larger relative performance drops.DeepSeek-v4-Flash explored more both at baseline and under transformed views, while GPT-5.4-mini remained more interaction-efficient but more sensitive to eroded repository cues.Source: B. RQ2: Effect on Agent Behavior… · p. 7

C. RQ3: Transfer to Other Tasks

On SWE-QA, alternative repository instantiations generalize the behavioral shift seen in issue resolution: they increase context-reconstruction effort, while answer-quality effects vary by model and transformation level.

C. RQ3: Transfer to Other Tasks

  • SWE-QA evaluation covered 144 instances across Conan, Reflex, and Streamlink, with 48 instances per repository.The findings indicate that reliance on familiar repository cues and the resulting context-reconstruction effort extend beyond issue resolution to repository-level question answering.Source: C. RQ3: Transfer to Other Tasks · p. 8
  • 70.35 to 65.71 for GPT-5.4-mini and 72.97 to 72.42 for DeepSeek-v4-Flash were their SWE-QA average scores under alternative repository representations.Namespace Mapping (Level 2) had the largest impact; Problem Statement Reconstruction (Level 1) had almost no effect, and Levels 3 and 4 produced only minor answer-quality changes.Source: C. RQ3: Transfer to Other Tasks · p. 8
  • 6.28 to 7.42 actions for GPT-5.4-mini and 24.49 to 35.02 for DeepSeek-v4-Flash accompanied alternative representations, with DeepSeek input tokens rising 59.10%.The added interaction effort primarily went toward recovering repository context rather than producing final answers.Source: C. RQ3: Transfer to Other Tasks · p. 8
  • Levels 3 and 4 had smaller effects than in issue resolution because repository QA locates and understands context without code changes, testing, or iterative validation.Thus, local implementation changes and functionality-preserving rewrites have less influence after relevant context is found.Source: C. RQ3: Transfer to Other Tasks · p. 8
  • DeepSeek-v4-Flash largely preserved QA performance by exploring more, whereas GPT-5.4-mini’s efficient canonical-view context construction was more sensitive to alternative instantiations.GPT-5.4-mini consequently showed both higher interaction cost and degraded answer quality.Source: C. RQ3: Transfer to Other Tasks · p. 8

D. RQ4: Does SchrodingerRepo Increase Task Difficulty? · E. Case Study: Familiar vs. Unfamiliar Repository Views

On temporally held-out SWE-rebench instances, SchrodingerRepo preserved Pass@1 while increasing interaction costs. A case study shows that transformed repository views prompted broader exploration, even though the agent ultimately produced the same fix.

D. RQ4: Does SchrodingerRepo Increase Task Difficulty?

E. Case Study: Familiar vs. Unfamiliar Repository Views

  • In the familiar Django view, issue terminology led the agent directly to the field implementation, where it found unconditional generated-method installation and identified an existence-check guard as the fix.Source: E. Case Study: Familiar vs. Unfamiliar… · p. 9
  • The transformed view required broader source, test, and configuration exploration before analogous render_%s_label logic was found, taking 37→217 actions while yielding the same patch.The agent could not rely on original Django path and class names and also inspected a minimal reproduction setup.Source: E. Case Study: Familiar vs. Unfamiliar… · p. 9, Figure 4 · p. 9

VI. THREATS TO VALIDITY · VII. RELATED WORK · A. SWE-bench Variants

The paper notes limits on generalizing its findings beyond Python-heavy, open-source SWE-style benchmarks and command-line interactions. It situates SchrodingerRepo among SWE-bench extensions and motivates varying repository presentation at evaluation time to retain benchmark coverage while assessing robustness.

VI. THREATS TO VALIDITY

  • Findings may not generalize beyond Python-based open-source benchmarks or to other ecosystems with different repository conventions and development practices.SWE-bench Verified and SWE-QA may also reflect SWE-bench-style interaction patterns, leaving generalization an open question.Source: §VI · pp. 9–10
  • The implementation targets command-line repository interaction; IDE APIs, language-server queries, and AST navigation may require adaptation to keep transformed representations consistent across tools.Extending SchrodingerRepo to support these richer interfaces is identified as future work.Source: §VI · p. 10

VII. RELATED WORK

  • SWE-bench extensions address reliability, instance quality [34], recency [20], [21], long-horizon issues, and multimodal inputs [22],.SWE-bench Verified uses manual validation; SWE-Bench Pro targets complex issues from actively maintained repositories, while SWE-bench Multimodal covers mixed text and visual inputs.Source: A. SWE-bench Variants · p. 10
  • Other SWE-style benchmark work labels instance characteristics, [29], builds executable environments and procedural settings, [31], and scales repository-instance generation [32].SPICE labels issue clarity, test coverage, and effort; SWE-Effi studies agent effectiveness under resource constraints.Source: A. SWE-bench Variants · p. 10

A. SWE-bench Variants

  • SWE-bench variants trade coverage for recency: newer or curated instances can improve evaluation cleanliness but omit long-tail errors in mature repositories, limiting failure-mode breadth.The paper argues that no single variant captures the breadth of real-world software engineering failure modes.Source: A. SWE-bench Variants · p. 10
  • SchrodingerRepo responds by preserving existing benchmark coverage while varying repository presentation at evaluation time, enabling robustness assessment without sacrificing instance diversity.Source: A. SWE-bench Variants · p. 10

B. Static Benchmark Perturbation · VIII. CONCLUSION

SchrodingerRepo creates fresh, semantics-preserving repository representations at evaluation time. Full transformations lower benchmark scores, with most additional actions spent exploring and localizing.

B. Static Benchmark Perturbation

  • Prior static benchmark perturbations expose sensitivity to surface regularities, but fixed transformed instances may later enter training or evaluation corpora.This limits their ability to prevent repeated exposure to specific repository representations.Source: B. Static Benchmark Perturbation · p. 10
  • SchrodingerRepo generates fresh, seed-conditioned repository views for each evaluation run while preserving execution semantics and correctness criteria.Agents are evaluated on resolving the issue from the realized repository structure rather than relying on familiarity with canonical presentations.Source: B. Static Benchmark Perturbation · p. 10

VIII. CONCLUSION

  • Full transformations lower SWE-bench Verified Pass@1 by 6.0–14.4 percentage points; SWE-QA scores fall 0.75–4.64 points as actions rise 18.15–43.02%.These results indicate that current coding agents rely substantially on familiar repository-side cues.Source: §VIII · p. 10
  • 81.6–83.6% of extra actions are spent on exploration and localization, rather than editing or testing.Trajectory analysis identifies repository exploration and localization as the primary sources of additional interaction cost.Source: §VIII · p. 10
  • The results motivate treating repository representation as an explicit experimental variable and measuring robustness across semantically equivalent realizations.This shifts evaluation beyond performance on a single potentially familiar repository presentation.Source: §VIII · p. 10
  • Future work will support repository-aware interfaces such as IDE APIs and language-server queries beyond command-line agent workflows.The aim is to keep representation-robust evaluation consistent across tool interfaces.Source: §VIII · p. 11

Related papers

Loading 2609.27891v1…