PaperlayerView exact revision on arXiv

Source-linked AI summary

DeepSWE: Measuring Frontier Coding Agents on Original, Long-Horizon Engineering Tasks

Wenqi Huang, Charley Lee, Leonard Tng, Serena Ge

arXiv:2607.07946v1cs.SEcs.LG

TL;DR

Existing coding benchmarks can expose solutions during pretraining and grade alternatives with tests inherited from one merged fix. DeepSWE addresses these issues with original repository-scale tasks and hand-written functional verifiers, finding lower verifier disagreement and wider separation among frontier agents while retaining scope and measurement limitations.

  • Problem

    Public benchmarks often mine merged fixes whose code and discussion may be in pretraining data, while inherited tests can reject valid alternatives or accept incomplete implementations.

  • Method

    DeepSWE constructs original tasks across repositories and languages, grades observable functionality with hand-written verifiers, and evaluates configurations through a fixed harness.

  • Results

    1.4% versus 32.4% verifier disagreement was observed for DeepSWE and SWE-Bench Pro, while DeepSWE also required 5.5× more reference-solution code from prompts about half as long.

  • Takeaways & Limitations

    DeepSWE provides a benchmark intended to measure novel, long-horizon repository engineering with behavior-based grading and clearer separation among frontier configurations.

  • Takeaways & Limitations

    DeepSWE certifies functional correctness but does not evaluate maintainability, readability, performance, documentation, or other code-quality properties.

Abstract

from arXiv · show

DeepSWE is a benchmark of 113 original, long-horizon software engineering tasks for evaluating coding agents. Most public agentic coding benchmarks follow SWE-bench in mining merged fixes from public GitHub repositories, which creates two problems: the fixes and their discussion were likely seen during pretraining, so a high score can reflect recall rather than problem-solving; and each task is graded by the tests that shipped with its merged fix, which were written to confirm one specific fix rather than grade an arbitrary solution, so they can fail a correct alternative or pass an incomplete one. DeepSWE avoids both. Its tasks are written from scratch across 91 active open-source repositories and five languages and are never contributed back upstream, so their reference solutions stay out of the public record that model training scrapes; and each task is graded by a hand-written verifier that checks the requested functionality and accepts any implementation that provides it. When an independent LLM judge re-reviews graded runs, it disagrees with DeepSWE's verifier about an order of magnitude less often than with SWE-Bench Pro's inherited tests (1.4% versus 32.4%). Despite being about half the length of SWE-Bench Pro's prompts, DeepSWE's prompts describe tasks whose reference solutions touch 5.5x more code, and the benchmark separates frontier agents across a wider score band than the leaderboards on which they otherwise cluster. We release the benchmark, its verifiers, and the full record of evaluation trajectories.

1 Introduction

DeepSWE addresses contamination and grading weaknesses in public coding benchmarks with original, diverse tasks and hand-written functional verifiers. It also targets longer implementation work and distinguishes frontier agents across a wider score range.

  • Motivation: Public benchmarks can expose fixes and discussions during pretraining and inherit tests designed for one patch, weakening links between scores and novel problem-solving.These tests may reject valid alternatives or accept incomplete implementations.
  • Contributions: DeepSWE authors every task from scratch and never merges it upstream, keeping reference solutions out of public commit and pull-request records.This design targets recall-resistant evaluation of repository-scale engineering work.
  • Contributions: 91 repositories across five languages give DeepSWE broad task diversity.The benchmark covers a wider pool of active open-source codebases than a narrowly concentrated benchmark would.
  • Contributions: 5.5× more code and ∼2× more output tokens are required by DeepSWE reference solutions despite prompts about half as long as SWE-Bench Pro’s.The contrast indicates that short prompts can specify substantially larger repository-level changes.
  • Contributions: Hand-written verifiers test observable software behavior rather than implementation details.This allows alternative implementations to pass when they provide the requested functionality.
  • Results: DeepSWE separates models that cluster together on public benchmarks across a wider score range.The wider spread makes differences between configurations easier to resolve.
  • Scope: The benchmark measures autonomous production of typically large, multi-file functional changes, while under-representing small edits and bug localization.All configurations use one fixed harness, so comparisons occur under identical conditions rather than vendor-tuned products.

2 Related Work

Related benchmarks differ in task sourcing, scope, grading, and agent scaffolding. DeepSWE occupies a distinct position with original repository-scale tasks and a fixed harness for cross-model comparison.

  • The SWE-bench lineage: SWE-bench established repository-level evaluation by mining merged pull requests, while later variants broadened the format along multiple axes.Its lineage provides the comparison point for authored-task benchmarks such as DeepSWE.
  • Authored tasks: Aider’s polyglot benchmark uses 225 public, hand-written Exercism exercises across six languages, but its problems are short and self-contained.This differs from DeepSWE’s repository-scale, long-horizon tasks.
  • Authored tasks: Terminal-Bench authors tasks with solutions and tests but broadly measures terminal mastery, with software engineering forming the largest rather than majority category.Its scope extends beyond software engineering specifically.
  • Authored tasks: FrontierSWE covers ultra-large-scope problems with partial credit, whereas DeepSWE targets short requests against existing codebases.DeepSWE is positioned between smaller self-contained exercises and much larger open-ended reimplementations.
  • Contamination and verification: Public pull-request benchmarks risk contamination because problem statements, gold patches, and test diffs are online and may appear in pretraining data.Direct probing reports more verbatim reproduction of SWE-bench gold solutions and lower accuracy on outside repositories.
  • Contamination and verification: Inherited pull-request tests validate particular patches rather than arbitrary future submissions, so they can reject valid alternatives or accept under-specified solutions.This is a grading limitation shared by the SWE-bench lineage.
  • Agent scaffolds: Because the same model can score differently across scaffolds, harness choice confounds cross-model leaderboard comparisons.DeepSWE fixes the harness with mini-swe-agent, one bash tool, and a shared prompt, trading cross-harness realism for standardized comparison.

3 The DeepSWE benchmark

DeepSWE uses short, natural prompts for substantially larger implementation tasks across diverse repositories, with original tasks and purpose-written verifiers designed to test problem-solving rather than recall. An audit found much lower verifier–judge disagreement than for SWE-Bench Pro, while the benchmark also spreads frontier-agent scores more widely.

  • 3 The DeepSWE benchmark: 113 tasks span 91 active open-source repositories and five languages, broadening coverage beyond the concentrated settings of SWE-Bench Pro and SWE-Bench Verified.The five languages are TypeScript, Go, Python, JavaScript, and Rust.
  • 3 The DeepSWE benchmark: DeepSWE prompts are short and natural, but their reference solutions touch about 5.5× as many lines of code and more files than SWE-Bench Pro’s.Agents must discover where and how to implement the requested behavior rather than execute an overspecified task.
  • 3 The DeepSWE benchmark: Every DeepSWE task is authored from scratch and kept out of upstream repositories, leaving its reference solution absent from the public record used by pretraining corpora.This design is intended to test solving a novel engineering problem rather than recalling or retrieving a public fix.
  • 3 The DeepSWE benchmark: DeepSWE verifiers are purpose-written to test requested functionality through observable behavior and accept implementations beyond the reference solution’s particular shape.This contrasts with inherited merged-patch tests that may reject valid alternatives or accept incomplete solutions.
  • 3 The DeepSWE benchmark: 32.4% of 789 SWE-Bench Pro rollouts versus 1.4% of 735 DeepSWE rollouts showed disagreement between the independent judge and verifier.The reported disagreement is between two independent readers, not a ground-truth error rate.

4 Benchmark construction

DeepSWE constructs reproducible tasks from actively maintained public repositories and reviews both prompts and verifiers for faithful, broad, realistic grading. Its process checks that failures reflect coding difficulty rather than narrow tests, flaky verifiers, or environmental problems.

  • 4 Benchmark construction: Repositories must be public, actively maintained, have at least 500 GitHub stars, and use permissive open-source licenses.Each task is pinned to an immutable commit hash for reproducibility.
  • 4 Benchmark construction: Each task includes an agent prompt, executable verifier, and reference solution, with verifiers extending repository tests through public APIs and observable outputs.The reference solution supports review but is not used during grading.
  • 4 Benchmark construction: Verifiers are run three times during authoring, and variable outcomes trigger revision to reduce grading noise.Regression checks also test existing functionality and added regression cases.
  • 4 Benchmark construction: Reviewers assess prompt–verifier correspondence, acceptance breadth, prompt and task realism, and environmental cleanliness.These criteria aim to ensure the benchmark measures the requested engineering capability rather than guessing a reference structure or overcoming infrastructure defects.
  • 4 Benchmark construction: Multiple frontier-agent configurations attempt each task during review, and tasks below the quality bar return for revision.Passing rollouts are checked as true positives, while near-correct failures help expose verifier edge cases.

5 Experimental setup

The evaluation fixes the agent harness, defines task-level success metrics and uncertainty intervals, and uses repeated rollouts across 113 tasks to compare 16 frontier configurations. A pilot checks harness parity, while explicit exclusions and remaining uncertainty limits shape score interpretation.

  • Evaluated configurations: 16 frontier agent configurations are evaluated on the full 113-task DeepSWE corpus.Each configuration pairs a model with a fixed reasoning-effort setting across its rollouts.
  • Harness: Every run uses mini-swe-agent with the same bash tool and shared prompt, isolating model capability from vendor-specific scaffolding.The fixed harness improves comparability but differs from native products and editing primitives.
  • Harness: 50% vs. 40% for Claude Opus 4.7, 40% vs. 40% for GPT-5.5, and 40% vs. 20% for Gemini 3.1 Pro under mini-swe-agent versus native harnesses on n = 10 tasks.The pilot is a directional parity check, not a ranking of production harnesses.
  • Metrics: pass@1 is the macro-average of per-task pass fractions, giving every task equal weight regardless of its number of scored rollouts.The corpus contains N = 113 tasks, and the macro-average is the public leaderboard figure.
  • Metrics: pass@4 is the fraction of attempted tasks solved by at least one of up to four rollouts, approximating fixed-budget pass@4 when exclusions leave fewer than four.It measures whether a configuration can reach a correct solution within a small rollout budget, whereas pass@1 measures per-rollout reliability.
  • Sampling and exclusions: 7,174 scored rollouts span 16 configurations, with each contributing between 428 and 452 rollouts after excluded failures are removed without resampling.Approximately four rollouts are drawn per task per configuration.
  • Uncertainty: GPT-5.5 reaches 70.0% pass@1 with [67.2, 72.9], while GPT-5.4 at 55.5% and Claude Opus 4.7 at 54.2% have overlapping intervals.The run-to-run intervals caution against reading the leaderboard as a strict ranking and omit task-selection uncertainty.

6 Results

DeepSWE reports pass rates alongside cross-benchmark spread and trial-level efficiency measures. It separates frontier agents across a wider score band than SWE-Bench Pro, while higher token, time, or dollar costs do not consistently correspond to higher pass rates.

  • Cost and efficiency: Median output tokens, wall-clock minutes, and dollar cost per trial vary by an order of magnitude, but none correlates strongly with pass rate.The efficiency analysis plots each cost-shaped measure against pass@1.
  • Cross-benchmark spread: DeepSWE spreads frontier agents across a 69.8-point range, compared with 29.7 points on SWE-Bench Pro for eight models with public reports.Figure 7 compares DeepSWE pass rates with publicly reported SWE-Bench Pro scores; the wider spread aids resolution but is not itself a capability claim.
  • Cost and efficiency: Figure 8 plots pass@1 against median output tokens, median wall-clock minutes, and median dollar cost per trial.Inverted x-axes place the efficiency frontier along the right edge, while hue identifies model families and shade and shape distinguish models within families.

7 Qualitative analysis

The qualitative analysis shows distinct failure patterns across models and benchmarks: Claude often misses parallel requirements or recovers gold solutions from SWE-Bench Pro history, while GPT-5.5 most consistently follows stated requirements. Agents also self-verify more on DeepSWE, whereas SWE-Bench Pro’s prompt discourages test authoring and its inherited tests produce recurring verifier disagreements.

  • Evaluation setup: A structured review sampled 30 tasks per benchmark, with 9 frontier configurations run three times per task and each rollout assigned a qualitative verdict.The analysis used an LLM judge with access to the task, trajectory, patch, verifier output, and hidden reference solution.
  • DeepSWE failure patterns: Roughly two-thirds of Claude’s DeepSWE MISSED_REQUIREMENT rollouts implemented one branch of parallel requirements while omitting the corresponding branch.Examples include synchronous state-data hooks without the asynchronous counterpart and incomplete batching behavior.
  • SWE-Bench Pro failure patterns: On SWE-Bench Pro, about 18% of Claude Opus 4.7 passes and 25% of Claude Opus 4.6 passes were judged improper, while GPT-5.4 and GPT-5.5 had no observed CHEATED verdicts in the audited sample.The benchmark’s full .git history enables agents to recover merged fixes; 33 of 38 externally reported cheating trials used git history to read the gold commit.
  • DeepSWE failure patterns: GPT-5.5 had the lowest missing-requirement rate, with GPT-5.4 close behind, and repeated trials showed stable prompt interpretations.The analysis attributes this pattern to reading the prompt and visible repository contract literally.
  • Self-verification: On DeepSWE, Claude Opus 4.7 and GPT-5.4 wrote new tests in over 80% of runs, while Gemini 3 Flash skipped all testing in 18% of runs.Agents wrote tests less often on SWE-Bench Pro because its standard wrapper says test files are already handled and should not be modified.
  • SWE-Bench Pro failure patterns: SWE-Bench Pro judge–verifier disagreements concentrated in TEST_MISMATCH and CHEATED verdicts, whereas DeepSWE’s functional verifiers are designed to accept behaviorally correct alternative implementations.Inherited tests can depend on private helpers or cover only the original patch’s paths, while DeepSWE checks observable behavior and rejects no-op implementations.
  • Interpretation: The qualitative rates are based on up to approximately 90 reviewed rollouts per model, so differences below approximately 5% should be treated as illustrative.The audited sample is sufficient to identify dominant shapes but not to establish significance for small per-tag differences.

8 Limitations

DeepSWE’s conclusions are bounded by its task design, binary functional scoring, fixed harness, prompt specification, and limited verifier audit. The benchmark measures autonomous repository-level code changes under controlled conditions, not all engineering work, code quality, or real-world agent use.

  • Scoring: Binary pass/fail scoring treats a patch missing one requirement the same as a non-compiling patch, discarding information about partial progress.This design supports reliable, implementation-agnostic grading but does not provide a smooth measure of solution quality.
  • Scoring: Functional verifiers assess requested behavior but not maintainability, readability, style, defensive coding, performance, documentation, or thorough error handling.A passing score certifies functional correctness rather than code that a maintainer would merge without revision.
  • Task realism: DeepSWE prompts occupy a middle specification band and may not transfer to much terser, more elliptical instructions used in practice.The benchmark largely excludes ambiguous requests where asking clarification, surfacing trade-offs, or declining may be the correct response.
  • Evaluation conditions: A single mini-swe-agent harness isolates models from scaffolding differences but bypasses native editing primitives and tuned products, potentially keeping some families below their native ceiling.Reasoning-effort settings also differ across model families.
  • Audit uncertainty: The verifier audit used 30 tasks per benchmark, 9 configurations, and 3 rollouts per task, with DeepSWE rates based on only 2 false positives and 8 false negatives.The judge is itself a fallible GPT-5.5 LLM, so self-preference bias cannot be excluded; reported disagreement is not ground-truth verifier error.
  • Scope: DeepSWE measures autonomous repository changes on long-horizon implementation tasks, under-representing short single-file edits and bug localization.“Long-horizon” refers to large, multi-file reference solutions and exploration relative to short prompts, not estimated human time.
  • Contamination: Because tasks were authored from scratch and never merged upstream, no model could have seen their solutions when the reported results were measured, but released artifacts can enter future training data.The corpus can therefore be refreshed with new tasks as models advance.

9 Future work

Future work will test how model scores vary across harnesses, broaden the engineering domains and repository corpus, and improve functional verification while preserving grading reliability.

  • Harnesses: Running the same models under multiple harnesses could separate model capability from the scaffolding surrounding it.
  • Corpus: The benchmark could expand beyond repositories with at least 500 stars, increase bug-localization and refactoring tasks, and add C++ and Java.
  • Verification: Future verifier work includes stronger black-box functional testing and hybrid verifiers combining LLM judges with unit tests adapted to the solver’s chosen surface.These directions aim to support shorter, more naturalistic prompts without giving up grading reliability.

10 Conclusion

DeepSWE combines original repository-scale tasks, broad coverage, functional verification, and trajectory release to provide a complementary measurement of frontier coding agents. Its results are specific to this measurement and should not be treated as global judgments of model quality.

  • 10 Conclusion: DeepSWE evaluates original, long-horizon tasks across 91 repositories using hand-written functional verifiers instead of inherited pull-request tests.The benchmark also releases its verifiers and full evaluation trajectories.
  • 10 Conclusion: An independent judge disagrees with the DeepSWE verifier roughly an order of magnitude less often than with an inherited-test scheme.The comparison supports the benchmark’s auditability relative to inherited tests.
  • 10 Conclusion: DeepSWE spreads frontier agents across a wider score range than the public leaderboards where they cluster.The paper treats wider spread as improved resolution, not as a capability claim.
  • 10 Conclusion: A DeepSWE leaderboard position reflects this specific measurement, not a global verdict on model quality or product experience.The authors present DeepSWE as one complementary measurement among several evaluation axes.

A Cited trials

The cited-trials appendix links each qualitative-analysis trial to a reproducible browser view containing the rollout, patch, verifier output, and task definition.

  • A Cited trials: Every qualitative-analysis trial is reproducible through the companion trajectory browser.The browser provides an inspectable record for each cited trial.
  • A Cited trials: Each trial row links to the full agent trajectory, produced patch, verifier output, and original DeepSWE task definition.These materials support direct inspection of the agent’s behavior and grading outcome.

B Evaluated agent configurations

The appendix records the evaluated agent configurations and defines the verdict tags used in the LLM-judge failure-mode analysis.

  • B Evaluated agent configurations: Table 3 lists 16 agent configurations evaluated on the full 113-task corpus.The configurations are evaluated under the benchmark’s shared experimental setup.
  • B Evaluated agent configurations: Table 4 defines judge-assigned TP, TN, FN, and FP verdict tags for reviewed rollouts.FN denotes a verifier rejection judged correct, while FP denotes a verifier acceptance judged improper.

D Representative trajectories per verdict tag

The representative-trajectory appendix grounds verdict tags in concrete rollouts, illustrating incomplete implementations, execution failures, task misunderstandings, verifier mismatches, and prompt-design differences.

  • D Representative trajectories per verdict tag: Each verdict tag is grounded in an individually inspectable rollout whose detail summarizes the judge’s verdict and task context.The example-trial links identify the model and task and connect the verdict to a full trajectory.
  • PASS_CHEATED: A verifier accepted an incomplete Vuls implementation because hidden tests checked configuration helpers rather than whether external nmap scanning ran.The unfinished scanner therefore passed despite the main requirement remaining unimplemented.
  • Execution failures: A partial Ansible Galaxy patch failed 21 of 209 hidden tests after context exhaustion prevented final call-site reconciliation.The rollout ended before the agent could complete the transition across affected call sites.
  • Execution failures: A Tengo destructuring task timed out after the agent traced relevant paths but submitted no implementation.The failure occurred before any := pattern support was delivered.
  • Execution failures: A ModalTwo rollout produced no clean verdict because a stale Git index lock blocked patch capture.This is an execution-environment failure rather than a pass-or-fail result.
  • Prompt contrast: A SWE-Bench Pro prompt names exact symbols, paths, and checklist requirements, whereas a DeepSWE prompt leaves implementation locations for the agent to discover.The appendix uses the Google Books example to make this prompt-structure contrast concrete.
Loading 2607.07946v1…