Source-linked AI summary
SWE-bench Science: Can Coding Agents Resolve Engineering Tasks in Science?
Zhipeng Xu, Jiahao Lu, Yining Zheng, Yuxin Wang, Xipeng Qiu
TL;DR
Scientific software is part of the instrument producing scientific evidence, yet aggregate coding-agent evaluations reveal little about repair failures. The paper introduces SWE-bench Science and analyzes agents, failure mechanisms, and scientific-information ablations; the strongest result is 47.90% Pass@1, while information effects depend on grounding and organization.
Problem
Existing coding-agent evaluations provide limited insight into why agents fail when repairing scientific software, despite scientific code contributing to evidence behind conclusions.
Method
The paper builds a 119-task repository-level benchmark across 98 repositories and 20 domains, organizes tasks into three paradigms, analyzes failures, and runs a paired scientific-information ablation.
Results
47.90% Pass@1 is the strongest result across eight coding-agent configurations, and auxiliary scientific information has model-dependent effects on exact repair success.
Takeaways & Limitations
Scientific software engineering requires abstraction, disciplined exploration, system-wide integration, and generalization beyond visible cases; scientific information alone does not guarantee better repair.
Takeaways & Limitations
Domain-level comparisons may be unreliable because the number of tasks in each scientific domain remains relatively limited.
Abstract
from arXiv · showhide
Software increasingly functions as part of the scientific instrument itself, making failures in scientific code capable of compromising not only program behavior but also the evidence underlying scientific conclusions. Yet existing evaluations of coding agents largely emphasize aggregate task success, providing limited insight into why agents fail when repairing scientific software. We introduce \textbf{SWE-bench Science}, a repository-level benchmark for scientific software engineering comprising 119 tasks from 98 GitHub repositories across 20 scientific domains. Each task is organized into one of three paradigms: Issue-driven, Expert-exploratory, and Engineering-integration. Even the best-performing agent, \textbf{Claude Code with Opus-5 (max), achieves a pass@1 below 50\%}, highlighting the substantial challenges posed by scientific software engineering. We identify four recurring failure mechanisms: deficits in scientific knowledge or abstraction, misguided exploration or surface-level repair, incomplete repair coverage or system integration, and failures to generalize scientific knowledge beyond observed cases in our analysis. We further conduct a paired ablation that removes explicit scientific guidance while preserving the repository and executable engineering context. The results show that scientific knowledge is not uniformly beneficial: well-grounded information can constrain repair and improve average performance and token efficiency, whereas poorly aligned guidance can induce anchoring and does not necessarily improve exact repair success. Together, SWE-bench Science provides a broad testbed for studying both the capabilities and failure mechanisms of coding agents in scientific software engineering.
1 Introduction
Scientific software is part of the instrument producing scientific claims, so coding-agent repairs must be evaluated for both task success and failure mechanisms. SWE-bench Science addresses this with a broad benchmark, failure analysis, and a paired scientific-information ablation.
- Scientific software defects can corrupt program outputs and the evidence underlying scientific conclusions.
- Aggregate test scores provide limited guidance because they cannot distinguish scientific abstraction, exploration, integration, and generalization failures.
- SWE-bench Science contains 119 tasks from 98 unique GitHub repositories spanning 20 scientific domains.
- The benchmark organizes tasks into Issue-driven, Expert-exploratory, and Engineering-integration paradigms.
- The analysis identifies four recurring failure mechanisms in unsuccessful scientific software repairs.
- Scientific information can improve average scores and token efficiency, but does not automatically improve exact repair success.
2 Related Work
Prior benchmarks progress from self-contained code synthesis to repository-level engineering and scientific workflows, but scientific software engineering remains comparatively underrepresented. SWE-bench Science broadens coverage across scientific domains and task paradigms.
- Early benchmarks evaluate short-function synthesis from natural-language prompts using unit-test correctness.
- Repository-level benchmarks such as SWE-bench evaluate patches for real GitHub issues against project test suites.
- Scientific-code benchmarks include self-contained programming, paper-grounded implementation, research-code reproduction, workflow execution, and scientific discovery.
- SWE-bench Science covers 20 scientific domains and 119 tasks, extending scientific coding-agent evaluation toward repository-level software engineering.
3 SWE-bench Science
SWE-bench Science evaluates scientific software repair using frozen agent-visible inputs, an agent coding loop, and clean evaluation with public diagnostics and private scientific cases. Its benchmark spans 119 tasks across diverse repositories and scientific domains.
- The benchmark covers 119 tasks from 98 repositories across 20 scientific domains, with 52 Issue-driven, 49 Expert-exploratory, and 18 Engineering-integration tasks.
- The agent receives a frozen repository snapshot, a pre-specified problem statement, required scientific context, and public tests.
- Private tests and evaluator-only metadata remain outside the agent workspace until patch submission.
- Public checks support debugging, while private scientific cases determine whether the submitted patch completes the intended repair.
- The evaluation pipeline separates inputs, the agent-driven coding loop, and clean evaluation.
4 Benchmark Construction
The benchmark uses an auditable construction pipeline and three class-specific redesigns to test localized repair, autonomous scientific reasoning, and end-to-end integration. Hidden validation targets semantic correctness beyond visible cases.
- Class-Specific Task Re-design: The three paradigms separately target localized repair, autonomous investigation of scientific discrepancies, and cross-module completion of scientific workflows.
- Benchmark Construction: The unified construction process samples and screens repositories, freezes and reproduces snapshots, isolates public information, and validates hidden behavior.
- Class-Specific Task Re-design: The taxonomy avoids conflating agent capabilities and provides a structured basis for realistic scientific software-engineering evaluation.
- Issue-driven: Issue-driven tasks reproduce known defects through historical analysis, rollback, minimal reproducible examples, and hidden tests across scales and input transformations.
- Expert-exploratory: Expert-exploratory tasks define scientific scenarios, extract exploration points, separate phenomena from root causes, and test mechanism generalization under changed conditions.
- Engineering-integration: Engineering-integration tasks analyze end-to-end call chains, preserve repository context, and validate architecture-level integration across modules.
- Scientific Auxiliary Information Separation: Scientific auxiliary information is separated in paired conditions while the repository, execution environment, tests, and required scientific context remain fixed.
5 Experiments
The benchmark evaluates eight coding-agent configurations on 119 scientific software tasks using public and private tests and multiple repair metrics. No single model leads every metric, and even the strongest overall Pass@1 remains below 50%.
- Main results: Claude-Opus-5 achieves the best overall Pass@1 and leads Issue-driven and Expert-exploratory tasks, while DeepSeek-V4-Pro leads public score and Engineering-integration tasks.GPT-5.6-sol leads private score, Fail2Pass, and Pass2Pass.
- Main results: Even frontier models achieve Pass@1 below 50%, underscoring the challenge of repository-level scientific software repair.The benchmark reports task-level means over the common 119 tasks.
- Performance and token consumption: Claude-Opus-5 achieves the highest Pass@1 with a moderate token budget, while GPT-5.6-sol reaches a similar level with much shorter outputs.The comparison indicates that performance-token balance depends on how effectively configurations use context and generation budgets, not token volume alone.
6 Analysis
The analysis characterizes unsuccessful repairs through four scientific failure mechanisms and examines how scientific information changes scores, exact success, and token use. Its effects vary by model: guidance improves some outcomes for DeepSeek-V4-flash but lowers GPT-5.6-sol’s Pass@1.
- 6.1 Observed Failure Mechanisms: Four recurring mechanisms cover incorrect scientific abstractions, surface-level repairs, incomplete system integration, and failure to generalize scientific principles beyond observed cases.The mechanisms distinguish local symptom treatment from repairs that preserve interactions, data flow, invariants, compatibility, and unseen scientific variants.
- 6.1 Observed Failure Mechanisms: Claude-Opus-5 records the lowest categorized scientific-error count at 58, plus 4 runtime or evaluation-path failures.DeepSeek-V4-flash has the fewest scientific-knowledge generalization errors, while DeepSeek-V4-Pro has the fewest abstraction and system-integration errors.
- 6.2 How Scientific Information Affects Performance: On the 91-task separable subset, scientific information raises DeepSeek-V4-flash Pass@1 from 16.48% to 23.08% but lowers GPT-5.6-sol Pass@1 from 36.26% to 31.87%.The same intervention raises all three reported scores for DeepSeek-V4-flash, while GPT-5.6-sol’s public and private scores increase slightly.
- 6.2 How Scientific Information Affects Performance: Scientific information changes task-level outcomes in both directions: GPT-5.6-sol solves 8 tasks only with information and 12 only without it, while DeepSeek-V4-flash shows 9 and 3.These transitions accompany model-specific score and token responses rather than a uniform benefit.
- 6.2 How Scientific Information Affects Performance: Scientific information can provide semantic constraints absent from local symptoms, but it can also induce anchoring, scope spillover, or premature reliance on supplied explanations.The analysis emphasizes executable evidence and independent validation alongside scientific guidance.
- 6.2 How Scientific Information Affects Performance: Within this comparison, scientific information produces a larger Pass@1 gain for the lower-baseline DeepSeek-V4-flash configuration than for GPT-5.6-sol.The passage presents this as a pattern suggesting weaker-performing models may benefit more from external guidance.
7 Limitations
Comparisons across scientific domains are constrained because each domain contains relatively few tasks. The paper also characterizes its analysis of scientific knowledge as preliminary.
- Limitations: Relatively few tasks per scientific domain may reduce the reliability of comparisons across domains.The paper also reports insufficient exploration of how domain-specific knowledge is used and contributes to successful task completion.
8 Conclusion
SWE-bench Science evaluates scientific software repair through repository-level tasks, scientific contracts, and evidence-sensitive testing. Results show a substantial gap between public-test performance and exact correctness, while scientific information has model-dependent effects.
- 119 tasks from 98 GitHub repositories across 20 scientific domains are organized into Issue-driven, Expert-exploratory, and Engineering-integration paradigms.The benchmark evaluates whether agents can repair scientific software while preserving its scientific contracts.
- The Chain-of-Evidence Protocol separates public and private tests and measures repair progress, exact success, and regression preservation.This distinguishes visible-test performance from complete private-test correctness.
- 47.90% Pass@1 contrasts with a 96.64% public score for the strongest coding-agent configuration.The gap indicates that visible-test performance does not ensure complete repair correctness.
- Auxiliary scientific information lowers GPT-5.6-sol Pass@1 from 36.26% to 31.87% but raises DeepSeek-V4-flash from 16.48% to 23.08%.The paired comparisons use the 91-task scientific-information subset and show model-dependent effects.
- Scientific information alone does not guarantee better repair because its value depends on connecting it to repository evidence and validating it through execution.The conclusion also notes that broader model coverage is needed to understand these effects.
A Scientific Domain Coverage
The benchmark covers 119 SWE-bench Science tasks distributed across 20 scientific domains.
- Table 4 reports the distribution of 119 SWE-bench Science tasks across 20 scientific domains.
B Task-Level Repository and Knowledge Inventory
The task-level inventory links all benchmark tasks to scientific domains, repositories, provenance, knowledge scope, and scientific focus. Its examples span domain-specific contracts involving numerical stability, representation semantics, system integration, and boundary conditions.
- Inventory structure: The appendix contains all 119 tasks and links each task to a scientific domain, upstream repository, issue or pull request when available, knowledge-domain scope, and scientific focus.A missing issue or pull-request reference indicates that no reference was provided in the source annotation.
- Scientific contracts: Split-read interval intersection requires accumulating block overlap before applying the fraction gate, rather than treating each block as an independent hit.
- Scientific contracts: Edited-MRS polarity correction and phase-cycle control must propagate through every subspectrum path rather than remain confined to one loader.The inventory separately records the generalized four-sub spectra requirement.
- Scientific contracts: Spectral registration requires drift-sign consistency with optimization residuals so that compensation direction is not reversed.
- Scientific contracts: Vacuum-wall and projection-wall tasks require geometry, coefficients, derivatives, and outer-region payloads to propagate together through downstream calculations.
- Scientific contracts: Additional inventory entries cover alternate perturbation representations, sideband propagation, 3-D mode coupling, boundary-regime closure, and end-to-end TERPSICHORE integration.These requirements reject local or single-fixture repairs when the scientific contract spans multiple system components.