SWE-Chain: Benchmarking Coding Agents on Chained Release-Level Package Upgrades

1 Introduction

• Existing lifecycle benchmarks use boundaries that do not correspond to shipped versions, while isolated release transitions do not test whether agents carry their own modifications forward.Earlier work explores software lifecycles and release-level tasks, but leaves the chained maintenance question open.Source: §1 · p. 2
• SWE-CHAIN asks whether agents can maintain a real package through chained releases while carrying prior changes forward without breaking existing functionality.It addresses noisy release-level specifications by aligning maintainer-authored release notes with behavioral changes in gold code diffs.Source: §1 · p. 2
• Across nine frontier configurations, agents average 44.8% resolving, 65.4% precision, and 50.2% F1 under Build+Fix, while current agents still struggle to preserve functionality across releases.The benchmark comprises 12 upgrade chains across 9 packages, 155 version transitions, and 1,660 grounded tasks.Source: §1 · p. 2

2.1 Benchmark Construction

• The benchmark collects package releases, notes, and related issues or pull requests, reviews parsed tasks, and extracts code and test diffs between consecutive versions.Chain-specific Dockerfiles support all versions; a pre-validation sanity check requires execution without setup errors and fewer than 0.25% failed tests on each version’s suite.Source: §2.1 · pp. 2–3, Figure 1 · p. 3, Table 1 · p. 3
• Upgrade-related tests are cases that fail or error on the old gold codebase and pass on the new one.The set is identified by running the new version’s test suite on both gold codebases.Source: §2.1 · p. 3

2.2 DecompSynth

• DecompSynth assigns hunks from batches of file-level diffs to release-note tasks, merges batch mappings into global assignments, and synthesizes each task’s upgrade specification from its matched hunks.Specifications contain a problem statement, expectations and constraints, behaviors, and acceptance criteria.Source: §2.2 · p. 4

2.3 Version-Upgrade Protocol

• Each upgrade step uses the agent’s previous codebase as the next step’s starting point, with one Build+Fix attempt allowed for execution-level errors.The correction addresses setup incompatibilities such as import or collection failures.Source: §2.3 · p. 4

3.1 Environment and Configuration

• Each chain uses a persistent container and workspace, with completed steps archived outside the active workspace.The environment is built from the starting version, and the agent runs as a non-root user under /app.Source: §3.1 · p. 4
• Agents are instructed not to access released source code, while shared tool restrictions and domain blacklists limit shortcuts and information leakage.Restrictions include disabling web search and package-installation commands and blacklisting code-hosting and package-registry domains.Source: §3.1 · p. 5

3.2 Model and Agent Configuration

• The evaluation covers eight frontier models from four providers across three agent CLIs, with reasoning effort and CLI versions specified.Experiments ran from April 17 to May 5, 2026.Source: §3.2 · p. 5
• A unified headless harness starts a new agent session at each step, placing the specification and agent-produced codebase in the workspace.This avoids distractions from history compactions and reflects developers returning to a project after time away.Source: §3.2 · p. 5

3.3 Evaluation

• Evaluation uses isolated target-version containers and pytest plugins that preserve coverage and reporting despite partial breakage.The protocol also removes incompatible pytest options from the target version’s configuration.Source: §3.3 · p. 5
• Tests are classified by upgrade-related status and outcome transitions, then micro-averaged across steps for chain-level resolving, precision, and F1.Table 2 summarizes the test categories; resolving measures implementation of required behavior, precision penalizes regressions, and F1 balances resolving and preservation.Source: §3.3 · p. 5, Table 2 · p. 6

4.1 Overall Performance on SWE-CHAIN

• Build+Fix averages were 44.8% resolving, 65.4% precision, and 50.2% F1; Claude-Opus-4.7 (Claude Code) led at 60.8%, 80.6%, and 68.5%, respectively.The authors describe the benchmark as feasible and discriminative, while noting that agents leave more than half of upgrade-related test behaviors unresolved on average.Source: §4.1 · p. 6
• Frontier closed-source models generally outperform peers in their families; GPT-5.3-Codex slightly outperforms GPT-5.4 under Codex CLI, while GPT-5.4 performs better with Codex than OpenCode.Source: §4.1 · p. 6
• Build+Fix raises average precision from 54.0% to 65.4% and resolving from 39.7% to 44.8%, with additional fixing mainly addressing brittle failures and regressions.The regime allows one correction based on error feedback to mitigate over-penalization from setup incompatibilities; average resolving remains 44.8%.Source: §4.1 · pp. 6–7

4.2 Cross-Chain Difficulty

• Chain-level average resolving spans 23.3%–68.5%; PyJWT and Jinja2 average below 70%, while conan and xarray can fall below 30%, and no agent leads every chain.Difficulty depends jointly on package size, upgrade scale, and change density; open-weight models remain competitive on easier chains but degrade on harder ones.Source: §4.2 · p. 7, Figure 2 · p. 7

4.3 Effect of Specification Granularity

• On pytest 8.0, the largest L2-to-L3 gap favors precision over resolving; L1 is noisy, while L4 and L5 are more oracle-like, making L3 the guidance-and-leakage balance.L3 adds conceptual expectations and constraints; L4 and L5 add more grounded details and acceptance criteria.Source: Table 4 · p. 8, §4.3 · p. 8

4.4 Efficiency and Resource Usage

• Claude-Opus-4.7 and GPT-5.5 lead performance but cost $150.39 and $131.34 per chain on average; greater resource use does not guarantee better results.Tool-call success rates are consistently high across models, so performance gaps are unlikely to be driven by tool failures.Source: §4.4 · p. 8

• Coding benchmarks range from functional correctness, library API usage, and contamination controls to repository issue resolution, interactive tool use, and long-horizon command-line tasks.Source: §5 · pp. 8–9
• Long-horizon benchmarks study generation, changing requirements, iterative code degradation, CI traces, or milestone-level evolution, but do not align task boundaries with official package releases.Source: §5 · p. 9
• SWE-CHAIN distinguishes itself by segmenting evolution at maintainer-defined release nodes and using fine-grained synthetic specifications to address coarse release-upgrade specifications.Source: §5 · p. 9

6 Conclusion

• Across 9 frontier configurations, agents average 44.8% resolving and 50.2% F1 under Build+Fix.The benchmark comprises 12 upgrade chains, and the results indicate that agents still struggle to maintain functionality across chained releases.Source: §6 · p. 9
• DecompSynth aligns maintainer release notes with gold code diffs to produce grounded, reliable, informative, and feasible upgrade specifications at scale.Source: §6 · p. 9

A Notation

• Appendix A contains a notation table for SWE-CHAIN.Source: Table 6 · p. 14

B Benchmark Scale and Difficulty Ranking

• Table 7 reports source-tree sizes at each chain's starting and ending versions, including Python-only, all-file, and unrestricted LOC counts.Its “no tests” columns exclude the package test folder.Source: Table 7 · p. 15
• The ranking averages min–max-normalized Python-only starting codebase size excluding tests and average code-change burden per upgrade step.Source: B Benchmark Scale and Difficulty Ranking · p. 15
• Larger Python codebases and denser per-upgrade code changes generally correspond to higher difficulty.Source: B Benchmark Scale and Difficulty Ranking · p. 15
• Table 8 ranks chains using normalized starting codebase size and LOCD/#Upgrades, with rank 1 denoting the hardest chain.Source: Table 8 · p. 15

C Stability of the Divide Step

• Across 45 run pairs on pytest 8.2.2→8.3.0, exact match was 93.66%, Jaccard similarity 94.54%, and core agreement 95.38%.The transition contains 13k diff lines, 102 files, and 986 hunks; core agreement tolerates extra assigned hunks when label sets overlap or both are empty.Source: C Stability of the Divide Step · p. 16

D Forbidden Tools

• Artifact 1 restricts agents to local files, packages, and tools, barring external dependency installation and remote repository access.The listed restrictions include web access, remote shell and file transfer, and package-installation or dependency-download commands.Source: D Forbidden Tools · p. 17

E Pseudocode for SWE-CHAIN

• SWE-CHAIN pseudocode specifies DecompSynth’s divide and conquer procedures and a version-upgrade protocol with Build+Fix regularization.Divide batches diffs under 20-file, 150-hunk, and 3,000-line limits and retries coverage feedback; conquer validates synthesized specifications, while upgrade execution allows one controlled repair after hidden-test errors.Source: E Pseudocode for SWE-CHAIN · pp. 18–19

F.1 Prompts for DecompSynth

• DecompSynth prompts match diff hunks to release-note tasks, then synthesize grounded user-facing requirements from relevant code diffs in structured JSON.Specifications capture requirements, grounded and conceptual constraints and exceptions, behaviors, and acceptance criteria; test diffs provide supplementary context rather than a source for requirements.Source: §F.1 · pp. 20–21
• New task specifications use a structured schema, while excluding version bumps, formatting, import reordering, and unobservable internal refactors.The schema includes a task title and type, runtime impact, description, synthesized requirement, acceptance criteria, significance, confidence, and difficulty.Source: §F.1 · p. 22, §J.1 · p. 31
• The task-wise synthesis prompt reviews unmatched code and test diffs for meaningful behavioral changes not already covered by synthesized tasks.The code diff is the primary source of truth; tests may clarify or validate behavior but cannot alone justify a new task.Source: §F.1 · p. 23

F.2 Prompts for Evaluation

• In Build, agents implement the specified upgrade from the old version to the target version with the original test suite removed.The prompt also requires consistency across files, including renamed or removed symbols and APIs, and prohibits external resource access.Source: §F.2 · p. 24
• In Fix, agents receive a hidden-test error report and make one controlled repair attempt focused on reconciling public API names and signatures.The upgrade specification remains available, and external resource access is prohibited.Source: §F.2 · p. 24

G Build+Fix Trajectories

• Most agents show a larger upward than rightward shift from Build to Build+Fix, indicating a greater precision improvement than resolving-rate improvement.Open-weight models remain below closed-source models after Build+Fix.Source: G Build+Fix Trajectories · p. 25, Figure 3 · p. 25
• Hollow and filled markers denote Build and Build+Fix, respectively, while gray dashed curves indicate F1-score iso-curves.Source: Figure 3 · p. 25

H Cross-Chain Precision and F1

• Across most chains, precision is substantially higher than resolving rate, raising F1 and suggesting agents often target upgrade-specific behaviors.The precision and F1 heatmaps are shown in Figures 4 and 5.Source: H Cross-Chain Precision and F1 · p. 26, Figure 4 · p. 26, Figure 5 · p. 26
• Lower precision on urllib3 and conan 2.23.0 further lowers their F1 scores.These chains are exceptions to the broader pattern.Source: H Cross-Chain Precision and F1 · p. 26
• F1 offers a balanced view of chain difficulty and maintenance capability, while resolving rate more directly reflects requirement implementation ability.Figure 4 shows precision and Figure 5 shows F1 across agents and chains ordered by increasing difficulty.Source: H Cross-Chain Precision and F1 · p. 26, Figure 4 · p. 26, Figure 5 · p. 26

I Per-Step Resolving Trajectories

• Agents progress smoothly across patch upgrades, but larger differences emerge around broader minor and major upgrades as missed requirements accumulate.Figure 6 plots cumulative resolving progress at each upgrade step across all chains.Source: I Per-Step Resolving Trajectories · p. 26, Figure 6 · p. 27

J.1 DecompSynth Examples

• The pytest 8.2.0 →8.2.1 example shows the pipeline decomposing release notes, matching tasks to code-diff hunks, synthesizing task specifications, and integrating them into an upgrade specification.Source: §J.1 · p. 28
• For pytest 8.2.1, the specification requires unreadable unselected directories to stop causing collection crashes while explicitly requested tests continue running.Accessible directories that qualify as Python packages remain collectable.Source: §J.1 · p. 31
• The specification targets Python 3.13 compatibility for traceback and pytester workflows that depend on frame-local state, and rejects invalid Pytester.makefile() calls with ext=None.Source: §J.1 · p. 32
• The examples require selected-test runs to tolerate unreadable unrelated directories while preserving accessible-directory discovery behavior.The specification limits the fix to crashes caused by unreadable directories outside the requested test scope.Source: §J.1 · pp. 31–32
• During collection, pytest must surface KeyboardInterrupt and SystemExit immediately, while retaining existing behavior for ordinary collection errors.Only true system-level interruptions receive special handling.Source: §J.1 · p. 32
• On read-only squashfuse_ll mounts, cache-directory creation failures reported as “Function not implemented” should be recoverable without hiding unrelated errors.The specification says to skip cache creation and continue operating for recognized read-only mount failures.Source: §J.1 · p. 32
• The pytest 8.2.1 example requires .pytest_cache directories to follow normal system permissions rather than becoming owner-only, without broadening permissions beyond system defaults.The stated motivation is avoiding permission errors in repository-scanning tools.Source: §J.1 · p. 32
• Another requirement lets users verify the provenance of official pytest source distributions and wheels without changing runtime behavior.Source: §J.1 · p. 32

J.2 Specification Granularity Examples

• The study defines L1–L5, progressing from raw release notes with GitHub context through problem statements and grounded expectations to a full specification with behaviors and acceptance criteria.L1 uses release notes with linked issues and pull requests; L2 contains problem statements only, while L3 uses conceptual and L4 grounded expectations and constraints.Source: §J.2 · pp. 33–36
• L5 adds behaviors and acceptance criteria to problem statements and grounded expectations and constraints.The example begins by summarizing Python 3.13 support and explicit rejection of pytester file-creation calls without an extension.Source: §J.2 · pp. 33, 36
• The L2 example lists upgrade problems including Python 3.13 support, collection crashes, system-exit handling, read-only mounts, cache permissions, and artifact provenance.Its task statements specify intended outcomes but do not add grounded expectations or constraints.Source: §J.2 · p. 33
• L4 grounds Python 3.13 compatibility in proxy-style frame locals and requires rejecting ext=None while preserving recursion detection and valid extension handling.It also limits unreadable-directory handling to PermissionError during package discovery, preserving existing behavior for accessible directories.Source: §J.2 · p. 35
• L4 further specifies re-raising collection-time KeyboardInterrupt and SystemExit, treating ENOSYS as recoverable for cache setup, and deriving cache permissions from the process umask.The constraints preserve existing reporting for other collection outcomes, propagate unrelated OSError values, and avoid changing the public cache API.Source: §J.2 · p. 35

Task 1

• On Python 3.13, traceback recursion detection and pytester hook-recording checks should work despite changes to frame-local representations.The checks concern caller locals and should not fail because locals are no longer plain dictionaries.Source: §J.2 · p. 36
• Pytester.makefile() should raise TypeError for ext=None while preserving creation of files with valid extensions under existing validation rules..Source: §J.2 · p. 36

Task 2 Avoid collection crashes from unreadable unselected directories

• Collection should ignore an unreadable, unselected directory that raises PermissionError while checking package status and continue collecting explicitly requested tests.The stated goal is to prevent selected-path runs from crashing because unrelated project directories are unreadable.Source: §J.2 · p. 36

Tasks 3–5

• Other named upgrade examples cover preserving system-exit behavior during collection, avoiding crashes on read-only squashfuse_ll mounts, and restoring default .pytest_cache permissions.Source: §J.2 · p. 33

• Users upgrading to pytest 8.2.1 should be able to verify the provenance of official source distributions and wheels without changing runtime behavior.Source: §J.2 · p. 36

Example 9

• L5 specifications combine problem statements, grounded expectations and constraints, behavioral descriptions, and acceptance criteria.The examples below show how those elements describe concrete upgrade behaviors.Source: §J.2 · p. 36

K Limitations

• The specifications receive no human quality assessment because reviewing them requires package-specific examination of release notes, linked issues or PRs, code changes, and synthesized requirements.The authors say this is prohibitively expensive at the scale of 155 transitions and 1,660 tasks.Source: K Limitations · p. 37
• Coverage is limited to 12 Python package chains, and evaluation omits repeated sampling-based runs because long-horizon evaluation is costly.A single chain can cost up to around $150 for strong models; evaluation also uses only two open-weight models across eight models and three agent CLIs.Source: K Limitations · p. 37

L Broader Impacts

• Because the benchmark uses open-source Python packages with release metadata, code diffs, and executable tests, its conclusions may not generalize directly to proprietary, non-Python, UI-heavy, or poorly documented settings.The authors also identify unsafe or poorly reviewed large-scale code changes as a risk, and describe the benchmark as controlled, using sandboxed Docker environments with restricted tool access.Source: L Broader Impacts · p. 37

M Declaration of LLM usage

• LLMs assisted with draft revision and basic framework code, and authors manually reviewed all generated or assisted code line by line before and after integration.Source: M Declaration of LLM usage · p. 37