ChainSWE: Benchmarking Coding Agents on Multi-Bug Software Maintenance

1 Introduction

• Single-issue benchmarks start from clean checkouts and do not test whether patches avoid unnecessary edits, resolve latent dependencies, or preserve context across related fixes.This leaves overshoot and undershoot failure modes unmeasured in sequential rollouts.Source: §1 · p. 1
• In overshoot, extra edits can pollute files targeted by later issues; in undershoot, omitted supporting refactors can leave downstream preconditions unmet.The paper illustrates both failure modes with CHAINSWE rollouts.Source: §1 · p. 1, Figure 1 · p. 2
• CHAINSWE evaluates time-ordered bug-fix chains with overlapping code, replaying accumulated gold patches and tests while each agent fix inherits the prior repository state.The dataset contains 100 chains spanning 304 instances and 54 repositories from six SWE-bench-family datasets.Source: §1 · p. 2
• Per-bug success drops by up to 70% at the deepest chain positions compared with single-issue settings.The evaluation covers seven language models under multiple context-management configurations.Source: §1 · p. 2

Repository-level issue resolution benchmarks

• Repository-level SWE benchmarks grade one issue per rollout from a clean commit, whereas CHAINSWE carries repository state across related issues and measures per-bug and full-chain success.This evaluates failures caused by accumulated, self-generated state.Source: §2 · pp. 2–3

Agent harnesses for code

• Code-agent harnesses vary context management, including structured interfaces, history summarization, sub-agent delegation, and fixed step-by-step pipelines.CHAINSWE uses SWE-EDIT to compare full history, summarization, and sub-agent editing as conversation history and code edits accumulate.Source: §2 · p. 3

Sequential and continual reasoning benchmarks

• Sequential reasoning and memory benchmarks test multi-step planning or retained dialogue state, while CHAINSWE combines real test-validated repository fixes, explicit code overlap, and per-bug versus full-chain evaluation.The paper identifies this combination as distinct from prior settings that reset tasks or focus on dialogue state.Source: §2 · p. 3

3.1 Task Features

• CHAINSWE pools Python-project issues from six benchmarks, representing each issue with its repository, commit, problem, patches, tests, and commands.A CHAINSWE instance is an ordered, chronological list of these issues, initialized from the first bug’s base commit.Source: §3.1 · pp. 3–4

3.2 Mining Chains

• Chains join consecutive issues by commit succession or clean application of the later gold patches, then require code overlap and successful ordered test replay.The overlap filter checks modified files, functions, or classes; replay verifies FAIL_TO_PASS tests pass without PASS_TO_PASS regressions.Source: §3.2 · p. 4

3.3 Data Selection

• Selection drops length-2 chains and retains only chains where Qwen3.7-Max in ORACLE mode solves more than half the bugs.This screens for chains whose difficulty is not dominated by individual bug difficulty.Source: §3.3 · pp. 4–5

3.4 Dataset Statistics

• 100 accepted chains average 3.04 issues and contain 304 bug-fix tasks from 54 repositories, spanning April 2019 to August 2025.The chains are drawn from six existing repository-level SWE benchmarks.Source: §3.1 · p. 3, §3.4 · p. 5

4.1 Agent Configurations

• BASELINE, SUMMARIZE, and SUB-AGENT vary how trajectory context is retained or handled while sharing the same model, tools, and prompts.SUMMARIZE compresses history beyond a 50K-input-token budget; SUB-AGENT delegates file viewing and editing without summarization.Source: §4.1 · p. 5

4.2 Language Models

• Seven language models are evaluated, including GPT-5.5, Claude-Opus-4.7, DeepSeek-V4-Pro, and Gemini-3.1-Pro.For reasoning-capable models, thinking effort is set to medium; other inference parameters use API defaults.Source: §4.2 · p. 5

4.3 Evaluation Modes

• ORACLE applies earlier gold fixes on a reset base; SEQ carries model patches in a shared repository, and SEQ+MEM also preserves conversation history.ORACLE is an oracle-state per-bug baseline, while the sequential modes measure the cost of carrying model-modified state across a chain.Source: §4.3 · pp. 5–6, Figure 3 · p. 5

4.4 Experiment Settings

• Evaluations run on a 32-core, 64-GB CPU-only Linux server, with a 100-turn and 30-minute limit per instance.Four evaluations run in parallel.Source: §4.4 · p. 6

4.5 Metrics

• Metrics include task-level %Resolved, per-position and full-chain success, and average API cost per task and chain.A chain succeeds only when all bugs are resolved.Source: §4.5 · p. 6

5.1 Overall Success Rate

• 58.9% under ORACLE falls to 36.5% under SEQ and 36.9% under SEQ+MEM; all 21 model-configuration cells lose per-bug accuracy from ORACLE to SEQ.Chain accuracy is already low under ORACLE (20.0%) and is comparable under SEQ (17.0%).Source: §5.1 · p. 6, Table 2 · p. 7
• The ORACLE-to-SEQ per-bug accuracy drop grows from 33% under BASELINE to 40% under SUMMARIZE and 41% under SUB-AGENT.Raw history is more robust than active rewriting, which can introduce information loss and inconsistency; rewriting can sometimes lower cost but often with lower robustness.Source: §5.1 · p. 7

5.2 Success Rate by Position

• Under ORACLE, later bugs are not inherently harder, but under SEQ success falls along the chain and the first bug remains near the isolated baseline.For GPT-5.5, SEQ resolution drops from 62.5% at position 1 to 40.5% and 28.2%; its ORACLE rates rise from 61.2% to 70.4% and 75.9%.Source: §5.2 · p. 8, Table 4 · p. 8
• At position 3 under SEQ, BASELINE retains 27.7% resolution versus 20.6% for SUMMARIZE and 17.5% for SUB-AGENT.The rewriting configurations show a 70% relative drop from the ORACLE baseline; SEQ+MEM offers a small, inconsistent last-position improvement without changing the ordering.Source: §5.2 · p. 8, Table 3 · p. 8

5.3 Failure Mode Analysis

• 48% of downstream SEQ failures are chain errors, rising from 43% at position 2 to 52% at position 3; traced errors more often follow under-edits than over-edits.Figure 4a splits chain errors by earlier under-edit versus over-edit, roughly nine to one.Source: Figure 4 · p. 9, §5.3 · p. 9
• Among attempted chain errors, 40% break previously passing code, 25% leave the bug unfixed, and 34% produce no usable patch.The most common outcome is breaking working code, which the authors connect to structural damage already present in the inherited repository state.Source: §5.3 · p. 9

6 Conclusion

• Per-bug success drops by up to 70% at the deepest chain positions; carrying prior-fix history yields only marginal gains, while active summarization and sub-agent editing show larger degradation.The comparison spans models and context-management configurations.Source: §6 · p. 9
• CHAINSWE mines dependent issue chains from six SWE-bench-family datasets and validates them through containerized test replay, exposing failure modes hidden by isolated bug grading.Source: §6 · p. 9

Limitations

• Noisy source tests sometimes enforce unstated requirements, so some failures reflect requirements that cannot be inferred from earlier issue descriptions rather than incorrect earlier fixes.Source: Limitations · pp. 9–10
• The Python-only release supports chains of at most five instances, limiting generalizability to other ecosystems and evaluation of longer maintenance horizons.The authors state the construction pipeline is not Python-specific and could extend as executable SWE datasets expand.Source: Limitations · p. 10

• Each chain is a time-ordered list of bug-fix tasks sharing one Docker image and base commit.Source: §A.1 · p. 11
• Each per-bug entry is a self-contained SWE-bench-style task instance.Source: §A.1 · p. 11

• 54 repositories are represented; tobymao/sqlglot has the most chains (12), followed by s-knibbs/dataclasses-jsonschema (7).Table 9 lists the chain and bug counts for all repositories.Source: §A.2 · p. 11

A.3 Per-Source Breakdown

• 52 chains (52%) come from SWE-rebench-v2, the largest source contribution; SWE-rebench contributes 17, SWE-bench Live 12, SWE-Gym 7, SWE-bench 6, and cross-source chains 6.SWE-bench Pro contributes no chains surviving filtering.Source: §A.3 · p. 12

A.4 Context Usage

• SEQ+MEM reaches average maximum contexts 10–16k tokens above corresponding ORACLE and SEQ runs across the three context-management configurations.The passage attributes the larger contexts to carrying conversation history across bugs.Source: §A.4 · p. 12

• Across every source-dataset group, SEQ and SEQ+MEM underperform ORACLE, with gaps ranging from 2.0 to 10.8 percentage points.Source: B Contamination Analysis · p. 12, Table 10 · p. 13
• CHAINSWE’s per-repository breakdown covers 54 repositories, 100 chains, and 304 bugs.Source: Table 9 · p. 13
• For issues dated within one year before model release, success falls from 46.7% under ORACLE to 42.4% under both sequential modes.Source: B Contamination Analysis · p. 13, Table 11 · p. 13

C Failure Mode Examples

• Across the examples, each downstream bug passes from its oracle prior state but fails from the agent’s accumulated state, defining a chain error.All examples are real BASELINE rollouts.Source: C Failure Mode Examples · p. 13
• An overshoot in dspy bug 2 adds an out-of-scope change that interferes with bug 3, causing four previously passing tests to fail in SEQ.Bug 3 passes from ORACLE; the unintended change to functional/functional.py causes the final repository state to diverge from the oracle.Source: C Failure Mode Examples · p. 14
• An undershoot in dataclasses-jsonschema bug 2 leaves its target file unchanged, so bug 3’s patch is applied to an incomplete state and several previously passing tests fail.The bug 2 patch is only a standalone reproduction script; bug 3 passes from ORACLE.Source: C Failure Mode Examples · p. 14
• In meltano bug 2, the agent fixes the target bug in SEQ but regresses previously working behavior, causing test_add_sqlsink_and_get_sink to fail.Bug 2 passes from ORACLE; the regression follows a modification to singer_sdk/target_base.py on an inherited state.Source: C Failure Mode Examples · p. 15
• Amaranth bug 1 is only half-finished, leaving two required files untouched; bug 2 then passes without new regressions, but test_is_compliant still fails in SEQ.Bug 2 passes from ORACLE; its patch is applied on top of bug 1’s incomplete state.Source: C Failure Mode Examples · p. 15
• The brutils bug 2 patch changes only two of seven gold files, leaving bug 3’s SEQ run unable to collect tests and exiting with code 2.Bug 3 still succeeds from ORACLE.Source: C Failure Mode Examples · p. 17

D Agent Prompts

• The appendix’s D section lists the prompts used by SWE-EDIT.Source: D Agent Prompts · p. 17

D.1 Viewer Subagent Prompt

• The viewer subagent receives numbered file contents and a natural-language query, then returns relevant line ranges for the main trajectory.Its prompt requires focused, sorted, inclusive 1-indexed ranges covering complete logical blocks, merging adjacent ranges, or returning an empty array if nothing is relevant.Source: §D.1 · pp. 17–18
• The viewer subagent is instructed to analyze the file content and query, then output relevant line ranges as a JSON array.Source: §D.1 · p. 18

D.2 Editor Subagent Prompt

• The editor subagent must return exact search-replace blocks, with unique SEARCH matches and no extra explanation; an empty SEARCH block requests a full-file rewrite.Multiple edits may be supplied as separate blocks.Source: §D.2 · p. 18

D.3 Context Summarization Prompt

• When context exceeds its token budget, the SUMMARIZE configuration replaces earlier trajectory with a structured summary preserving task, repository state, changes, commands and results, next steps, and key facts.The prompt asks for exact paths, symbols, errors, and commands where known, and says not to invent information.Source: §D.3 · pp. 18–19

D.4 System Prompt

• The system prompt directs the agent to make minimal changes to non-test files, reproduce and fix the issue, rerun the reproduction, and consider edge cases.It says tests have already been handled and should not be modified; the development environment is preconfigured.Source: §D.4 · p. 19