PaperlayerView exact revision on arXiv

Source-linked AI summary

SWE-Shift: Evaluating Coding Agents in Evolving Workspaces

Victor Shea-Jay Huang, Rongqing Li, Zhangyue Yin, Kelong Mao, Yaqi Zhao, Xinyuan Wang, Yao Yin, Lirui Luo, Renrui Zhang, Hongsheng Li

arXiv:2610.32196v1cs.AIcs.CLcs.LG

TL;DR

Repository coding benchmarks largely assume tasks and environments stay fixed, leaving mid-execution changes insufficiently tested. SWE-Shift evaluates matched continuations from a shared saved state across task-relevant updates. All 17 systems have lower Shifted than Clean success, showing that unchanged-task success may not survive an update.

  • Problem

    Repository-level benchmarks largely assume tasks and environments remain fixed during execution, leaving agents’ ability to handle mid-execution changes insufficiently tested.

    Source: §1 · pp. 1–2

  • Method

    SWE-Shift resumes matched Clean and Shifted continuations from the same saved history and workspace, applying a task-relevant update only in Shifted.

    Source: §1 · p. 2

  • Results

    All 17 evaluated systems have lower observed success after intervention; GPT-6 Astra falls from 67.35% on Clean continuations to 24.79% on Shifted continuations.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Success under unchanged conditions often does not survive an update, and family-level and trajectory analyses reveal failures in state recovery, preservation, and evidence use.

    Source: §6 · p. 10, §5 · p. 10

    The benchmark uses one update per run across 120 designed scenarios, with model randomness uncontrolled; repeated changes in natural workflows remain future work.

    Source: §5 · p. 10

Abstract

from arXiv · show

Repository-level coding benchmarks largely assume that tasks and environments remain unchanged throughout execution. In software development, code, dependencies, runtime state, and requirements can change while an agent is still working. We introduce SWE-Shift, a benchmark for evaluating coding agents under such mid-execution changes. From the same saved history and workspace, a Clean continuation proceeds without intervention, while a matched Shifted continuation receives a task-relevant update. Success requires completing the current task, preserving still-valid work, and satisfying the applicable state and evidence checks. The benchmark contains 120 scenarios across 12 change families, each evaluated at nine scheduled intervention positions. We evaluate 17 coding-agent systems, all of which have lower Shifted than Clean success. GPT-6 Astra drops from 67.35% to 24.79%, with 20.34% paired success. Tool/session changes and stale evidence yield zero paired success across all systems. Late-position paired success is lower for some systems and higher for others. These results show that success on the unchanged task can coexist with failures to preserve work, recover state, or refresh evidence after an update.

1 Introduction

Static coding benchmarks do not test whether agents can preserve success when tasks or environments change mid-execution. SWE-Shift uses matched Clean and Shifted continuations to evaluate that robustness and finds lower Shifted success across all 17 systems.

1 Introduction

  • Matched Clean and Shifted continuations resume from the same saved history and workspace, isolating whether success from that state survives a task-relevant change.Clean proceeds without intervention; Shifted receives the update, and each continuation is judged under its applicable conditions.Source: §1 · p. 2, Figure 2 · p. 2
  • The benchmark contains 120 scenarios across 12 change families, evaluated at nine intervention positions, and requires preserving valid work and satisfying post-update requirements.Changes are designed to matter without making successful completion impossible.Source: §1 · p. 2
  • All 17 evaluated systems have lower observed success after intervention; GPT-6 Astra drops from 67.35% on Clean continuations to 24.79% on Shifted continuations.The gap varies across change families and intervention positions.Source: Figure 1 · p. 1, §1 · p. 2
  • SWE-Shift evaluates robustness to mid-execution change, distinguishing failure on the underlying task from loss of a successful continuation after an update.Existing repository benchmarks such as SWE-bench largely assume tasks and environments remain unchanged during execution.Source: §1 · pp. 1–3

2 Related Work

Prior repository benchmarks commonly evaluate issue resolution with fixed tasks and environments. SWE-Shift compares benchmark designs across several forms of change and adds matched continuations, broader environment changes, and fresh-evidence checks.

2 Related Work

  • SWE-Shift combines matched continuations from the same recorded prefix, changes across multiple parts of the task environment, and checks that reject loss of still-valid behavior despite passing original task tests.Related evaluations address user edits, repository inconsistencies, revised requirements, state dependencies, and asynchronous events.Source: §2 · p. 3
  • Repository benchmarks such as SWE-bench normally evaluate issue resolution while keeping the task and environment fixed during execution.Table 1 compares evaluation designs; “mid-episode” denotes an external intervention after execution begins, and “fresh evidence” checks completion claims against the current state.Source: §2 · p. 3, Table 1 · p. 3

3 SWE-Shift Benchmark

SWE-Shift evaluates task-relevant, recoverable changes through paired continuations from a shared execution state. Its success criteria cover current task requirements as well as preservation, recovery, and valid evidence.

3 SWE-Shift Benchmark

  • SWE-Shift builds scenarios by combining repository tasks with task-relevant changes introduced during execution.The benchmark draws on 23 task-root snapshots from 17 repositories.Source: §3.1 · p. 3

3.1 Designing Task-Relevant Changes

  • Changes must be relevant, so ignoring them can invalidate an otherwise successful continuation, and recoverable, so at least one valid completion path remains available.The scenario checks reflect post-update requirements and require preserving work and effects that remain valid.Source: §3.1 · pp. 3–4

3.2 Matched Continuation Protocol

  • Clean and Shifted continuations restore the same recorded prefix and workspace; only after the intervention can their observations and actions diverge.The Clean continuation is a matched baseline for completing the task from that state, while Shifted tests whether success survives the change.Source: §3.2 · pp. 4–5

3.2.1 Intervention Timing

  • Nine scheduled positions, T10–T90, span 10%–90% of eligible reference progress, but do not represent normalized progress through each evaluated agent’s task.The evaluated system may have reached a different workspace state at the same scheduled position.Source: §3.2.1 · p. 5

3.3 Success Criteria and Metrics

  • A continuation succeeds only when it passes every check applicable to its ending conditions, including task, preservation, recovery, and evidence requirements.Paired credit covers joint Clean and Shifted success, while early completion before intervention can also receive credit; PSR summarizes these forms of credit.Source: §3.3 · pp. 5–6
  • Conditional Retention Rate measures the fraction of Clean success also retained in the Shifted continuation.Scenario aggregation weights change families equally, then clusters within families and scenarios within clusters equally.Source: §3.3 · p. 6

4 Experiments

The experiments evaluate 17 coding-agent systems across matched Clean and Shifted continuations, probing overall success, change-family and timing effects, task-test blind spots, and diagnostic controls. All systems show lower Shifted than Clean success, while outcomes vary by change family, intervention timing, state handling, and token budget.

4.1 Experimental setup

  • The evaluation uses the same 120 scenarios and nine intervention positions for all 17 systems, with a 1,000,000-token post-fork budget per continuation.The budget counts cumulative input plus output tokens across post-fork model calls.Source: §4.1 · p. 6, §4 · p. 6

4.2 RQ1: Does coding capability survive a workspace change?

  • Across 17 systems, Clean success ranges from 10.71% to 67.35%, versus 1.16–24.79% Shifted and 0.90–20.34% PSR; GPT-6 Astra scores 67.35%, 24.79%, and 20.34%, respectively.Its CRR is 30.20%, meaning fewer than one-third of its weighted Clean successes also succeed under change.Source: §4.2 · p. 7, Table 2 · p. 7

4.3 RQ2: Which changes are persistent failure points?

  • PSR is 0.00% for all 17 systems on tool/session changes and stale evidence; 16 systems score zero on runtime/resource changes, where MiMo-V2.5-Pro reaches 0.60%.For disclosure changes, 13 systems score zero; Claude Fable 5.1 reaches 16.20% and GPT-6 Astra 14.90%.Source: §4.3 · p. 7

4.4 RQ3: Does the timing of a change affect continuation?

  • Mean PSR at T70–T90 is lower than at T10–T30 for both GPT, both Claude, and both MiniMax systems, but higher for GLM-5.3-Flash, both DeepSeek, and both Kimi systems.GPT-6 Astra loses 20.05 percentage points, while GLM-5.3-Flash gains 2.97 points.Source: §4.4 · p. 8, Figure 4 · p. 8

4.5 RQ4: What do task tests miss after an update?

  • Original tests can pass while post-update checks fail: Django’s 39 tests pass despite removal of a collaborator’s API, and Astropy’s successful compensation still fails workspace preservation.In the scikit-learn case, original checks pass despite missing fresh evidence and an edit-scope violation.Source: §4.5 · pp. 8–9

4.6.1 State refresh and retained history

  • Adding history reset gives the largest incremental gain in four of five systems; the complete state-access intervention adds 5.98–8.71 PSR points, yet final PSR remains 10.89–17.54%.Qwen3.8-27B gains more from state refresh (2.89 points) than history reset (2.63).Source: §4.6.1 · p. 9, §4.6 · p. 9, Figure 5 · p. 9

4.6.2 Post-fork token budget

  • Each post-fork token-budget doubling through 4B adds 0.79–2.67 PSR points across five systems, while quadrupling beyond 4B adds only 0.19–0.28 points total.At 16B, PSR remains 7.15–11.98%.Source: §4.6.2 · p. 9, Figure 5 · p. 9

5 Implications and Limitations

Tool/session changes and stale evidence are shared failure points, while trajectory analysis shows original-task tests can miss failures to preserve collaborator work or use current evidence. The evaluation is bounded to designed scenarios with one update per run, and some factors remain uncontrolled.

5 Implications and Limitations

  • Tool/session changes and stale evidence are shared failure points across the evaluated systems.Trajectory analysis shows agents may pass original task tests while removing valid collaborator work or relying on outdated evidence.Source: §5 · p. 10
  • The evaluation covers 120 designed scenarios, some sharing task roots, with one update per run.Repeated changes in natural software-development workflows remain a next step.Source: §5 · p. 10
  • Pairing fixes recorded history and workspace state, but model randomness remains uncontrolled.Intervention position also varies with execution stage, accumulated context, and remaining work; the post-fork token budget is fixed.Source: §5 · p. 10

6 Conclusion

SWE-Shift addresses the limited testing of mid-execution changes with matched Clean and Shifted continuations and success checks for completion, preservation, recovery, and evidence validity. The authors also disclose Codex assistance and the systems’ experimental execution settings.

6 Conclusion

  • SWE-Shift tests whether success under unchanged conditions survives an update, using checks for task completion, preservation, recovery, and evidence validity.Its analyses identify recurring state-recovery, preservation, and evidence-use failures that original task tests can miss.Source: §6 · p. 10

AI use statement

  • OpenAI Codex assisted with manuscript preparation and benchmark data selection and construction.Experiments used Codex app for GPT-5.6 Sol and GPT-6 Astra, Claude Code app for Claude Opus 5 and Claude Fable 5.1, and local serving for open-weight systems.Source: AI use statement · p. 10

A Extended Method

The extended method defines scenarios around meaningful, recoverable updates and evaluates matched continuations from a shared saved state. It validates scenario construction and schedules nine intervention checkpoints across eligible reference actions.

A.1 What changes, and what must survive?

  • Scenarios cover 12 change families, with updates to code, dependencies, collaborators, runtime, sessions, effects, goals, and evidence.A correct continuation must meet current requirements while preserving work and effects that remain valid.Source: §A.1 · p. 15

A.2 What is held fixed at the fork?

  • Clean and Shifted continuations resume from the same saved prefix, including history, workspace, controller state, tools, settings, and counters.The agent receives task and visible state information, while evaluator actor and version bindings are held out; available commands vary by scenario.Source: §A.2 · p. 15

A.3 Construction checks

  • Each scenario must include a relevant update and a preservation-safe recovery accessible through agent tools, and pass construction checks.Negative examples ensure blind continuation fails task or preservation checks, while stale repair fails an applicable Shifted check even if original task tests pass.Source: §A.3 · p. 15, Table 3 · p. 16

A.4 How T10–T90 is constructed and executed

  • Checkpoint construction uses eligible actions before update-dependent reads, tool calls, or mutations, and rejects prefixes that cannot yield nine real actions.The procedure may split the longest eligible file read into contiguous line ranges to reach nine actions, without adding empty reads or no-ops.Source: §A.4 · pp. 15–16
  • Nine checkpoints span 10%–90% of eligible reference progress, but do not require evaluated agents to reach the same semantic task stage.At evaluation, each checkpoint is the number of actual agent actions before the fork, and replay checks the update and recovery at every checkpoint.Source: §A.4 · p. 16

B Extended Experimental Setup

The setup evaluates 17 systems on the same scenarios and intervention positions, using a fixed post-fork token budget. Success requires applicable task, preservation, recovery, and state checks; aggregation weights families, clusters, and scenarios.

B.1 Systems and evaluation conditions

  • All 17 systems use the same 120 scenarios and nine intervention positions, with a post-fork budget of 1,000,000 cumulative input and output tokens per continuation.A system includes its model, wrapper, backend, parser, tools, decoding policy, and execution limits.Source: §B.1 · p. 16

B.2 Success requires meeting all task requirements

  • Strict Clean success requires task and preservation checks plus applicable effect, disclosure, goal, verifier-version, and token-budget checks.Shifted additionally requires recovery and fresh-observation checks; only applicable checks enter the conjunction.Source: §B.2 · p. 17

B.3 Aggregation

  • Paired Success Rate averages paired credit over the nine intervention positions, using family-, cluster-, and scenario-level weighting.Clean, Shifted, and joint-success rates use the same weights; the overall CRR compares time-averaged joint success with time-averaged Clean success, excluding early-completion credit.Source: §B.3 · p. 17

C.1 Base task sources

This section documents the benchmark’s task sources, intervention requirements, validation, and strict success checks. It also specifies how early completion is credited before a scheduled update.

C.1 Base task sources

  • 120 change scenarios use pinned tasks from three source benchmarks and share 23 task-root snapshots from 17 repositories.A task root can support scenarios in more than one change family.Source: §C.1 · p. 17, Table 4 · p. 17

C.2 Requirements by change family

  • The taxonomy groups changes by the dependency they affect and the recovery they require, drawing on prior dynamic-agent evaluations and development incidents.Table 5 lists the requirements for each of the 12 change families.Source: §C.2 · p. 18, Table 5 · p. 18

C.3 Validation checks

  • All 120 T0 cases and 1,080 intervention cells pass construction checks covering preconditions, paired-continuation isolation, recovery, preservation, information exposure, and replay or restore.Each admitted scenario also has a successful reference recovery using only its declared agent-accessible tools, without privileged state-setting; final state and event history are checked independently.Source: §C.3 · p. 18

C.4 Early completion

  • Early-completion credit requires replaying the completed prefix and verifying task, preservation, and freshness conditions before the scheduled intervention.The future update is not applied to the completed terminal state, and CRR uses the joint-success component R.Source: §C.4 · pp. 18–19

D Executable Success Checks

  • Strict success checks apply to the workspace, controller, modeled adapters, and event history, while freshness observations are assessed separately from task and preservation correctness.The event history checks action order and external effects; an observation can meet the observation requirement, while executable checks assess the resulting state.Source: D Executable Success Checks · p. 19, Table 6 · p. 19

E Extended Related Work and Benchmark Comparison

The related-work discussion distinguishes SWE-Shift from benchmarks that study repository repair, specification histories, stateful interactions, or measurement validity. Its comparison emphasizes controlled continuations and post-update requirements.

Repository repair and shared editing

Interaction and specification histories

  • Unlike specification-history studies that reveal requirements before implementation [33] [32], SWE-Shift changes an already-started execution and its external state.This contrast separates clarifying requirements before action from continuing after a dependency or authority changes.Source: E Extended Related Work and Benchmark… · p. 19

State, side effects, and time

Behavioral testing and measurement validity

  • The scoring separates task tests, preservation checks, and required observations, reflecting measurement-validity work [4] and targeted behavioral testing [29].Freshness checks record allowed observations, while task and preservation checks assess behavior; construction checks validate these measures before agent evaluation.Source: E Extended Related Work and Benchmark… · p. 20

Updating information and rebuilding dependencies

  • The taxonomy distinguishes changes to the world from new information about a fixed world, while dependency-rebuild work motivates checking which observations and derived results need refreshing.Source: E Extended Related Work and Benchmark… · p. 20

F Coding-Agent Systems and Reproducibility

This section identifies the evaluated systems and the limits governing their runs. It also points to the data files used for reported results and benchmark counts.

F Coding-Agent Systems and Reproducibility

  • Table 8 lists the 17 evaluated coding-agent systems, with model identifiers and available checkpoint revisions in data/model-cards.json.The table caption explains year links, the rc1* version notation, backend-limit dashes, and per-call context and output limits.Source: Table 8 · p. 22, F Coding-Agent Systems and… · p. 22
  • Each continuation receives a post-fork budget of 1,000,000 cumulative input-plus-output tokens, counted separately for Clean and Shifted runs.Locally served systems also have a 4,096-token per-response Max output limit; this is distinct from the cumulative budget.Source: F Coding-Agent Systems and… · p. 22, Table 8 · p. 22
  • Reported tables and plots use data/results.json, state-handling and token-budget controls use data/controls.json, and construction counts use data/benchmark.json.Source: F Coding-Agent Systems and… · p. 22

G 24-Case Qualitative Audit Set

The 24-case qualitative audit pairs recorded agent actions with independent checks of task success, recovery, preservation, and evidence freshness. Its cases examine how completing the original task or correctly handling an update can still leave revised requirements, workspace state, or evidence checks unmet.

G 24-Case Qualitative Audit Set

  • The audit presents two cases per change family, separating agent actions and self-reported tests from independent evaluator findings; success requires passing all applicable checks.Each case card reports Shifted task, recovery, preservation, and freshness, with additional failed checks shown separately.Source: G 24-Case Qualitative Audit Set · p. 23
  • Astropy and Druid’s guarded registry cases pass Clean and Shifted checks while preserving the updated critical section, helper, and locking behavior.Astropy’s repair extends helper eligibility to properties; Druid registers POW in the locked operation registry and preserves its locking behavior.Source: G 24-Case Qualitative Audit Set · pp. 24–25
  • Bytes passes the original task and V2 recovery check but fails preservation of current API requirements and workspace; its evaluator also reports an unexpected test-mutation digest.The task combines legacy sign-extension repair with a newly introduced framed integer API.Source: G 24-Case Qualitative Audit Set · p. 26
  • Sphinx passes the original task but fails API compatibility because the repair assumes BoundaryAnalyzer has a modname attribute.The repair needed to use the supported attribute-document snapshot API rather than assume earlier analyzer attributes remained available.Source: G 24-Case Qualitative Audit Set · p. 27
  • Druid’s OperationSpec case passes with POW using the shared metadata abstraction and order-sensitive cache behavior, while Caddy’s cookie repair fails despite retaining collaborator registration.Django’s original 39 task tests pass, but preservation fails because the repair removes the shared can_reduce_index_with policy API and makes out-of-scope edits.Source: G 24-Case Qualitative Audit Set · pp. 28–30
  • The gRPC DNS repair passes task, temporary-failure recovery, policy API, and reporting checks while preserving the collaborator’s controls.The resolver stops publishing an empty address list on errors and reports lookup errors through the surviving policy helper.Source: G 24-Case Qualitative Audit Set · p. 31

Express

  • The active-worker URL task passes, but combined preservation fails and one prohibited-behavior event is recorded.The case asks for URL-object support in res.location/res.redirect while retaining replacement transport and redirect behavior.Source: G 24-Case Qualitative Audit Set · p. 32

Fluentd

  • The original task and replacement-worker activity pass, but the generation-aware ACK-path preservation check fails.The evaluator found that a predecessor-generation acknowledgment was accepted.Source: G 24-Case Qualitative Audit Set · p. 33

Caddy

  • Correct session and workspace bindings do not prevent the cookie-helper and task checks from failing, alongside one prohibited-behavior event.The requested change was to set Secure and SameSite=None on sticky cookies.Source: G 24-Case Qualitative Audit Set · p. 34

Pylint

  • The updated similarity-checking interface is preserved, but the original and current task checks fail despite a report of 18 passes.The terminal task run contains one failure, showing that the available test command did not establish current task acceptance.Source: G 24-Case Qualitative Audit Set · p. 35

External effects

  • Astropy and Bytes cases pass cancellation authority and compensation checks, yet fail frozen-task or workspace-preservation checks.In Bytes, exact compensation and linkage pass, but signed-getter behavior and workspace preservation fail; an out-of-scope edit remains outside the getter bodies.Source: G 24-Case Qualitative Audit Set · pp. 36–37

Disclosure

  • Django’s original and case task checks pass and disclosure is recorded, but preservation fails.The newly disclosed integration constraint requires exact index names, canonical model identity, and idempotence.Source: G 24-Case Qualitative Audit Set · p. 38
  • SymPy task, recovery, observation-order freshness, and disclosure validity all fail, while preservation passes.The case reveals a fixed-goal requirement for Unicode normalization and stable digit suffixes; widening the character match alone does not establish compatibility.Source: G 24-Case Qualitative Audit Set · p. 39

Corrective feedback

  • Scikit-learn’s canonical joblib task behavior passes, but preservation flags an unexpected tracked diff scope including an added test edit.The feedback required joblib immediately after sklearn while retaining prior ordering and missing-module behavior.Source: G 24-Case Qualitative Audit Set · p. 40

Fluentd

  • Fluentd’s nil-safe ACK race check passed, but preserving failed-delivery semantics for missing ACK metadata and the exclusive-debug check failed.The evaluator found that missing ACK metadata returned nil.Source: G 24-Case Qualitative Audit Set · p. 41

Pylint

  • Pylint’s revised zero-mode goal failed because the behavioral check found similarities computed, despite correct goal binding.The revised requirement was a true ingestion no-op: no stream reads, decoding, or LineSet allocation at zero, while retaining nonzero behavior.Source: G 24-Case Qualitative Audit Set · p. 42

Sphinx

  • Sphinx’s revised inherited-attribute check failed with a missing shared entry, although the original task, goal binding, and workspace scope passed.The revised request required once-per-module-per-call parsing and nearest-MRO precedence without a global cache.Source: G 24-Case Qualitative Audit Set · p. 43

Stale evidence

  • Stale-evidence cases failed when a repair left the evidence route stale or restored the current route without refetching fresh evidence.In gRPC, route mode remained stale; in scikit-learn, task and route identity passed, but refetch, freshness, disclosure, and write-scope checks failed.Source: G 24-Case Qualitative Audit Set · pp. 44–45

Misbound evidence

  • Misbound-evidence cases failed because observations remained cross-session or misbound despite code edits and, for Express, passing location behavior and the original task.The audit requires observations to belong to the current execution and permitted workspace scope.Source: G 24-Case Qualitative Audit Set · pp. 46–47

Related papers

Loading 2610.32196v1…