PaperlayerView exact revision on arXiv

Source-linked AI summary

SWE-Touch: Benchmarking Coding Agents When Users Touch the Code

Yuqiao Tan, Jinxiang Meng, Fangyu Lei, Minzheng Wang, Shizhu He, Jun Zhao, Kang Liu

arXiv:2608.02499v1cs.SEcs.AIcs.CL

TL;DR

Existing benchmarks do not test how coding agents respond when users modify the shared repository during a task. SWE-Touch introduces controlled task-conflicting Counter-Edits and finds that they lower mean resolve rate by 7.7 points, with degradation persisting on longer-horizon benchmarks.

  • Problem

    Existing benchmarks omit user modifications to shared executable repositories, leaving agents’ responses to code changes during ongoing tasks unevaluated.

  • Method

    SWE-Touch injects validated, task-conflicting Counter-Edits into task-relevant code during agent execution, accompanied by contextual user messages.

  • Results

    7.7 points: Counter-Edits lower mean resolve rate across nine models on SWE-bench Verified, with degradation persisting on SWE-Bench Pro and DeepSWE.

  • Takeaways & Limitations

    Current coding agents do not reliably re-inspect user-modified code and adapt repairs to the resulting workspace state.

  • Takeaways & Limitations

    SWE-Touch characterizes robustness to controlled, simulated task-conflicting edits rather than the full distribution of collaborative user behavior.

Abstract

from arXiv · show

Real-world software development requires coding agents to operate in shared workspaces where users may inspect and modify code during an ongoing task, yet existing repository-level benchmarks typically evaluate agents working alone or restrict user participation to messages. This leads us to ask: how do coding agents understand and respond to code changes in a shared workspace? We introduce SWE-Touch, a framework that stress-tests this setting through validated Counter-Edits: plausible edits to task-relevant code that conflict with task completion. SWE-Touch mines task-critical regions from multiple repair trajectories, uses a separate User Patch Generator to construct the edits, and injects them with contextual user messages when agents reach the relevant code. We evaluate nine coding models on SWE-bench Verified, with additional experiments on longer-horizon tasks from SWE-Bench Pro and DeepSWE. Counter-Edit lowers average resolve rate by 7.7 percentage points on SWE-bench Verified, with degradation also persisting on both longer-horizon benchmarks. Trajectory analysis links these failures to limited awareness of the evolving workspace: agents may retain conflicting code or replace it without sufficiently re-inspecting the repository and validating the revised code with targeted tests. These findings show that strong autonomous performance does not yet ensure the state awareness and adaptive behavior needed for shared-workspace collaboration, and point to detecting workspace changes, reconciling conflicting edits with the task, and verifying the affected behavior as key capabilities for future optimization.

1. Introduction

SWE-Touch extends coding-agent evaluation to shared workspaces where users directly modify executable repository state, a prevalent interaction channel omitted by message-only benchmarks. Its Counter-Edit experiments show that agents often fail to detect, reconcile, and validate conflicting user changes despite strong autonomous performance.

  • Motivation: 59.0% of SWE-chat sessions contain repository changes attributed to users, showing that code edits are an important interaction channel alongside messages.Each user edit changes the program state observed by subsequent agent actions.
  • Method: SWE-Touch injects validated, task-conflicting Counter-Edits into ongoing agent trajectories to evaluate adaptation to user-modified workspace state.It identifies task-critical regions from multiple trajectories and uses a separate User Patch Generator to construct locally plausible edits.
  • Results: 7.7 percentage points: Counter-Edit lowers mean resolve rate across nine models on SWE-bench Verified, with model-level losses ranging from 1.3 to 16.5 points.The degradation persists on longer-horizon tasks from SWE-Bench Pro and DeepSWE, and substantially reshuffles model rankings.
  • Failure modes: 63.3% of failed runs retain the user’s conflicting code, while other failures involve missing modifications, deferring to conflicting code, or revising it without rechecking behavior.These patterns indicate unreliable re-inspection of user-modified code and adaptation to the new workspace state.
  • Implication: Autonomous benchmark performance does not ensure the state awareness required for shared-workspace collaboration.Agents may retain conflicting user code or replace it without sufficiently re-inspecting the repository and validating revised behavior.

2. Related Work

Prior coding-agent benchmarks progressed from self-contained synthesis toward repository-level issue resolution, while interactive benchmarks increasingly model user–agent collaboration and shared-state changes. SWE-Touch extends this line by requiring agents to continue from a user-modified workspace.

  • Benchmarks for coding agents: Coding benchmarks evolved from self-contained program synthesis to repository-level issue resolution using real codebases, repositories, issues, and test suites.This progression also includes complex instructions, realistic edits, diverse function and library use, and multi-step data-science programming.
  • User interaction with agents: Interactive agent benchmarks study multi-turn dialogue, non-collaborative user behavior, shared environments, and agent tracking of user-introduced state changes.Within coding, related benchmarks examine clarification, selective help-seeking, evolving requirements, and corrective feedback.
  • User interaction with agents: SWE-Touch places coding-agent evaluation in a modified workspace, requiring agents to continue from a repair trajectory after user edits.Table 1 compares SWE-Touch with prior interactive coding benchmarks and their user interaction modes.

3. SWE-Touch

SWE-Touch evaluates coding agents in shared workspaces by injecting validated, task-conflicting user edits during execution. It identifies task-critical regions, generates localized Counter-Edits, and delivers them deterministically when agents access affected code.

  • Framework overview: SWE-Touch combines task-critical regions, validated task-conflicting patches, and a deterministic delivery rule for runtime user interventions.The framework applies synthetic user patches with contextual messages while the agent continues from the resulting workspace state.
  • Task-critical regions: Task-critical regions are selected from overlapping read and edit evidence across repair trajectories, prioritizing implementation edits and retaining at most eight regions per task.Edit-based evidence takes priority over read-based evidence, followed by implementation files over tests or metadata.
  • Counter-Edit construction: A separate User Patch Generator produces small, syntactically valid implementation diffs whose behavior conflicts with verified task requirements.Counter-Edits cannot modify tests or benchmark metadata, and their changed regions may extend beyond the selected critical regions.
  • Counter-Edit construction: Counter-Edit validation requires that the user edit alone fails, the reference patch succeeds, and their combination still fails verification.This prevents the intervention from solving the task independently or being trivially combined with the reference repair.
  • Deterministic delivery: Delivery attempts occur when an agent reads or modifies code overlapping the patch region, with K=3 by default and messages delivered regardless of whether the patch changes the workspace.The agent observes the resulting repository state and contextual user message before its next action.

4. Experimental Setup

The experiments evaluate coding agents under autonomous and task-conflicting Counter-Edit conditions on SWE-bench Verified, with separate longer-horizon evaluations on SWE-Bench Pro and DeepSWE. The setup specifies task sampling, intervention schedules, model coverage, and verification-based metrics.

  • Tasks and agent interface: 200 SWE-bench Verified tasks form the main evaluation sample, with retained code patches available for 96.0% of tasks.The remaining 4.0% use text-only feedback because no applicable, non-solving patch is retained; 6.4% of scored Counter-Edit runs contain no scheduled repository change.
  • Longer-horizon evaluations: 25 tasks each from SWE-Bench Pro and DeepSWE are evaluated separately using a 500-step budget, two independent runs, and localized edits at fixed trajectory fractions.These experiments use the same task sets as SWE-Interact, with trigger rules and patch schedules differing from the main evaluation.
  • Models and metrics: Nine coding models are evaluated under Vanilla autonomy and Counter-Edit interventions that schedule task-conflicting runtime edits.The model set includes Claude Opus 4.8, GPT 5.5, GLM 5.1, MiniMax M2.7, MiniMax M2.5, Qwen 3.7 Max, Qwen3-Coder-480B-A35B, Kimi K2.6, and DeepSeek V4 Pro.
  • Models and metrics: Resolve rate is reported as the percentage of tasks passing the complete verifier, alongside retention, defined as the fraction of majority-solve…The supplied passage truncates the remainder of the retention definition.

5. Main Results

Counter-Edit consistently reduces coding-agent performance, with effects varying substantially across models, benchmarks, intervention frequencies, and failure modes. The results indicate that robust recovery requires detecting conflicting edits, reconciling them correctly, and verifying the revised behavior.

  • Aggregate performance: 7.7 percentage points: average resolve rate decreases under Counter-Edit, with every model declining and losses ranging from 1.3 to 16.5 points.Counter-Edit is therefore not a uniform offset to autonomous performance.
  • Aggregate performance: 85.2% to 83.3%: Claude Opus 4.8 remains first, while GPT 5.5 remains second after moving from 80.5% to 79.2%.Rank instability is substantial below the leading pair: MiniMax M2.7 falls from rank 3 to rank 8, while MiniMax M2.5 leads it after Counter-Edit at 66.2% versus 62.7%.
  • Aggregate performance: 96.0% and 95.0%: Claude Opus 4.8 and GPT 5.5 retain the largest shares of majority-solved Vanilla tasks, whereas Qwen3-Coder-480B retains 60.8% and loses 16.5 points.Greater resource use also does not consistently reduce losses: Claude Opus 4.8 adds calls and tokens while losing only 1.8 points, whereas DeepSeek V4 Pro adds calls and tokens while losing 11.0 points.
  • Longer-horizon benchmarks: Both SWE-Bench Pro and DeepSWE show persistent degradation, while additional calls fail to ensure recovery; seven of nine models take more steps on Pro and eight of nine on DeepSWE.On DeepSWE, GLM 5.1 adds 31.4 calls while still declining, and Claude Opus 4.8 adds 9.9 calls while losing 10.0 points.
  • Controls and intervention frequency: −0.1 points: non-solving Co-Edit changes average resolve rate across seven models, compared with 7.2 points lost under Counter-Edit.This contrast indicates that difficulty comes from workspace changes conflicting with task completion, rather than external modification alone.
  • Failure mechanisms: 15.9%: Counter-Edit overturns majority-solved Vanilla pairs, with solved-to-unresolved transitions outnumbering reverse transitions by roughly three to one.Among audited failures, retained conflict accounts for 63.3%, incorrect replacement for 13.9%, incomplete reconciliation for 11.6%, and off-target implementation for 5.5%; revision rate correlates with performance change at Spearman ρ = 0.80, but revision alone does not guarantee recovery.

6. Conclusion

SWE-Touch evaluates coding agents in evolving shared workspaces where simulated users modify task-relevant code during ongoing repairs. Across nine models, Counter-Edit reduces mean resolve rate by 7.7 points on SWE-bench Verified, with degradation persisting on longer-horizon benchmarks.

  • SWE-Touch evaluates coding agents in an evolving shared workspace where a simulated user modifies task-relevant code during an ongoing repair.
  • 7.7 points: Counter-Edit lowers mean resolve rate across nine models on SWE-bench Verified and changes model ordering.
  • Degradation persists on longer-horizon SWE-Bench Pro and DeepSWE tasks, with the most affected models varying across benchmarks.
  • Additional calls often increase without producing recovery under longer-horizon benchmark conditions.

7. Limitations & Future Work

SWE-Touch provides interpretable results through controlled, region-triggered interventions, but future work should model adaptive full-duplex collaboration and optimize agents for nuanced shared-workspace responses.

  • Controlled interventions as a foundation: SWE-Touch uses controlled, region-triggered Counter-Edits to enable matched comparisons and interpretable attribution.The Co-Edit control provides a complementary reference for responses to task-aligned external contributions.
  • Toward full-duplex collaboration: Future evaluations should use adaptive user simulators that observe live agent actions and dynamically decide whether, when, and how to intervene.This would more closely model users monitoring output, diffs, and test results while adapting to the agent’s response.
  • From evaluation to optimization: Future training objectives can reward detecting external changes, inferring user intent, integrating compatible contributions, resolving conflicts, and revalidating shared code.These capabilities target collaboration-aware coding agents in richer environments.
  • From evaluation to optimization: Training across helpful, incomplete, ambiguous, and conflicting interventions could support calibrated behavior beyond always accepting or automatically reverting user edits.The proposed behavior includes seeking clarification when needed and resolving conflicts.

A. Ethics and Data Governance

The evaluation uses synthetic messages and edits without human participants, while released artifacts exclude credentials and private requests. Trajectory releases follow upstream licenses, and reported token costs use endpoint-displayed rates from 27 July 2026.

  • The evaluation uses only synthetic messages and edits, with no human participants involved.
  • Released artifacts contain aggregate statistics but exclude credentials and private requests.
  • Trajectory releases follow upstream licenses, while Figure 3a token costs use serving-endpoint rates displayed on 27 July 2026.OpenRouter rates were promotional prices shown at that time, and cached input was charged at the input rate.

B. Reproducibility Details · B.1 SWE-bench Verified (Main Evaluation)

The main evaluation uses a seeded, frozen cohort of 200 SWE-bench Verified tasks with complete autonomous trajectories from GPT 5.5, GLM 5.1, and MiniMax M2.7. Results are identified by model, setting, run, and task, with model settings documented in Table 8.

  • B. Reproducibility Details: Table 8 documents the model settings used in the main experiments.A dash indicates that a parameter was not set.
  • B.1 SWE-bench Verified (Main Evaluation): 200 SWE-bench Verified tasks form the seeded random main-evaluation cohort.The cohort includes only tasks where GPT 5.5, GLM 5.1, and MiniMax M2.7 each produced complete autonomous trajectories.
  • B.1 SWE-bench Verified (Main Evaluation): GPT 5.5, GLM 5.1, and MiniMax M2.7 define the cohort’s complete-trajectory eligibility condition.
  • B.1 SWE-bench Verified (Main Evaluation): 101 Django tasks constitute the largest repository group in the frozen list.
  • B.1 SWE-bench Verified (Main Evaluation): 31 SymPy and 15 Sphinx tasks are included in the frozen list.
  • B.1 SWE-bench Verified (Main Evaluation): Astropy, Matplotlib, scikit-learn, pandas, Pylint, Requests, and pytest contribute 14, 10, 9, 8, 6, 3, and 3 tasks, respectively.
  • B.1 SWE-bench Verified (Main Evaluation): Each result is identified by model, setting, run, and task.

B.2 SWE-Bench Pro and DeepSWE (Longer-Horizon Extension)

On longer-horizon tasks, Counter-Edits reduce performance by 4.9 points on SWE-Bench Pro and 3.4 points on DeepSWE. These tasks use a 500-step interaction budget and involve substantially larger reference repairs than SWE-bench Verified.

  • Evaluation setup: The extension evaluates 25 selected DeepSWE tasks and 25 selected SWE-Bench Pro tasks with the interaction budget increased from 100 to 500 steps.The agent interface and verifier remain unchanged, and these tasks do not use region-based triggers.
  • Performance degradation: −4.9 points on SWE-Bench Pro and −3.4 points on DeepSWE are the mean differences across two independent runs at K = 3.Figure 3b also reports separate results for K ∈{1, 3, 5}.
  • Task scale: Reference patches change 361.0 lines across 5.44 files on SWE-Bench Pro and 730.2 lines across 7.24 files in DeepSWE, versus 13.3 lines across 1.20 files in Verified.Counter-Edits remain local, averaging 13.0 changed lines in SWE-Bench Pro and 10.8 in DeepSWE.

B.3 Metrics and Runtime · C. User Simulator Prompt and Inputs

The evaluation fixes shared runtime and metric definitions while constructing validated Counter-Edits from trajectory-derived critical regions and contextual simulator messages. It reports broad edit delivery, application, and token-use effects across the evaluated tasks.

  • B.3 Metrics and Runtime: Resolve rate averages three independent runs per model, with equal model weighting; majority outcomes require at least two scored runs, and retention conditions on majority-solved Vanilla tasks.Steps count agent model calls.
  • B.3 Metrics and Runtime: Counter-Edit raises token consumption for seven of nine models, including 37.8% for Claude Opus 4.Tokens sum normalized input, cached-input, and output counts, counting cached input once.
  • B.3 Metrics and Runtime: Critical regions prioritize intersected implementation-file edit spans, then broader edit or read spans, while excluding tests and metadata at designated tiers.Adjacent or overlapping intervals are merged, with at most eight regions and reference-repair fallback when all tiers are empty.
  • B.3 Metrics and Runtime: Across 200 tasks, 174 yield edit-based regions, 24 read-based regions, and 2 reference-repair regions; 192 code edits contain 268 trigger regions.At least one trigger overlaps a changed line for 180 edits, while every trigger does so for 155.
  • B.3 Metrics and Runtime: The separate GPT 5.5-backed User Patch Generator inspects selected code, proposes candidate edits, runs original fail-to-pass tests, and returns validated unified diffs with beliefs and evidence.Harbor evaluates the user edit, reference repair, and cleanly composable combination.
  • B.3 Metrics and Runtime: All 200 tasks are scored in both conditions: 192 receive code edits and 8 receive text-only messages, including runs where agents never reach trigger regions.Edits use context-matched unified diffs rather than fixed line numbers.
  • B.3 Metrics and Runtime: Across 5,399 scored Counter-Edit runs, edits are applied zero times in 346 runs (6.4%), once in 253, twice in 374, and three times in 4,426.Of 192 code edits, 150 preserve reference-patch applicability but break the combined repository, while 42 prevent clean application.
  • C. User Simulator Prompt and Inputs: The patch is applied before simulation; GPT-4o then generates the message from task, delivery stage, target span, latest command/output, recent actions, prior message, and applied diff.The simulator persona is a confident, mistaken repository stakeholder who believes the edit is correct.

D. Failure Analysis Details … E.1 User Patch Generator Prompt

The paper audits Counter-Edit failures with evidence-grounded, independently labeled trajectory analysis, then specifies a User Patch Generator contract for producing and validating plausible task-relevant conflicts. Across sampled trajectories, agents most often oppose user edits, while 150 of 192 code edits satisfy full three-state validation.

  • D. Failure Analysis Details: The audit covers 526 Counter-Edit runs that were solved under Vanilla but became unresolved, spanning 229 model–task pairs, with identities and scores hidden from judges.Each item includes the task, user diff, post-edit events, final repository diff, and verifier evidence.
  • D. Failure Analysis Details: Two independent GPT 5.5 calls agree on the main failure type for 494 of 526 runs (93.9%), while a third call adjudicates disagreements using concrete evidence.Labels must cite specific events, and code or verifier evidence takes precedence over stated intent.
  • D. Failure Analysis Details: Across 90 trajectories, 64 (71.1%) are Changed/Opposed, 25 (27.8%) Kept/Extended, and 1 No Clear Action; 18 opposing trajectories still end unresolved.After the final user edit, pooled median activity is 17 reads, 5 edits, and 1 test.
  • D.1 Trajectory-Labeling Prompts: The trajectory-labeling prompts require narrow, mutually exclusive diagnoses grounded in task, edits, final diffs, commands, and verifier evidence, using insufficient evidence rather than speculation.The rubric covers retained conflicts, incorrect replacements, incomplete reconciliation, off-target implementation, verification breakdown, other failures, and insufficient evidence.
  • D.1 Trajectory-Labeling Prompts: The post-edit prompt classifies each intervention window as COUNTERACT, FOLLOW, or NO_COMMITMENT and separately judges whether the final edit state is RECOVERED, RETAINED, or UNCLEAR.Judgments rely on decisive actions and final repository evidence rather than inferred intent or hidden outcomes.
  • E. Counter-Edit Generation Contract: The User Patch Generator receives the issue, suggested regions, reference repair, fail-to-pass tests, and surrounding code under a contract covering construction rules, validation commands, and output schema.It operates with repository and Bash access to construct one small Counter-Edit rather than solve the target issue.
  • E.1 User Patch Generator Prompt: The generator must produce a sparse, syntactically valid, locally plausible, task-relevant mistaken implementation that interferes with the verified reference repair while avoiding tests, metadata, generated files, and unrelated code.Preferred mistakes include overly narrow invariants, incomplete transformations, misplaced normalization, or removal of information needed by the correct repair.
  • E.1 User Patch Generator Prompt: 150 of 192 code edits pass full three-state validation; 42 are applicable and non-solving but prevent the reference patch from applying cleanly, while 8 tasks use text-only feedback.The three states test the reference repair alone, Counter-Edit alone, and their composition.

F. Model-Wise Paired Interaction Traces

The paired traces contrast solved Vanilla runs with unresolved Counter-Edit runs, showing that models respond inconsistently to user edits and often fail to resolve the task after repeated workspace changes. The examples include rejecting user messages as injected instructions, inspecting diffs before acting, and still leaving the original verifier unresolved.

  • Paired interaction traces: Each card pairs a solved Vanilla trajectory with an unresolved Counter-Edit trajectory for the same task and model.The trajectories are abridged to emphasize responses to the user edit; full event logs are available in the released casebook.
  • Claude Opus 4.8: Claude Opus 4.8 rejected repeated user messages as prompt-injection attempts, applied the reference fix independently, and left three copies of the user block in the final file.The agent used 7 tool calls, 1 edit command, and 2 test commands; its own check reported success, but the original verifier remained unresolved.
  • GPT 5.5: GPT 5.5 inspected the current code and git diff to preserve recarray behavior, but the user edit was applied three times and the original verifier returned unresolved.The trace shows repeated re-checking when source views appeared inconsistent, followed by 21 tool calls, 2 edit commands, and 9 test commands.
  • Cross-model outcomes: Across the remaining paired traces, every reported Counter-Edit outcome says the user edit was applied three times and the original verifier returned unresolved.The traces report varied effort, ranging from 13 to 59 tool calls and from 0 to 5 edit commands.
Loading 2608.02499v1…