PaperlayerView exact revision on arXiv

Source-linked AI summary

SWEeper-Bench: Can Agents Discover Bugs in Interactive Software?

Yang Yao, Haozhe Chen, Bingyi Kang, Karthik R Narasimhan, Zhuang Liu

arXiv:2610.14096v1cs.SE

TL;DR

Interaction-dependent bugs in generated software are hard to spot from code alone, leaving users to discover and report them. SWEeper-Bench evaluates agents on finding and fixing such bugs across 200 real web-app tasks; the best of 15 frontier agents passes 59.0%, with identifying buggy behavior the biggest bottleneck.

  • Problem

    Interaction-dependent bugs are hard to spot from code alone, so users often discover and report them, making the repair loop slow and expensive.

    Source: §1 · p. 2

  • Method

    SWEeper-Bench tests agents on 200 tasks from real web-app bug fixes, using open-ended requests, interactive browser tests, and a verifier that judges user-observable behavior.

    Source: §1 · p. 3, §2.1 · p. 3

  • Results

    59.0%: the best of 15 frontier agents passes only this share of tasks, with identifying buggy behavior the biggest bottleneck.

    Source: §1 · p. 3

  • Takeaways & Limitations

    SWEeper-Bench shifts coding-agent evaluation from repairing bugs to discovering them, a step toward agents that build, test, and fix software on their own.

    Source: §1 · p. 3

    SWEeper-Bench covers only web applications and scores one known bug per task, measuring recall rather than precision; its behavior tests assess functional correctness, not visual design, usability, or responsiveness.

    Source: §6 · p. 10

Abstract

from arXiv · show

Coding agents increasingly interact with users by generating interactive software on the fly rather than replying in text, yet many bugs in such software reveal themselves only after multi-step interactions. Today, users must discover these bugs and wait for fixes, making the development loop slow and expensive. Existing benchmarks typically assume the bugs have already been identified, leaving agents' ability to find interaction-dependent failures largely unevaluated. We introduce SWEeper-Bench, 200 tasks built from real bug fixes in 200 widely used open-source web applications. Each task gives the agent only the codebase and an open-ended request such as "Find and fix issues in Focalboard's board sharing and membership." Because these bugs are hard to detect from code alone, agents must design interactive tests to uncover them. An agentic verifier then judges each repair by what a user would observe in the browser. The best of 15 frontier agents passes only 59.0% of tasks. Through stage-by-stage analysis of agent trajectories, we show that the biggest bottleneck is identifying the buggy behavior: agents often reach the relevant feature but never trigger the failure or fail to recognize it. SWEeper-Bench measures progress toward agents that close this loop themselves, delivering software that works without waiting for user feedback.

1 Introduction

Interaction-dependent bugs often emerge only after specific user sequences, leaving users to discover and report failures; SWEeper-Bench asks whether agents can find and fix them on their own.

1 Introduction

  • Existing benchmarks typically begin with a reported issue, so they evaluate diagnosis and repair rather than discovery of interaction-dependent failures.Their hidden unit tests check internal state and can reject valid fixes that use a different approach.Source: §1 · p. 2
  • SWEeper-Bench contains 200 real bug-fix tasks from open-source web applications, with open-ended prompts and browser-based, user-observable verification.Agents must design interactive tests to expose failures before repairing them.Source: §1 · p. 3, Figure 3 · p. 3
  • 59.0% is the best pass rate across 15 frontier agents, and nine pass fewer than half of tasks.Agents reach the feature containing a bug in 75.4% of trajectories but identify the buggy behavior in only 50.0%.Source: §1 · p. 3
  • SWEeper-Bench shifts coding-agent evaluation from repairing known bugs toward discovering them and closing the build-test-fix loop independently.Interaction-dependent failures are hard to spot from code alone and often surface only when people use the software.Source: §1 · pp. 2–3

2 Constructing SWEeper-Bench

SWEeper-Bench pairs buggy web applications and open-ended tasks with browser-run behavior tests, covering task curation, agent prompts, and repair verification.

2 Constructing SWEeper-Bench

  • Each task pairs a buggy application and open-ended prompt with two browser-run behavior tests.The target test reproduces the bug; the preservation test checks that nearby features still work.Source: §2 · p. 3, §2.3 · p. 4

2.1 Curating Tasks

  • Each task pairs a distinct open-source web application with a reproducible, user-facing bug drawn from a merged fix, spanning seven domains.The benchmark uses the buggy parent commit and merged fix as the reference repair; selected bugs should be hard to find but obvious once seen.Source: §2.1 · pp. 3–4, Figure 4 · p. 4

2.2 Open-Ended Prompt

  • The agent receives a one-sentence product-area request, buggy code, a running application, test tools, and a browser, then chooses scenarios and when to stop.The prompt does not specify the failure, code location, or reproduction steps.Source: §2.2 · p. 4

2.3 Agentic Verifier

  • A verifier tests each repair in a fresh application copy through the browser; both target and preservation behavior tests must pass.Verifier configurations came within one percentage point of ideal rates across the tested buggy versions and reference fixes.Source: §2.3 · pp. 4–5

3 How Do Agents Perform?

SWEeper-Bench remains difficult for frontier agents, with failures concentrated in finding and recognizing buggy behavior rather than preserving working features. More time or browser activity alone does not predict higher pass rates.

3.1 Benchmark Results

  • 59.0% is the highest pass rate on SWEeper-Bench; nine of 15 agents pass fewer than half of tasks, while preservation pass rates remain 97.5%–100%.Grok 4.6 with Cursor leads, followed by Qwen 3.8 Flash at 57.5% and three agents at 56.5%.Source: §3.1 · p. 5, Table 2 · p. 6

3.2 Where Do Agents Stumble?

  • 75.4% of trajectories reach the target feature, but only 50.0% identify the buggy behavior; agents may miss the interaction or fail to recognize the observed failure.On Focalboard, all four inspected agents reach the empty recipient sidebar, but they respond differently: two fix it, one trusts unit tests and judges it intended, and one pursues other bugs.Source: §3.2 · p. 6, Figure 6 · p. 7, Figure 7 · p. 8
  • 43.8% of trajectories reach root-cause diagnosis, while 41.0% pass the target test and 40.8% also preserve nearby behavior.Remaining failures include misattributing causes and incomplete fixes, such as correcting a default stock value but omitting the API field that supplies it.Source: §3.2 · p. 6

3.3 Behavioral Tendencies

  • Longer work does not track higher pass rates: Qwen 3.8 Flash and Kimi K3 average 102 and 90 minutes per task for similar or lower pass rates than shorter-running agents.Long runs can fail to produce a patch, while short runs risk stopping before finding the bug.Source: §3.3 · pp. 6–7
  • Browser activity also does not track pass rate: browser actions comprise 17%–62% of recorded turns, yet Qwen 3.8 Max makes 517 actions per task and passes 28.0%.The section attributes performance more to which scenarios agents test and whether they recognize failures.Source: §3.3 · p. 7

4 What Affects Performance?

The experiments find that browser testing is rarely used without prompting, and that agents perform much better when given the bug description. Harness choice and additional time have smaller or limited effects.

4.1 Does Browser Prompting Help?

  • Without the browser-testing instruction, GPT-5.6 Luna uses a browser on 2 of 200 tasks and GLM 5.3 on 6, versus all 200 when instructed; pass rates fall to 39.5% and 31.5%, respectively.The browser remains available; without the instruction, agents instead read code, run shell commands, and send HTTP requests.Source: §4.1 · p. 8, Table 3 · p. 8

4.2 Is Finding Harder Than Fixing?

  • 96.0%–96.5% of tasks pass with a detailed bug prompt, versus 35.0%–51.5% in the main evaluation, and the gap between agents nearly disappears.For GPT-5.6 Luna, median browser launches fall from 26 to 14 and median changed lines from 49 to 13.Source: Table 4 · p. 8, §4.2 · p. 8

4.3 Does the Harness Matter?

  • 35.0%–42.5% of tasks pass across five GPT-5.6 Luna harnesses; Luna does best in Codex, while the model accounts for most variation across agents.The comparison holds the model constant across harnesses.Source: §4.3 · p. 9

4.4 Does Working Longer Help?

  • Pass rates rise from 20 to 40 minutes but not beyond; on 40 tasks, peak rates are 50.0%, 52.5%, and 62.0% for GPT-5.6 Luna, GPT-5.6 Terra, and GLM 5.3.These differences are not statistically significant, and the experiment gives no sign that more time alone closes the gap.Source: §4.4 · p. 9

5 Related Work

Prior software-engineering benchmarks generally specify the bug or goal, while SWEeper-Bench evaluates discovery from a product-area prompt. Related testing and code-review approaches differ in whether they assess repairs and interaction-dependent failures.

Software engineering agents and benchmarks

  • Repository-level and software-task benchmarks typically state what to fix or achieve, evaluating stages after a bug report; SWEeper-Bench names only a product area and evaluates producing the report.The related work describes SWE-bench and successors as starting from reported issues.Source: §5 · p. 9

Testing and verification

  • SWT-Bench (Mündler et al., 2024) starts from a known issue and WebProber (Ye et al., 2025) reports usability issues without evaluating repairs; SWEeper-Bench tests and fixes bugs before they are known.Its agentic verifier checks step-by-step browser behavior and accepts fixes that restore what users observe.Source: §5 · p. 10

AI code review

  • Code-review tools and benchmarks examine unreported problems, but mostly analyze diffs and surrounding code; SWEeper-Bench targets bugs exposed by sequences of actions in a running application.Source: §5 · p. 10

6 Conclusion and Discussion

Agents pass at most 59.0% of SWEeper-Bench tasks when they must find bugs themselves, despite fixing bugs reliably once given a report. The discussion identifies testing and recognizing failures as the gap, while limiting the benchmark to web applications and one known bug per task.

6 Conclusion and Discussion

  • Agents pass at most 59.0% of SWEeper-Bench tasks when they must find bugs themselves, although they fix bugs reliably once given a report.The gap lies in testing: agents must choose scenarios, exercise the interface, and recognize failures.Source: §6 · p. 10
  • As agents build software on the fly, the authors argue they should use applications as users do and judge behavior by intended function, not only expected tests.User-observable verifiers could also reward this kind of testing during training.Source: §6 · p. 10
  • The benchmark covers only web applications and one known bug per task, measuring recall rather than precision and not scoring visual design, ease of use, or responsiveness.Other real fixes earn no credit, and unnecessary changes go unpenalized unless they break a preservation test.Source: §6 · p. 10

Ethics statement

  • Experiments ran in isolated environments without affecting original repositories, deployed products, or real user data; the benchmark excludes security vulnerability discovery and exploitation.The work involved no human subjects or private user data.Source: Ethics statement · p. 11

A Prompts

The prompt gives agents an open-ended product-area request and asks them to find and fix issues through browser-based exploration. It also specifies local-environment constraints, verification procedures, and a required code change.

A Prompts

  • The default task description names a product area; the detailed-prompt ablation replaces it with a description of the target bug while leaving the rest of the prompt unchanged.For Focalboard, the open-ended request concerns board sharing and membership, while the detailed version specifies the expected sidebar behavior after sharing.Source: A Prompts · p. 15
  • Agents must use the local checkout and installed tools, avoid network access, and explore the running product in Chromium and Playwright as a real user.They are instructed not to stop after the first issue, but to keep exploring screens and controls in the task area.Source: A Prompts · p. 15
  • The prompt directs agents to verify through the product UI and run appropriate tests or build scripts using the prescribed product-container tools.It specifies opening the UI with preinstalled Playwright/Chromium and locating scripts in project documentation or CI.Source: A Prompts · p. 15
  • The deliverable is an in-place product-source change with a clean, reviewable diff; an empty diff is not acceptable and agents must not commit.Source: A Prompts · p. 15

B Example Behavior Test

Table 6 gives the Focalboard task’s target behavior test, reproduced verbatim from the benchmark. The verifier follows its browser steps and returns a verdict at the final step.

B Example Behavior Test

Related papers

Loading 2610.14096v1…