Source-linked AI summary
SWE-INTERACT: Reimagining SWE Benchmarks as User-Driven Long-Horizon Coding Sessions
Mohit Raghavendra, Anisha Gunjal, Aakash Sabharwal, Yunzhong He
TL;DR
Existing SWE benchmarks largely evaluate autonomous implementation from complete specifications, leaving interactive goal discovery and iterative refinement under-tested. SWE-INTERACT adapts 75 benchmark tasks into multi-turn workflows with a persona-conditioned simulator that progressively reveals requirements and inspects the workspace. The best models resolve roughly 50% of single-turn tasks but only roughly 25–27% of corresponding multi-turn tasks, showing that strong autonomous coding performance does not reliably transfer.
Problem
Existing SWE benchmarks provide complete requirements upfront, while real coding-agent sessions often involve vague instructions, iterative feedback, and evolving goals.
Method
SWE-INTERACT adapts 75 SWE tasks into multi-turn workflows using persona-conditioned users that reveal requirements progressively, inspect workspaces, and provide targeted feedback.
Results
Roughly 50% resolve rate on single-turn tasks falls to roughly 25–27% in the multi-turn setting for the best models.
Takeaways & Limitations
SWE-INTERACT evaluates interactive goal discovery and iterative refinement as a capability distinct from autonomous implementation performance.
Abstract
from arXiv · showhide
We introduce SWE-Interact, a new testbed for evaluating coding agents on multi-turn, interactive, user-driven software engineering tasks. Existing frontier SWE benchmarks typically provide complete requirements upfront and evaluate agents on autonomous implementation. In contrast, SWE-Interact places agents in a realistic developer workflow: a carefully designed user simulator starts with vague or incomplete instructions, progressively reveals requirements, inspects the agent's workspace, and provides targeted feedback, revisions, and new constraints until the full task goal has been handed off. Grounded in large-scale studies of real coding-agent interactions, this setup tests whether agents can discover user intent, adapt to evolving requirements, and build on their own prior work. Across a suite of frontier and open-weight models, we find that strong performance on single-turn SWE tasks does not reliably transfer to multi-turn, user-driven workflows: the best-performing models solve roughly 50% of single-turn baseline tasks but only 25% of the corresponding SWE-Interact tasks. The strongest models in our evaluation, including Opus 4.8 and GPT 5.5, start strong even in the face of vague initial instructions, persevere until all the requirements are surfaced by the user, integrate them better and write clean code. However, they still suffer from over-agentic coding, forgetting requirements and technical mistakes. Weaker models start poorly under ambiguity, give up early, forget or ignore instructions and rework their code more. Overall, SWE-Interact measures an orthogonal, real-world capability axis for frontier model development: interactive goal discovery and iterative refinement with a user in the loop.
1 Introduction
SWE-INTERACT addresses the gap between autonomous, fully specified SWE benchmarks and real developer-agent workflows, where users progressively reveal and revise requirements. It introduces a multi-turn testbed, realistic interactive user agents, and analyses of agent capabilities and failures.
- Testbed: SWE-INTERACT converts one-shot implementation into a multi-turn workflow where users inspect work and progressively reveal requirements.The setup tests intent discovery, adaptation to evolving requirements, and correctness across turns.
- Motivation: Real coding sessions commonly begin with vague, incomplete instructions and evolve through goal discovery, planning, approval, implementation, revision, and submission.This contrasts with long-horizon benchmarks that provide complete requirements upfront.
- Motivation: Existing user-in-the-loop benchmarks usually start with reasonably specified prompts and static simulated users that cannot inspect workspaces or run commands.They therefore emphasize recognizing missing details more than discovering goals and building on prior revisions.
- Contributions: The benchmark provides 75 multi-turn tasks adapted from popular SWE benchmarks, together with a harness and evaluation setup.The tasks are designed to support interactive development rather than only autonomous implementation.
- Contributions: Its interactive user-agent framework uses detailed personas, task-specific goals, and tools for interacting with the agent and its workspace.This makes the simulated user more closely resemble a real developer collaborator.
- Contributions: The paper analyzes multiturn capability gaps through failure modes, trajectory analysis, and evolution of agent behavior.These analyses complement the testbed and user-agent framework.
2 Problem Setup
The problem setup adapts fully specified SWE tasks into interactive workflows while preserving their codebases and verifiers. A persona-conditioned simulator reveals requirements iteratively, inspects the workspace, and communicates through a constrained user-agent harness.
- Original task setting: A standard SWE task supplies a preformulated goal and evaluation suite, leaving the coding agent primarily focused on implementation.Correctness is commonly checked with unit tests and/or rubrics.
- Interactive setting: Real developer-agent sessions instead begin with brief, incomplete, or vague instructions and require iterative refinement of both goals and implementations.The agent is expected to incorporate user feedback throughout planning and implementation.
- Task Design: 75 tasks are selected from three frontier benchmarks: 25 each from SWE-bench Pro, SWE Atlas (Refactoring), and DeepSWE.The adapted tasks retain their codebases, dependencies, Docker environments, and verifier suites.
- User Simulator: The user simulator separates task goals, user personas, and an agentic harness within a sandboxed architecture.The design treats persona construction as a first-class component of realistic developer environments.
- User Simulator: The persona design is grounded in thousands of SWE-chat messages and emphasizes the iterative, exacting behavior of the Expert Nitpicker persona.These users tend to communicate briefly, care about API details, and add requirements over time.
- User Simulator: The simulator receives workspace snapshots and can run commands such as git, grep, sed, and find, while the solver agent communicates through user messages.Future versions could let the user modify code or run tests, but the current harness does not provide those actions.
3 Experimental Results
Under identical agents, environments, and verifiers, progressively revealed requirements make the same SWE tasks substantially harder and more expensive. Multi-turn trials also involve extended user interaction and workspace inspection.
- Single-turn vs. Multi-turn Results: All models score lower in multi-turn than single-turn evaluation, despite using more steps, tokens, and cost per trial.The comparison uses the original fully specified task as the single-turn baseline and averages two multi-turn user-simulator runs.
- Single-turn vs. Multi-turn Results: Roughly 50% resolve rate on single-turn tasks falls to roughly 25–27% in the multi-turn setting for the best models.Multi-turn trajectories often use 3-4X as many steps as their single-turn baselines.
- User-agent interactions: Trials average about 7 user messages, while some trajectories end after an early correction before further requirements are revealed.Kimi is identified as an outlier that often submits after the first user correction.
- User-agent interactions: The longest reported trajectory contains 27 user messages, 332 user workspace-tool calls, and more than 1000 agent steps.These interactions support targeted feedback and revisions grounded in inspection of the agent’s work.
4 Discussion
The analysis tracks how agents discover task goals from planning through implementation and examines why failures persist despite user feedback. It also evaluates how persona and simulator-model choices shape interactive trajectories and outcomes.
- Goal discovery during planning and implementation: Agents’ initial plans and implementation checkpoints are scored against independently generated sub-goal rubrics to measure goal coverage over the task lifecycle.The same rubrics assess the saved plan and successive implementation revisions, enabling comparison across normalized trajectory progress.
- Goal discovery during planning and implementation: The best models can identify more than 80% of latent goals during planning, but most initially lose coverage when converting plans into concrete implementations.The decline reflects both more generous plan grading and implementation errors that omit key specifications.
- Goal discovery during planning and implementation: Near-perfect goal discovery is necessary but not sufficient for correctness: almost all verifier-correct solutions exceed 90% goal coverage, but some high-coverage agents still fail.Goal discovery measures whether stated goals appear addressed, whereas verifier reward measures whether the final patch passes correctness checks.
- Failure modes: Among 287 audited failed trajectories, Technical implementation bug and Forgotten requirement each account for roughly one third of assigned semantic labels.Misinterpretation/Bad assumption accounts for about 14%, while Missing user requirement accounts for about 12%; labels are independently assigned and may overlap.
- User simulator ablations: The Expert Nitpicker persona and simulator-model choice alter trajectory length, critique behavior, and final performance relative to neutral or alternative simulator designs.GPT 5.5-based simulators often produce 1.5–2X longer interactions and explore workspaces 4–5X more using tool calls.
5 Background and Related Work
Prior benchmarks extend software-engineering tasks across multiple steps or simulate user interaction, but often predetermine turns or use cooperative users. SWE-Interact instead combines latent requirements, persona-conditioned feedback, workspace inspection, and verifier-based success.
- SWE Benchmarks: Recent SWE benchmarks create long-horizon, stateful, or multi-round codebase changes, but their interaction turns are predetermined rather than interactive.These settings test preserving prior behavior while pursuing new objectives across multiple steps.
- Multi-turn Agent Studies: Other benchmarks study underspecification, clarification, and user-intent modeling, but their simulated users are described as simple, non-agentic, and cooperative.This differs from real-world coding-agent users discussed in the paper’s motivating studies.
- SWE-Interact: SWE-Interact gives the simulated user latent requirements that are revealed through persona-conditioned feedback and workspace interactions, while final success remains judged by the original verifier.The design combines user-agent interaction with final-state verification in the software-engineering domain.
6 Conclusion
SWE-INTERACT reframes software-engineering benchmark difficulty around interaction by progressively revealing requirements through a realistic, tool-using user simulator. The evaluation finds that interactive workflows substantially reduce resolution rates even as frontier models recover goals and revise code.
- Conclusion: SWE-INTERACT uses a persona-conditioned simulator that begins with incomplete instructions, progressively reveals requirements, inspects the workspace, and provides targeted feedback until handoff is complete.Final success is judged by the original task verifier rather than simulator approval.
- Conclusion: The interaction-centered shift roughly halves the resolve rates of the strongest models while substantially increasing interaction length and token use.The benchmark holds the implementation tasks and verifier constant while changing the workflow to an interactive developer setting.
- Conclusion: Frontier models often persevere under ambiguity, recover most intended goals, and produce cleaner code through revision, but still introduce technical bugs and forget requirements.They also struggle to maintain correct implementations as feedback accumulates.
A.1 Multi-turn Resolve-Rate Confidence Intervals
The appendix reports Wilson 95% confidence intervals for pooled multi-turn resolve rates across two simulator runs for each model.
- Multi-turn resolve-rate confidence intervals: Wilson 95% confidence intervals are reported for pooled multi-turn resolve rates, with each model evaluated across two 75-task simulator runs.The intervals summarize uncertainty around pooled performance estimates.
A.2 Qualitative Trajectory Comparison
The trajectory comparison contrasts real and simulated coding sessions in which brief, vague requests are refined through successive user corrections. The examples show agents implementing successive API, schema, relationship, and metric changes as requirements emerge.
- Initial request: The task begins with a vague request to add multiplexing over kcp-go, carrying multiple independent ordered streams over one connection.The neutral version additionally specifies per-stream flow control and priority scheduling, while another prompt leaves file layout and framing to the agent.
- Progressive corrections: Requirements emerge incrementally across API signatures, configuration fields, database tables, SNMP counters, relationships, and stream-priority behavior.User feedback specifies exact constructor and stream-open surfaces, schema extraction, foreign keys and join tables, and required counter integration points.
- Schema refinement: The schema trajectory expands from TokenUsage and FilePath tables to SessionLink, CheckpointMetadata, nested-field tables, and explicit relationships.Later feedback adds one-to-many links, foreign keys, join tables, and separate tables for nested JSON fields.
- Handoff: The trajectory ends with user approval after the requested surfaces and relationships are revised.The examples include approval of a plan and a final “Looks good, ship it” response.
- Progressive corrections: The simulated trajectory records repeated implementation commits followed by targeted corrections rather than a single complete handoff.Examples include initial implementations, configuration feedback, SNMP-counter feedback, and stream-open API feedback, followed by commits addressing each stage.
- Trajectory setup: The comparison includes a real SWE-chat session, an expert-nitpicker simulated trajectory, and a neutral-simulator trajectory for the same task.Long user specifications are represented by placeholders, while user turns remain otherwise written as in the sessions.
A.3 Goal-Progress Rubrics and Scoring
The goal-progress analysis converts complete task instructions into atomic weighted rubrics and scores both planning and cumulative implementation coverage. Binary rubric items provide a lifecycle view of how much task intent is identified and achieved over revisions.
- Rubric construction: Each complete task instruction is converted into an atomic progress rubric used for both plan coverage and cumulative implementation scoring.The plan is scored for intended coverage, while each implementation checkpoint is scored for achieved coverage.
- Scoring lifecycle: PLAN.md supplies plan coverage, while cumulative commit patches supply implementation progress.The judge scores each intermediate revision using the task goal and that revision's patch without seeing future revisions or final test results.
- Rubric construction: Rubrics contain 6–18 binary items, with positive integer weights summing to 100.Most tasks use 10–14 items, smaller bug fixes use 6–9, and broad feature tasks may use up to 18.
- Scoring design: Each rubric item has one requirement, one concrete achieved_when condition, and one not_achieved_when condition.Binary scoring assigns either zero or the item's full weight, with no partial credit inside an item.
- Scoring design: Weights prioritize central user-visible behavior while also covering explicit edge cases, errors, API exports, compatibility constraints, and interfaces.Plans, explanations, repository exploration, status updates, and unrequested tests do not receive rubric credit.
A.4 Plan Coverage Example
The plan-coverage example shows that models given the same brief task description can differ sharply in how much hidden task intent they identify. GPT 5.5’s plan covers nearly the full rubric, whereas Gemini 3.5 Flash focuses on identifier lookup and misses downstream requirements.
- Coverage comparison: GPT 5.5’s plan covers 96/100 rubric points, while Gemini 3.5 Flash’s plan covers 19/100 for the same author-import task.Both agents received a brief high-level description and had to discover remaining details by exploring the repository.
- Task goal: The task requires matching imported authors by Open Library key first, remote identifiers second, and name/date logic last.The requested flow accepts external identifiers, merges new identifiers on matches, and reports conflicts without partial saves.
- GPT 5.5 plan: The stronger plan preserves existing fields, merges and persists new remote IDs, retains identifiers on new authors, and handles conflicts safely.Its implementation outline includes batch persistence and focused tests for duplicate prevention, priority, merging, and unsaved conflicting changes.
- Coverage comparison: Gemini 3.5 Flash’s plan is labeled 19/100 and is contrasted with a separate plan labeled 96/100 for the same task.The comparison identifies the lower-coverage plan as focused mainly on identifier lookup while missing merge and conflict-handling details.
- GPT 5.5 plan: The stronger plan covers identifier extraction, supported-database queries, redirect resolution, and authoritative matches that bypass imperfect date checks.It also specifies concrete implementation locations and tests for identifier matches, date mismatches, and missing dates.
A.5 User-Simulator Prompt
The user simulator models a casual developer who starts vague, inspects work, and reveals concrete requirements through focused corrections. Its interaction rules govern disclosure, tone, private inspection, and approval behavior.
- A.5 User-Simulator Prompt: The simulator avoids narrating withheld topics, hidden evaluation details, or role-playing instructions, and it communicates in a casual developer tone.It should not bundle unrelated questions and should keep corrections direct rather than using corporate or specification-heavy phrasing.
- A.5 User-Simulator Prompt: The simulator’s shared invariants separate task intent, disclosure timing, interaction style, and later behavioral constraints.The task block is the source of truth, while unspecified implementation details remain the agent’s call.
- A.5 User-Simulator Prompt: The simulator adopts a busy, lazy vibecoder persona that opens with a brief vague overview and reveals requirements after seeing implementation.Requirements are introduced one small change at a time as the user notices mismatches.
- A.5 User-Simulator Prompt: The first pass asks for a quick plan when needed, approves a reasonable incomplete plan, and avoids volunteering unspecified details.When the task does not specify an answer, the simulator tells the agent to use its judgment.
- A.5 User-Simulator Prompt: During implementation critique, the simulator requests one concrete correction at a time and waits for revised work before raising another issue.Examples include deleting obsolete types, threading context, correcting constructor names, and exposing an API surface.
- A.5 User-Simulator Prompt: Before approval, the simulator privately inspects the committed repository state and corrects externally visible surfaces with exact task-defined shapes.It asks for a diff only when inspection is unavailable or no coherent commit exists.
- A.5 User-Simulator Prompt: Approval follows iterative correction until the implementation matches the user’s intent, including lifecycle, wiring, preservation, and other concrete behavior.The examples show successive fixes for service wiring, event lifecycles, and idempotent state preservation before approval.
- A.5 User-Simulator Prompt: The code-evolution analysis uses churn overhead to measure extra edit volume beyond the final handoff diff, with lower values indicating less revision rework.It defines changed lines per revision as additions plus deletions and compares the cumulative volume with the final diff.