Source-linked AI summary
SWE-Together: Evaluating Coding Agents in Interactive User Sessions
Yifan Wu, Zhuokai Zhao, Songlin Li, Ho Hin Lee, Jiacheng Zhu, Shirley Wu, Tianhe Yu, Serena Li, Lizhu Zhang, Xiangjun Fan, Shengzhi Li
TL;DR
Static coding-agent benchmarks miss the multi-turn, evolving instructions and user guidance of real coding assistance. SWE-Together reconstructs real sessions into a 109-task benchmark, and frontier-agent results show stronger models achieve higher correctness with fewer corrective interventions.
Problem
Most coding-agent benchmarks use fixed, single-turn tasks and final-code evaluation, omitting evolving instructions and the amount of user guidance required during interaction.
Method
SWE-Together reconstructs 109 real user-agent sessions into verifiable tasks and uses a state-conditional LLM user simulator with joint correctness and interaction evaluation.
Results
Claude Opus 4.8 leads with 63% pass@1, 59% SSR, 52% pass2, and 0.801 mean judge score, while requiring 1.38 corrective messages; User Correction correlates −0.92 with pass@1.
Takeaways & Limitations
Evaluating coding agents as collaborators reveals capability differences beyond final correctness, with stronger agents generally requiring less user intervention.
Takeaways & Limitations
The simulator cannot interrupt agents, edit files, or use visual interface information, limiting coverage of ambiguous, open-ended tasks and difficult-to-quantify user behaviors.
Abstract
from arXiv · showhide
Most coding-agent benchmarks are static: an agent receives a complete task description up front and is judged only by its final code. Real coding assistance is interactive, with users clarifying goals, adding constraints, and correcting mistakes over multiple turns. We introduce SWE-Together, a multi-turn benchmark reconstructed from real user-agent coding sessions. To make real interactions verifiable, we curate 109 repository-level tasks from 11,260 recorded sessions, selecting sessions with recoverable repository states, clear user goals, and observable outcomes. To replay these interactions across agents, we build a reactive LLM-based user simulator that preserves the original users' intents and provides feedback when the coding agent's progress requires it. To evaluate agents as collaborators, we measure both final repository correctness and the number of corrective feedback turns required during the interaction. Experiments with frontier coding agents show that stronger agents generally achieve higher final success rates while requiring fewer interventions, suggesting an improved user experience.
1 Introduction
SWE-Together addresses a central gap in coding-agent evaluation: static, single-turn benchmarks poorly represent interactive software work. It reconstructs real sessions into verifiable tasks and evaluates both repository outcomes and the guidance agents require.
- The benchmark gap: Real coding assistance unfolds across turns, as users clarify intent, refine requirements, and correct agent outputs.Static benchmarks usually provide the complete task description at the start and judge only submitted code.
- The benchmark gap: Raw conversation logs are difficult to benchmark because they may lack reproducible repositories, identifiable goals, or observable outcomes.Later user turns must also adapt to each evaluated agent’s trajectory while remaining anchored to the original intent.
- SWE-Together: SWE-Together reconstructs selected real sessions as sandboxed tasks with recoverable repository states, clear intents, and concrete outcomes.Each task includes an initial instruction plus decomposed intents and trigger conditions for feedback turns.
- SWE-Together: An anchored, state-conditional user simulator releases feedback when triggering conditions arise, preserving intervention order while adapting timing to each agent.This design helps attribute outcome differences to agents rather than simulator variation.
- Evaluation: The benchmark jointly scores final repository correctness and interaction behavior through User Correction and Intent Coverage.These measures assess both whether the coding request was satisfied and how much steering the agent elicited.
2 SWE-Together
SWE-Together turns recorded coding sessions into reproducible, repository-level interactions through staged filtering, sandboxed construction, and trajectory-conditioned simulation. Its evaluation combines deterministic correctness scoring with diagnostics of user guidance and simulator fidelity.
- Session-to-task construction: 109/11,260 sessions pass the filters, yielding a 0.97% conversion rate into executable tasks.The retained subset favors public, mature repositories, multi-turn interaction, concrete code changes, recoverable edits, and locally evaluable outcomes.
- Session-to-task construction: The construction pipeline uses deterministic eligibility filtering, LLM-based viability screening, and sandbox orchestration to produce complete task packages.The three stages progressively normalize candidates, test reproducibility, and generate executable benchmark directories.
- Session-to-task construction: Each task package contains the original session, initial instruction, pinned environment, deterministic verifiers, and a task-specific user-simulation prompt.The task-generation agent clones the repository at a pinned commit, identifies setup and test commands, writes artifacts, and audits the result.
- User simulator: The simulator consults the live trajectory after each agent turn and chooses between a user-facing intervention and no-op continuation.Its action space includes clarification questions, redirects, new requirements, and requests to inspect external artifacts.
- Evaluation method: Final correctness scores the repository state, while User Correction measures the feedback needed to reach that state.The evaluation separates task outcome from interaction behavior and applies multi-label identification to corrective interventions.
3 Experiments and Results
Across 109 tasks, the evaluation combines frozen-rubric correctness with interaction cost, showing that stronger agents solve more reliably while requiring less corrective steering. Results also examine efficiency, simulator fidelity, and whether simulated users resemble real users.
- Overall performance: Claude Opus 4.8 leads all four correctness metrics and requires the least corrective steering.It achieves pass@1 63%, SSR 59%, pass2 52%, mean judge score 0.801, and User Correction 1.38.
- Interaction cost: User Correction is strongly inversely correlated with pass@1, stable solve rate, and mean judge score.The Pearson correlations are −0.92, −0.84, and −0.93, respectively.
- Efficiency: GPT-5.5 is the most efficient cohort on both reported cost axes while ranking second on capability.It uses 29.9k output tokens and 10.7 minutes per task.
- Reference comparison: The reference baseline reaches mean judge score 0.90 and pass rate ≈78% on the 93 tasks with extractable reference patches.Reference scores below 100% partly reflect process requirements, extraction noise, and imperfect human solutions.
- Simulator fidelity: Intent Coverage remains broadly stable across cohorts, supporting comparisons between agents.Six of seven cohorts score between 0.70 and 0.72 overall, while GPT-5.5 scores 0.68.
- User simulator quality: Human annotators could not reliably distinguish simulated trajectories from real-user trajectories.The simulator achieved a Turing pass rate of 46%, with a 95% confidence interval of [40.5, 51.6]%, including chance-level 50%.
4 Related Work
SWE-Together occupies a distinct position among coding-agent benchmarks by combining repository interaction, sequential user feedback, and provenance from real coding sessions. Prior work typically supplies only some of these ingredients or synthesizes the interaction loop.
- Benchmark positioning: SWE-Together combines repository-level agent-environment interaction, interactive user-correction replay, and provenance from recorded user-agent sessions.Agent-environment multi-turn concerns iterative tool use for a fixed request, whereas interactive replay allows feedback, corrections, clarifications, or new requirements.
- Benchmark positioning: Table 3 distinguishes agent-environment multi-turn from interactive replay when comparing benchmark capabilities.The table uses ✓, ✗, ▲, and ♦ to indicate yes, no, partial support, and mixed or heterogeneous coverage.
- Agent-environment multi-turn: SWE-bench and Terminal-Bench support extensive environment interaction but retain fixed user requests.These benchmarks ground tasks in real codebases or terminal environments while evaluating agents against a static task specification.
- Simulated-user benchmarks: Interactive coding benchmarks introduce simulated feedback, but their loops generally derive from static tasks, curated policies, or generated scenarios.Examples include TiCoder, MINT, ConvCodeWorld, CodeAssistBench, and related collaborative-refinement benchmarks.
- Real coding-session data: Real coding-session datasets capture authentic interactions but are primarily used for characterization, preference modeling, or assistant evaluation.SWE-chat, BigCodeArena, and CodeChat exemplify this data-centric line of work.
5 Limitations and Conclusion
SWE-Together evaluates coding agents as collaborators by measuring both repository correctness and the corrective steering required during interaction. Its strongest evidence is tempered by a simulator designed mainly for clearly defined, measurable tasks.
- Limitations: The simulator cannot interrupt agent turns, directly edit files, or use visual interface information.Its textual, trajectory-based design limits interaction fidelity in settings requiring real-time intervention or visual context.
- Limitations: SWE-Together provides limited coverage of ambiguous, open-ended tasks and qualitative user behaviors that are difficult to quantify.The current design works best when user goals and constraints are clearly defined and outcomes are measurable, such as submitted patches.
- Conclusion: Across 109 tasks and seven frontier models, stronger coding agents generally achieve higher capability while requiring less user intervention.User Correction is strongly negatively correlated with pass@1, stable solve rate, and mean judge score.
- Conclusion: User Correction is strongly negatively correlated with performance, making corrective steering a user-centered diagnostic beyond final task success.The benchmark therefore evaluates not only whether an agent completes a task, but also how much guidance it elicits.
- Conclusion: The simulator maintains broadly consistent intent coverage across models and produces trajectories that human annotators cannot reliably distinguish from real-user interactions.These findings support using reconstructed sessions to study interaction quality rather than final code alone.