Source-linked AI summary
UserSWE: Evaluating Coding Agents in Multi-Turn Conversations with Dynamic Users
Zhan-Yu Wu, Ji Luo, Alyssa Wu, Junjielong Xu, Richeng Xuan, Zhichao Hu, Chongming Gao, Xinting Hu
TL;DR
Existing coding-agent benchmarks often omit interaction and face challenges in judging intermediate turns. UserSWE constructs multi-turn tasks, simulates dynamic users, and grounds turn-by-turn verdicts in executed tests. Agent rankings change with the turn budget, so single-turn performance alone does not characterize interactive coding ability.
Problem
Most coding-agent benchmarks present the issue in one turn, while multi-turn evaluation needs turn-level verdicts grounded in evidence that tests and unrestricted LLM judges alone may not provide.
Source: §1 · p. 2, §4.3 · p. 5Method
UserSWE builds multi-turn tasks from repository history and existing benchmarks, simulates users with information pools and persona transitions, and uses cross-family agents to write and review test-grounded verdicts.
Source: §1 · p. 2, §4.3 · p. 6Results
Agent rankings change with turn budget: Claude Fable 5.1 resolves the most tasks in one turn (31.9%), while GPT-6-Astra leads by turn eight (81.9%).
Source: §1 · p. 2Takeaways & Limitations
A single-turn evaluation observes only one point of the performance curve and cannot show how agent rankings change across turn budgets.
Source: §5.2.1 · p. 7UserSWE covers one maintenance episode over one or two issues, not longer-term work such as building a repository from scratch.
Source: §6 · p. 9
Abstract
from arXiv · showhide
As coding agents become more capable, developers increasingly use them to fix bugs through multi-turn interactions. Existing benchmarks do not adequately cover this scenario. Most present a complete issue report in a single turn without interaction. Their evaluation can produce both false positives and false negatives, accepting incorrect patches or rejecting correct ones. We introduce UserSWE for evaluating coding agents' ability to fix repository-level bugs through multi-turn interactions with dynamic users and execution-grounded verdicts at every turn. Its construction pipeline turns repository history and existing single-turn benchmarks into multi-turn tasks. At each turn, information pools constrain what the simulated user can reveal, while a persona chain sampled from a transition model calibrated on 20,859 development sessions governs how it speaks. Tests run after every turn, and a pair of agents from different model families writes and gates each verdict, using test results as primary evidence and model judgment only for aspects the tests cannot resolve. We evaluate eleven agents on 72 tasks from 42 repositories and find that agent rankings change with the interaction budget. Claude Fable 5.1 resolves the most tasks in one turn, GPT-6-Astra resolves the most by the eighth turn, and Claude Opus 5 leads at most intermediate budgets. These results indicate that single-turn performance is insufficient to characterize interactive coding ability.
1 Introduction
UserSWE addresses benchmarks that miss the conversational nature of debugging and the need to judge each turn from evidence. It combines multi-turn task construction, dynamic user simulation, and evaluation, revealing agent behaviors that single-turn results cannot show.
1 Introduction
- The benchmark evaluates coding agents in conversations where developers’ requests may begin incomplete and become clearer across turns.Figure 1 contrasts a fixed, precise simulator with one that begins vaguely, guesses a wrong cause, and reaches guidance at the third turn.Source: Figure 1 · p. 1, §1 · p. 2
- Most benchmarks provide a complete issue in one turn, while existing multi-turn benchmarks use fixed information release or a fixed conversational style.Turn-by-turn evaluation also needs to balance factual execution evidence with the risk that unrestricted LLM judges infer beyond test results.Source: §1 · p. 2
- UserSWE builds multi-turn tasks from repository history or existing benchmarks, simulates users with persona transitions drawn from 20,859 development sessions, and uses cross-family judging and review.Each task’s persona chain is drawn in advance so agents on the same task encounter the same user.Source: §1 · p. 2
- Claude Fable 5.1 resolves the most tasks in one turn (31.9%), while GPT-6-Astra leads by the eighth turn (81.9%).Claude Fable 5.1 finishes second on length-discounted pass rate, behind Claude Opus 5.Source: §1 · p. 2
- UserSWE’s contributions comprise a task-construction pipeline, a dynamic-persona user simulator, and a dual-agent judging framework with supporting mechanisms such as equivalent-pass testing.The authors state that UserSWE will be released with the evaluation framework.Source: §1 · p. 2
2 Related work
Related work covers multi-turn benchmarks and simulated users, alongside benchmark evaluation reliability. UserSWE combines dynamic persona sampling with turn-by-turn execution and cross-family review.
2 Related work
- Prior multi-turn benchmarks either release requirements incrementally with a fixed persona or reconstruct a task-specific simulator from a recorded session.UserSWE instead samples a persona chain from real development sessions and creates an information pool for each task.Source: §2 · p. 2
- Coding-agent benchmark reliability work addresses test strictness, LLM-judge bias, and evaluation-harness gaming, while panel judges and rule-based checkers remain imperfect.SWE-Interact verifies only the final solution and rubric-scores intermediate turns; UserSWE runs tests after every turn and uses cross-family review.Source: §2 · p. 3
- Table 1 compares UserSWE with recent coding-agent benchmarks; the supplied passage does not report the table’s individual comparisons.The related-work discussion distinguishes benchmarks by whether personas stay fixed and how much LLMs influence verdicts.Source: Table 1 · p. 3, §2 · p. 3
3 Preliminaries
The preliminaries describe the standard issue-level setup: an agent patches a repository state, and tests determine whether the issue is resolved without regressions.
3 Preliminaries
- A task begins from repository state b, with a gold patch g and test sets defined against the base and patched states.The fail-to-pass tests fail at the base and pass after the gold patch; pass-to-pass tests pass in both states.Source: §3 · p. 3
- A patch resolves a task when every fail-to-pass and pass-to-pass test passes after the patch.The reported score is the fraction of tasks resolved.Source: §3 · p. 3
4 UserSWE
UserSWE evaluates issue fixing through multi-turn conversations, checking the agent’s work after each turn. Its pipeline constructs tasks, controls what simulated users reveal and how they speak, and evaluates each turn.
4.1 Data Construction
- Tasks come from linked repository issues or existing single-turn benchmarks, which are staged as conversations.Information pools are frozen and checked for scope; tasks also pass executable-test checks before admission.Source: §4.1 · pp. 3–4
4.2 Multi-turn Interaction
- At each turn, direct questions take priority; otherwise, guidance addresses blocking failures, core business boundaries, then secondary edge cases.The release policy selects content from the information pool based on the agent’s response and canonical test results.Source: §4.2 · p. 4
- The user’s phrasing varies across symptom clarity and cause veracity, forming eight personas whose transitions are sampled from real development sessions.The persona chain is drawn in advance so agents on the same task face the same changes in user style.Source: §4.2 · pp. 4–5
4.3 Evaluation
- The opening request, persona chain, and permissible facts are fixed across agents, while later content selection and phrasing remain interactive and every verdict passes a different-family reviewer gate.The Judge and Reviewer are swapped between tasks.Source: §4.3 · p. 6
- Tests run after every turn, and model judgment contributes where tests are silent rather than overriding their results.The framework can count a genuinely modified test as an equivalent pass to avoid rejecting behaviorally correct patches solely for differing from the shipped test interface.Source: §4.3 · pp. 5–6
- Length-Discounted Pass Rate (LDPR) rewards resolving tasks in fewer weighted turns and assigns zero to failures within the turn limit.A turn without a code edit counts as half a turn.Source: §4.3 · p. 6
5 Experiments
Across the evaluation budgets, the leading agents change rank, and dynamic personas and review mechanisms affect the measured outcomes. The experiments also examine how reviewer rewrites and execution-based equivalent-pass handling shape verdicts.
5.1 Main results
- The 72 tasks comprise 22 repository-history tasks and 50 converted single-turn tasks, evaluated with 11 agents and shared task-specific information pools and persona trajectories.The two task subsets use different turn budgets: K = 5 for single-issue tasks and K = 8 for two-issue tasks.Source: §5.1 · p. 6
5.2.1 Multi-turn against single-turn
- Claude Fable 5.1 leads in first-turn resolutions but trails Claude Opus 5 on LDPR; GPT-6-Astra leads by turn eight yet ranks fourth on LDPR.The four leading agents take six different orderings across eight budgets, despite resolving between 76.4% and 81.9% by turn eight.Source: §5.2.1 · p. 7
- Figure 3 tracks the order of the four leading agents on published single-turn tasks and how it changes as the UserSWE turn budget grows.On 42 separate single-turn tasks, Claude Fable 5.1 ranks first at 78.6%, ahead of Claude Opus 5 at 64.3%, GPT-6-Astra at 61.9%, and Grok-4.6 at 59.5%.Source: Figure 3 · p. 7, §5.2.1 · p. 7
5.2.2 Dynamic conversational persona
- On the same 50 single-issue tasks, three of four agents score higher and finish sooner with a clear-and-correct user than with the dynamic persona chain; Kimi-K3 moves the other way.GPT-6-Astra’s LDPR rises from 0.580 to 0.656 while mean weighted turns k falls from 2.92 to 2.54 under the clear-and-correct user; across all 72 tasks, sensitivity to opening phrasing varies by agent.Source: §5.2.2 · p. 8, Table 4 · p. 8
5.3 Analysis of the evaluation framework
- The framework analysis reports rates from canonical runs of all eleven leaderboard models across 22 cascade tasks and 50 single-issue tasks using the same judging, review, and equivalence protocol.The section examines reviewer rewrites and equivalent-pass requests as parts of the evaluation framework.Source: §5.3 · p. 8, §5.3.1 · p. 8, §5.3.2 · p. 9
5.3.1 Reviewer rewrites
- Across canonical runs, the Reviewer rewrote 40.9% of Judge decisions, and 87.4% of rewrites were rated major or critical.Evidence made up 70.3% of Codex Reviewer objections; the Claude Code Reviewer’s objections split nearly evenly between evidence at 42.1% and guidance at 41.5%.Source: §5.3.1 · p. 8, Table 5 · p. 8
5.3.2 Equivalent passes
- The equivalent-pass mechanism is execution-based and permits targeted test rewrites only when they pass on the candidate and still fail at baseline.Across runs, 14.6% of cells filed an equivalence request; 51.2% of requests were granted and added to the scorecard.Source: §5.3.2 · p. 9, Figure 4 · p. 9
6 Limitations
UserSWE’s scope and data impose three stated limitations: evaluation cost, episode horizon, and the source of persona calibration data.
6 Limitations
- Running full sessions and rerunning tests after every turn makes multi-turn evaluation more expensive and limits the released set to 72 tasks.The paper reports detailed per-task costs in Table 15.Source: §6 · p. 9
- UserSWE covers one maintenance episode over one or two issues, not longer-term work such as building a repository from scratch.The paper identifies longer-term work as a direction for future extensions.Source: §6 · p. 9
- Persona transitions are calibrated from 20,859 sessions collected from an internally deployed commercial coding assistant; future versions could add task-type and request-phrasing signals.Source: §6 · p. 9
7 Conclusion
The paper presents UserSWE as a benchmark for coding agents in multi-turn conversations with dynamic users. It evaluates eleven agents on 72 tasks and reports that the benchmark reveals behaviors single-turn evaluation cannot show.
7 Conclusion
- UserSWE combines a task-construction pipeline, a dynamic-persona user simulator grounded in development sessions, and an evaluation framework using cross-family judging and review anchored to executed tests.Source: §7 · p. 9
- The benchmark is used to evaluate eleven agents on 72 tasks, uncovering behaviors that single-turn evaluation cannot reveal.Source: §7 · p. 9
Ethics statement
- The user model is calibrated from recorded sessions collected from an internally deployed commercial coding assistant; raw conversations and person-level attributes are not released.Only aggregate statistics needed to instantiate the simulator are released, reducing privacy and security risks associated with development logs.Source: Ethics statement · p. 9
Reproducibility statement
- The sampled persona trajectories are independent of agents and stored for turn-by-turn replay; the authors state that UserSWE and its evaluation framework will be released.Source: Reproducibility statement · p. 10
A How a task is built
The pipeline converts repository history and published single-turn tasks into multi-turn packages, then pairs information pools with a calibrated persona simulator. Admission checks and human review support task grading and plausible user turns.
A.1 Two source routes
- The pipeline builds tasks by mining linked repository issues or converting published single-turn tasks into conversations.The released set includes 22 mined cascades and 50 converted single-issue tasks.Source: A How a task is built · p. 11, §A.1 · p. 11
- Mined cascades retain ordered issue pairs only when both have merged fixes and the later record points to earlier work; six use reveal signals and sixteen use dependency signals.Source: §A.1 · p. 11
A.2 From source to task package
- For mined cascades, the pipeline packages fixes, grading images, tests, and phase-specific acceptance points; converted tasks retain their upstream repository state, image, and test verdicts.Source: §A.2 · p. 11
A.3 Information pools
- Each task has an information pool from which the simulator selects content before applying persona wording; admission checks verify base, gold, and intermediate test states.All 72 tasks also received read-only solvability review, with a human curator making final revisions and retention decisions.Source: §A.3 · p. 12, §A.4 · p. 12
B.1 Persona design
- Personas combine symptom clarity with a cause setting, and the cause axis determines what direction the user volunteers and how they react when challenged.Off-target and wrong causes share an opening turn and become distinguishable only through later interaction.Source: §B.1 · pp. 12–13, Table 7 · p. 13
B.2 Log labeling and transition chains
- The simulator’s persona transitions are calibrated from 13,530 code-changing sessions selected from 20,859 labeled sessions; the eight persona cells cover 98.65% of code-changing openings.It samples an opening and then a transition chain conditioned on that opening, using one matrix per opening persona.Source: §B.2 · p. 13, Table 8 · p. 13
B.3 Utterance policy
- The simulator speaks as a software user, while release selection uses test results to identify what remains open and limits messages to information in the pool.The sampled trajectory is fixed, but interaction rules can weaken a played cause after specified contradictions or when the pool lacks matching direction.Source: §B.3 · pp. 13–14
B.4 Human validation
- Human reviewers checked generated turns for plausibility, persona consistency, and information-pool and per-turn-budget compliance; every released task passed at least one such review.The 50 human-hours of review prompted changes to generation rules, prompts, or pools rather than sentence-by-sentence edits.Source: §B.4 · p. 15
C Evaluation framework
The framework controls what agents can access and how each turn is judged, then audits scores and standardizes experimental runs.
C Evaluation framework
- The framework separates environment controls, judging, reviewing, and a post-run audit.The audit flags conditions for review; a score is untrustworthy only if flagged behavior changed the final verdict outside a valid equivalence override.Source: C Evaluation framework · p. 15, §C.4 · p. 16, Table 12 · p. 17
C.1 Environment controls
- Agents see the task statement and workspace, while evaluation metadata is mounted separately and blocked from direct or indirect access.General documentation is allowed, but retrieving and using protected task-specific material is terminal.Source: §C.1 · p. 15
C.2 Judge
- The Judge derives item-level verdicts from canonical tests after handling policy and environment failures, with distinct states for unresolved, completed, and ongoing work.An equivalence state requires a candidate test rewrite, and evidence is kept separate from the user-facing message.Source: §C.2 · p. 15, Table 10 · p. 15
C.3 Reviewer and settlement
- The Reviewer checks verdicts against framework evidence, messages against persona and information constraints, and equivalence requests against their formal contract.A Reviewer failure closes the path instead of dispatching an unreviewed decision.Source: §C.3 · p. 16, Table 11 · p. 16
C.4 Audit rubric
- The rubric is updated after each evaluation batch, requiring about one human-day, and trustworthiness is assessed separately from rubric membership.An audit agent applies the rubric’s 13 items.Source: §C.4 · p. 16
D.1 Experimental setup
- Each task receives a seeded persona trajectory frozen before evaluation, shared across agents, and sessions stop once acceptance points are met.Single-issue tasks allow at most five turns and two-issue tasks at most eight.Source: §D.1 · p. 17, Table 13 · p. 17
- Judge and Reviewer roles are balanced between two model families, with the Reviewer seeing the Judge’s candidate and framework evidence rather than rerunning the agent.The roles reverse across assignments.Source: §D.1 · p. 17
D.2 Confidence intervals and paired comparisons
The appendix explains paired comparisons and LDPR, then details cost reporting, Reviewer rewrites, equivalence gates, and case-study coverage.
D.2 Confidence intervals and paired comparisons
- Table 14 reports each agent’s LDPR with a 95% interval from resampling tasks with replacement.All eleven agents run the same 72 tasks, pairing comparisons and removing task-difficulty spread from agent differences.Source: §D.2 · pp. 17–18, Table 14 · p. 18
- LDPR separates 17 agent pairs, versus 5 for first-turn resolution and 5 for resolution at any point, despite the latter’s 71.8% base rate.The length discount carries the additional signal; comparisons use paired permutation tests on 72 per-task differences with Benjamini-Hochberg correction across 55 pairs at a 5% false discovery rate.Source: §D.2 · p. 18
D.4 Reviewer rewrites by model
- Reviewer rewrite rates are calculated over reviewed turn decisions for each model–Reviewer configuration, while each row’s objection shares sum to 100%.Finalize confirmations and other audit checkpoints are excluded.Source: §D.4 · p. 18, Table 16 · p. 19
D.3 Cost
- Cost reporting separates agent costs from judging and uses available telemetry, with agent columns as means and judging costs and edit sizes as medians.Token totals are in thousands, exclude cache reads, and costs are in CNY.Source: §D.3 · p. 18, Table 15 · p. 19
D.5 Equivalence gates
- A refused equivalence request fails first at one of three gates: candidate pass without regression, base-state failure, or retained issue-required behavior.Table 17 assigns 62 refusals to the first gate each fails.Source: §D.5 · p. 20, Table 17 · p. 20
E Case studies
- The case studies cover a mined cascade, a full session, equivalence decisions, and a Reviewer rewrite.The cases trace examples from source issues through graded interactions and review.Source: E Case studies · p. 20
E.1 From two issues to a cascade task
The examples show how task construction, turn-by-turn information release, equivalence review, and Reviewer intervention shape multi-turn evaluation.
E.1 From two issues to a cascade task
- In the mined cascade, agents initially see only the first issue’s title and a request to locate and fix its cause, without test names or the second issue.The task has two acceptance points per phase, three regression tests, and a frozen pool of clear or vague symptoms and correct or incorrect candidate causes.Source: §E.1 · p. 20, Table 18 · p. 20
- The second issue is withheld until both first-phase acceptance points pass, so this cascade cannot be completed in one turn.The two issues were paired through an explicit maintainer link, and the second phase remains failing when the first fix alone is applied.Source: §E.1 · p. 20
E.2 A full session, end to end
- The Judge and cross-family Reviewer accepted the third-turn fix after the session surfaced the repository’s existing argument name.The first two implementations addressed only what had been disclosed so far.Source: §E.2 · p. 21, Table 19 · p. 21
E.3 An equivalent pass, granted and refused
- Two equivalence requests passed mechanical gates, but the Reviewer refused one because its rewritten test did not preserve the behavior being graded.The comparison places a grant beside a refusal and shows that candidate-test success alone is insufficient.Source: §E.3 · p. 21, Table 20 · p. 21
E.4 A reviewer rewrite
- On turn one, the Reviewer kept the Judge’s verdict but rewrote its supporting evidence and the simulated user’s next message.The verdict stayed the same, but the record and utterance used to build the next turn changed.Source: §E.4 · p. 21
F Opening clarity across the leaderboard
Opening-symptom clarity is associated with different LDPR patterns across agents, but the split is observational and groups contain different tasks.
F Opening clarity across the leaderboard
- 0.268 for Claude Fable 5.1 and −0.088 for DeepSeek-v4.1-Flash mark the extremes of the across-agent clarity-gap range, a spread of 0.356.The groups are 55 tasks with precise opening symptoms and 17 phrased as impressions; the gap is their LDPR difference.Source: F Opening clarity across the leaderboard · p. 22, Table 22 · p. 22
- Because the clarity groups contain different tasks, their LDPR levels mix phrasing with task difficulty, although comparing the gap across agents controls for task difficulty.The comparison uses the same tasks for every agent.Source: F Opening clarity across the leaderboard · p. 22