SWE-DIVE: Multi-Turn Coding Agent Evaluation with Dynamic User Simulation

• Real users often start with high-level, underspecified prompts and iteratively refine the problem, unlike the complete, detailed issues commonly used to evaluate agents.Source: §1 · p. 1
• Prior multi-turn benchmarks use predefined task sequences and do not adapt to agent questions, despite observed pushbacks indicating that clarification is often needed.Baumann et al. (2026) found that 41% of turns were pushbacks.Source: §1 · p. 2
• SWE-DIVE transforms existing software-engineering benchmarks into dynamic, interactive evaluations by splitting tasks into incremental subtasks and simulating the user’s discovery process.Source: §1 · p. 2

• Conventional benchmarks such as SWE-bench provide a single-shot issue prompt and use whether its associated tests pass to measure task success.Source: §2 · p. 2
• CodeFlowBench and CodeGen add iterative tasks, but their sequences remain predefined rather than reflecting dynamically evolving requirements.Source: §2 · p. 2
• Other approaches add outcome-based feedback or ambiguous, persona-based requests, while retaining structured tasks or focusing on clarification rather than ongoing new requirements.Source: §2 · p. 2
• SWE-DIVE addresses the gap of users introducing new tasks and requirements during development with a benchmark-agnostic approach to dynamic, multi-turn evaluation.Source: §2 · p. 2

3 SWE-DIVE

• SWE-DIVE transforms existing benchmark tasks into multi-turn interactive evaluations using a user simulator, conversation validator, and Task Completion Evaluator.Source: §3 · p. 3

3.1 User Simulator

• The LLM-based user simulator grounds interactions in validated task descriptions and iteratively discovers, clarifies, and refines requirements in response to agent progress.It adapts its next prompt using the agent’s latest response and code changes while maintaining alignment with the original objective.Source: §3.1 · pp. 3–4

3.2 Conversation Validator

• The conversation validator checks that subtasks and prompts cover the task, follow the assigned role and trajectory, and remain understandable and actionable.Invalid outputs receive feedback and are regenerated until approved.Source: §3.2 · p. 4

3.3 Task Completion Evaluator

• The Task Completion Evaluator runs the original benchmark environment to verify task success and reports success rates, conversation turns, and role-based performance.Evaluation occurs when the turn limit is reached or the simulator declares the conversation complete.Source: §3.3 · pp. 4–5

4 Proposed Evaluation Plan

• The plan hypothesizes persona-dependent waste and regression differences, defining waste as extra tokens and edits to files untouched in the golden run, and regression as previously passing tests failing.The proposed comparison also considers whether multi-turn task decomposition changes performance relative to the original benchmark.Source: §4 · p. 5
• Planned comparisons cover personas and lightweight models on SWE-bench Verified and Terminal-Bench against original benchmark results; direct integration requires a Git-diff progress signal.If a benchmark does not provide that signal, the Git-diff approach would need replacement.Source: §4 · p. 5

4.1 Qualitative Trajectory Analysis

• In the case study, Expert Nitpicker and Mind Changer resolved the issue, but Vague Requester’s rewrite caused 10 previously passing tests to fail.Expert Nitpicker resolved the issue within four turns; Mind Changer required adapting to a changed implementation strategy.Source: §4.1 · p. 6
• These observations come from one task and a small number of runs, so whether persona effects generalize across tasks and agents remains undetermined.The study presents the case as evidence of distinct behavior, not a general result.Source: §4.1 · p. 6

4 Proposed Evaluation Plan

• The pilot combined a qualitative task with 10 additional tasks per persona, evaluating all three personas and a single-turn baseline across 11 tasks.The initial plan describes the quantitative run as 10 additional tasks per persona.Source: §4 · p. 5, §4.2 · p. 6

4.2 Quantitative Results

• Targeted tests passed in two Expert Nitpicker and Mind Changer runs and one Vague Requester run, but additional changes also caused regressions.All runs had regressions in previously passing tests.Source: §4.2 · pp. 6–7
• Across 11 tasks, Mind Changer had regressions in 4 tasks versus 5 for Expert Nitpicker and Vague Requester, while all three personas had similar success rates.Table 1 summarizes the role results alongside the original-task baseline.Source: §4.2 · pp. 6–7, Table 1 · p. 7
• All personas used more tokens than baseline; Mind Changer and Vague Requester read and wrote more files, while Expert Nitpicker used fewer files.Vague Requester had the lowest pass rate and most unrelated file reads despite using the fewest turns.Source: §4.2 · p. 7

5 Discussion

• Different user personas can surface failure modes and robustness properties missed by single-turn or single-style evaluation.The discussion further reports distinct outcomes on the same Astropy task: the Vague Requester induced broader changes that introduced regressions, while the Expert Nitpicker guided a successful resolution through targeted corrections.Source: §5 · pp. 7–8
• The three roles—Expert Nitpicker, Mind Changer, and Vague Requester—vary how users communicate requirements, give feedback, and evolve goals.The roles are adapted from real-world user roles and provide controlled behavioral variation grounded in observed interactions; simulation fidelity depends on how closely behaviors reflect real interactions.Source: §5 · p. 7
• The simulator conditions each turn on the agent’s latest response and code changes, enabling corrections, follow-up requirements, or request modifications as interaction unfolds.Its interactions are grounded in validated benchmark tasks and underlying user behaviors.Source: §5 · pp. 7–8
• Average additional costs have high variance, with costs concentrated in particular tasks rather than occurring uniformly across instances.Source: §5 · p. 7
• A direct comparison between SWE-DIVE trajectories and held-out human SWE-Chat trajectories remains future work.Source: §5 · p. 8
• On the same Astropy task, the Vague Requester induced broader changes with regressions, while the Expert Nitpicker guided a successful resolution through targeted corrections.These trajectories suggest user behavior can influence observed agent performance and evaluation conclusions.Source: §5 · p. 8

6 Conclusion

• Across 11 tasks, the preliminary evaluation found that unconstrained or changing user requirements lead to higher code regression rates.Source: §6 · p. 8
• SWE-DIVE extends coding-agent benchmarks to a multi-turn setting with different user personas.Source: §6 · p. 8
• The benchmark uses a role-specific user simulator to communicate with the coding agent and guide it toward completing tasks.Source: §6 · p. 8

A Qualitative Agent Conversations

• Figure 3 presents a representative multi-turn trajectory with a fixed Vague Requester persona.Source: Figure 3 · p. 10
• Figure 4 presents a representative multi-turn trajectory with a fixed Expert Nitpicker persona.Source: Figure 4 · p. 11
• Figure 5 presents a representative multi-turn trajectory with a fixed Mind Changer persona.Source: Figure 5 · p. 12

B Detailed Results

• Table 2 reports 11 Expert Nitpicker runs, with resolution, fail-to-pass and pass-to-fail tests, turns, and task identified as fields.Source: Table 2 · p. 13
• Table 3 reports 11 Mind Changer runs using the same identified fields: resolution, test transitions, turns, and task.Source: Table 3 · p. 13
• Table 4 reports 11 Vague Requester runs, identifying resolution, test transitions, turns, and task.Source: Table 4 · p. 14
• Table 5 reports 11 base-task runs without multi-turn behavior, identifying resolution, test transitions, turns, and task.Source: Table 5 · p. 14