Saving SWE-Bench: A Benchmark Mutation Approach for Realistic Agent Evaluation

1 INTRODUCTION

• The mutation methodology uses telemetry-derived communication patterns to transform formal problem descriptions into realistic user queries while preserving their technical content.Source: §1 · p. 1
• The authors identify distinct templates that characterize real-world bug-fixing interactions with chat-based agents.Source: §1 · p. 1
• The authors create mutated versions of SWE-Bench Verified, Multi-SWE-Bench TypeScript, and SWE-Bench C# to test the methodology.Source: §1 · p. 1
• The paper evaluates OpenHands on original and mutated benchmarks, reporting performance gaps under realistic evaluation scenarios.Source: §1 · p. 2
• The authors incorporate the implementation into the official SWE-Bench repository and share prompts and mutations for SWE-Bench Verified.Source: §1 · p. 2
• Traditional benchmarks overestimate agent capabilities by 20–50% on public datasets and 10–16% on internal benchmarks like SWE-Bench C#.The authors attribute the public-benchmark gap to both over-specification and overfitting.Source: §1 · p. 2

2 BACKGROUND AND RELATED WORK

• Existing benchmarks use formal GitHub issues that differ from how developers communicate with chat-based agents in practice.Source: §2 · p. 2

2.1 Software Engineering Benchmarks

• SWE-Bench Verified provides 500 human-verified repository-level tasks, while related benchmarks expand task quality, language, or repository coverage [1, 13, 15, 22, 24, 25].The authors identify formal GitHub issue descriptions as a limitation shared by these benchmarks [25].Source: §2.1 · p. 2

2.2 Software Engineering Agents

• Agent research spans interfaces and platforms such as SWE-Agent [21] and OpenHands [19], while this work tests whether reported performance transfers to realistic user queries.The authors argue agents need to handle ambiguity and informality in real interactions.Source: §2.2 · p. 2

2.3 Developer Behaviour & Chat-based Assistants

• Prior studies examine developer use of assistants, including a finding that developers prefer short, task-focused chat interactions [10]; this paper focuses specifically on how they communicate bugs.Source: §2.3 · pp. 2–3

2.4 Data Contamination & Mutation-Based Benchmarks

• Dynamic benchmarks and contamination-mitigation methods address overfitting, while this mutation approach targets the separate gap between GitHub issue wording and chat queries [4] [8] [25] [26] [27] [28].The authors describe mutation as complementary to those approaches.Source: §2.4 · p. 3

3.1 Telemetry Analysis and Pattern Extraction

• The authors use internal telemetry from a widely used chat-based agent to identify interaction patterns as the foundation for benchmark mutation.The analysis focuses on real-world bug-fixing interactions.Source: §3.1 · p. 3

3.1.1 Data Collection and Labeling.

• A sample of 10,000 queries from about 6,000 users was categorized through LLM labeling with human validation; bug fixing and error resolution accounted for about 14%.The labeling process first generated high-level categories from about 500 queries, then assigned new queries to those categories.Source: §3.1.1 · p. 3

3.1.2 Understanding Bug-Fix User Queries.

• Telemetry bug-fix queries are typically 10–30 words, while benchmark problem statements such as SWE-Bench typically exceed 100 words.Users also favor targeted error stacks and file paths, whereas GitHub issues more often include reproduction code, expected behavior, and environment details.Source: §3.1.2 · p. 4
• LLM-assisted analysis of bug-fixing queries identified 11 communication templates that characterize how developers report bugs to chat-based agents.The templates span concise direct-fix requests, pasted errors with explanation requests, and other communication patterns.Source: §3.1.2 · p. 4

3.2 Mutation Methodology

• The pipeline transforms formal benchmark problem descriptions into realistic initial user queries while preserving their essential technical content.The authors describe the approach as extensible to almost any bug-fixing benchmark.Source: §3.2 · p. 5

3.2.1 Mutation Algorithm.

• The mutation pipeline uses benchmark descriptions, fixing patches, and telemetry-derived templates to generate multiple variants, then randomly selects one per problem.Patch information supports realistic references to files, functions, and line numbers.Source: §3.2.1 · p. 5

3.2.2 Mutation Example From SWE-Bench.

• For an Astropy SWE-Bench issue, one mutation discards the issue form’s broader details and uses only its call stack, following the “Paste Error/Stack Trace Only” pattern.Source: §3.2.2 · p. 5

3.2.3 Visualizing Mutated Queries Against Telemetry.

• Mutated SWE-Bench C# queries overlap more with telemetry embeddings than the original benchmark, whose cloud remains separate with little overlap.The visualization uses text-embedding-3-large embeddings reduced with PCA.Source: §3.2.3 · pp. 5–7

4.1 Benchmarks

• The study mutates SWE-Bench Verified (500 Python problems), internal SWE-Bench C# (150 problems), and the TypeScript subset of Multi-SWE-Bench (224 examples).The mutated datasets are named SWE-Bench Verified-Mutated, SWE-Bench C#-Mutated, and Multi-SWE-Bench (TypeScript)-Mutated.Source: §4.1 · p. 7
• For each problem, the authors generate multiple mutations with the Figure 11 prompt and randomly select one; an Astropy example retains only the original issue’s call stack.The prompt supplies the problem statement, code patch, and templates generated by the templatization algorithm.Source: §4.1 · p. 7, Figure 10 · p. 7, Figure 11 · p. 8

4.2 Agent & Model Configurations

• OpenHands is evaluated with GPT-4.1, Claude Sonnet 3.7, and Claude Sonnet 4, with at most 100 steps per task in isolated Docker containers.Success is assessed using the same harness as for the unmutated baseline.Source: §4.2 · pp. 7–8

4.3 Metrics

• The evaluation reports success rate, average token usage, and average steps, reflecting a trade-off between task completion and runtime or computational cost.Success requires a fix that passes hidden tests; token usage counts input and output tokens.Source: §4.3 · pp. 8–9

• The evaluation examines real-world performance of chat-based software engineering agents and how realistic problem presentation affects benchmark outcomes.Source: §5 · p. 9

5.1.1 SWE-Bench Verified Results.

• 20–40% relative success-rate drops occur across all models on mutated SWE-Bench Verified compared with baseline.The authors interpret the result as evidence that existing benchmarks overestimate bug-fixing capabilities; they report a higher drop for GPT-4.1 and offer context needs or overfitting as possible explanations.Source: §5.1.1 · p. 9

5.1.2 SWE-Bench C# Results.

• 2–5% absolute performance drops on mutated SWE-Bench C# are smaller than the 13–16% drops on SWE-Bench Verified.The authors suggest possible explanations including benchmark overfitting and lower starting performance on C#.Source: §5.1.2 · p. 9

5.1.3 Multi-SWE-Bench (TypeScript) Results.

• Mutation reduces performance for all three models on TypeScript, with relative degradation exceeding C#’s for each model; Sonnet 4 has the largest drop and also performs worse on baseline.Source: §5.1.3 · p. 9

5.2.1 SWE-Bench Verified Results.

• On SWE-Bench Verified, mutation increases steps but decreases token usage; Sonnet 4 is more verbose and takes more steps than the other models across benchmarks.The authors suggest fewer details per trajectory step may explain the token decrease despite longer trajectories.Source: §5.2.1 · p. 9

5.2.2 SWE-Bench C# Results.

• On C#, step increases are less pronounced than on SWE-Bench Verified, while Claude models use more tokens and GPT-4.1 uses roughly the same; Sonnet 4 uses almost twice as many tokens per instance as on Python.GPT-4.1 steps decrease slightly, but its performance is also lower, so the authors caution that this may not be positive.Source: §5.2.2 · p. 9

5.2.3 Multi-SWE-Bench (TypeScript) Results.

• On TypeScript, mutation raises steps and token use for all models, most drastically for Sonnet 4; it exhausts the step cap in about three times as many instances as Sonnet 3.7.GPT-4.1 also uses more steps and tokens than on the other benchmarks.Source: §5.2.3 · pp. 9–10

Language / Task Coverage

• The benchmarks cover Python, C#, and TypeScript bug-fixing tasks, leaving the method’s efficacy on other coding tasks unexplored.Bug fixing accounts for less than 15% of queries to chat-based agents.Source: §6 · p. 10

Agent Coverage

• Evaluation uses only OpenHands and single-interaction benchmarks, so it does not cover other agents or follow-up exchanges with clarifications and added context.Other agents may have different capabilities and performance patterns on mutated benchmarks.Source: §6 · p. 10

Template Extraction & Reliance on LLM-based Analysis

• Templates may not transfer across agents with different designs or UI elements, and domain-specific patterns may be absent from the telemetry sample.LLM-based categorization, extraction, and application may also introduce systematic biases from model and prompt choices, despite human validation.Source: §6 · p. 10

Problem Validity Following Mutation

• Mutation may discard essential technical information or change difficulty, and under-specified mutated problems can admit valid fixes other than the original developer’s solution.The validity assumption may therefore lead to unfairly penalizing agents for reasonable solutions.Source: §6 · p. 10

Unchanged Evaluation Methodology & Harness

• The evaluation harness remains unchanged and rigid despite the modified problem statements.The authors call for metrics that capture human-agent interaction and better approximate user satisfaction.Source: §6 · p. 10

6 LIMITATIONS

• Despite these limitations, the work presents an extensible approach that can be adapted to existing and new benchmarks.Source: §6 · p. 10

7 CONCLUSION

• >50% over baselines, existing bug-fixing benchmarks such as SWE-Bench Verified and Multi-SWE-Bench overestimate agent capabilities, which the authors attribute to formal GitHub issues, language parity, and overfitting.Source: §7 · p. 10
• The mutation methodology transforms GitHub-issue-style problems into more realistic chat-style queries, revealing performance gaps and risks of benchmark overfitting.Source: §7 · p. 11
• The authors advocate mutation-based evaluation and private benchmarks, with future work extending the method to other software-engineering tasks and multi-turn collaboration.Source: §7 · p. 11