Source-linked AI summary
Evaluating and Improving Automated Repository-Level Rust Issue Resolution with LLM-based Agents
Jiahong Xiang, Wenxiao He, Xihua Wang, Hongliang Tian, Yuqun Zhang
TL;DR
Rust issue resolution is difficult to automate, and existing benchmarks provide limited repository-level evidence for Rust. The paper constructs Rust-SWE-bench and evaluates agents before introducing RustForger’s automated testing and dynamic tracing approach. RustForger with Claude-Sonnet-3.7 resolves 28.6% of tasks, a 34.9% improvement over the strongest baseline, and uniquely solves 46 tasks.
Problem
Existing benchmarks contain sparse real-world Rust issues, limiting systematic evaluation of repository-level Rust issue-resolution agents.
Method
The paper constructs Rust-SWE-bench and introduces RustForger, which combines automated test-environment setup with Rust metaprogramming-driven dynamic tracing.
Results
28.6% of tasks were resolved by RustForger with Claude-Sonnet-3.7, a 34.9% improvement over the strongest baseline, with 46 uniquely solved tasks.
Takeaways & Limitations
Repository-wide code comprehension and issue reproduction are central challenges for automated Rust issue resolution, while RustForger provides the strongest evaluated performance.
Takeaways & Limitations
The generalizability of the findings depends primarily on the evaluation dataset and the selection of compared agents.
Abstract
from arXiv · showhide
The Rust programming language presents a steep learning curve and significant coding challenges, making the automation of issue resolution essential for its broader adoption. Recently, LLM-powered code agents have shown remarkable success in resolving complex software engineering tasks, yet their application to Rust has been limited by the absence of a large-scale, repository-level benchmark. To bridge this gap, we introduce Rust-SWE-bench, a benchmark comprising 500 real-world, repository-level software engineering tasks from 34 diverse and popular Rust repositories. We then perform a comprehensive study on Rust-SWE-bench with four representative agents and four state-of-the-art LLMs to establish a foundational understanding of their capabilities and limitations in the Rust ecosystem. Our extensive study reveals that while ReAct-style agents are promising, i.e., resolving up to 21.2% of issues, they are limited by two primary challenges: comprehending repository-wide code structure and complying with Rust's strict type and trait semantics. We also find that issue reproduction is rather critical for task resolution. Inspired by these findings, we propose RUSTFORGER, a novel agentic approach that integrates an automated test environment setup with a Rust metaprogramming-driven dynamic tracing strategy to facilitate reliable issue reproduction and dynamic analysis. The evaluation shows that RUSTFORGER using Claude-Sonnet-3.7 significantly outperforms all baselines, resolving 28.6% of tasks on Rust-SWE-bench, i.e., a 34.9% improvement over the strongest baseline, and, in aggregate, uniquely solves 46 tasks that no other agent could solve across all adopted advanced LLMs.
1 Introduction
Rust’s safety guarantees make issue resolution difficult, while existing agent benchmarks provide limited Rust coverage. The paper introduces Rust-SWE-bench, studies agents and models, and proposes RustForger to improve reproduction and analysis.
- Motivation: Rust’s strict type and ownership systems improve safety but contribute to a steep learning curve and coding difficulty.These properties provide compile-time protection against memory faults and data races.
- Motivation: Existing repository-level agent tasks require codebase understanding, targeted search, issue reproduction, and precise patching.Agents must generate a corrective patch that passes tests in one attempt.
- Empirical study: 21.2% of tasks were resolved by OpenHands with Claude-Sonnet-3.7, while agents remained limited by repository-wide comprehension and issue reproduction.The study evaluates four representative agents and four state-of-the-art LLMs.
- RustForger: RustForger combines automated test-workspace setup with cross-project dynamic tracing powered by Rust procedural macros.Its Trace command instruments target functions through AST modification to support dynamic analysis.
- Contributions: Rust-SWE-bench contains 500 verified repository-level Rust tasks from 34 popular repositories, addressing the lack of a large-scale Rust benchmark.The benchmark is intended to support systematic evaluation and future research.
- Results: 28.6% of tasks were resolved by RustForger with Claude-Sonnet-3.7, and it uniquely solved 46 tasks across the adopted LLMs.The framework is presented as outperforming the evaluated baselines.
2 Background & Related Work
LLM-based agents extend language models with planning, memory, perception, and tool-enabled action. Repository-level benchmarks test autonomous patch generation, but Rust remains underrepresented despite the importance of issue reproduction.
- LLM-based agents: LLM-based agents use an LLM as a controller that perceives and acts on an environment through planning, memory, perception, and tools.Their action component enables interaction with and modification of external environments.
- Repository-level tasks: Repository-level issue resolution requires understanding the codebase, searching precisely, reproducing issues, and editing code accurately.Issue reproduction supplies dynamic execution information for localization and patch generation.
- Agent paradigms: ReAct agents interleave reasoning and action in a thought-act-observe loop, with tools supporting repository search, editing, and navigation.SWE-agent and OpenHands instantiate this paradigm through specialized interfaces and CodeAct.
- Benchmarks: SWE-bench evaluates end-to-end software maintenance using 2,294 real-world tasks from 12 Python repositories.Each task requires an autonomous corrective patch that resolves the issue.
- Rust benchmark gap: Rust issues remain sparse in multilingual benchmarks, with 43 tasks in SWE-bench Multilingual and 239 in Multi-SWE-bench.The paper identifies a need for a larger Rust-specific benchmark for systematic evaluation.
3 Rust-SWE-bench
Rust-SWE-bench is constructed from verified pull-request tasks and manually reviewed for quality. Its 500 tasks span diverse repositories and involve substantially larger, more complex changes than Python SWE-bench tasks.
- Construction: Rust-SWE-bench construction begins with popular, maintained Rust repositories having extensive tests and rich pull-request histories.The process scrapes approximately 80,000 merged pull requests and their base-commit code states.
- Construction: Candidate pull requests are retained when linked to an issue and modifying the test suite, separating test patches from fix patches.This filtering supports issue-resolving task construction.
- Validation: Each task is validated through fail-to-pass execution, requiring a test transition to pass, no regressions, and a complete valid patch.The validation compares the base state with the test patch and then the full fix.
- Validation: Manual review checks issue quality, fail-to-pass integrity, and solution leakage before tasks are included.Three dimensions are assessed to confirm that instances are clear, deterministic, non-trivial, and not explicitly solved in the issue.
- Dataset characteristics: 500 manually reviewed tasks span 34 repositories and diverse domains, including command-line tools, development libraries, concurrency libraries, web frameworks, and specialized engines.The benchmark is larger and more diverse than the cited Rust SWE benchmarks.
- Dataset characteristics: Rust-SWE-bench projects average 993.6 files and 128,126 lines of code, while issues require 9.8 files, 9.9 hunks, and 139.9 changed lines on average.These values contrast with 1.25 files, 2.46 hunks, and 14.32 lines for Python SWE-bench Verified.
- Task distribution: 52% of tasks are bug fixes and 27% are feature implementations, indicating emphasis on maintenance and evolution workflows.The benchmark’s average 139.9 edited lines also reflects substantial code churn.
4.1 Study Setup
The study evaluates four representative SWE agents and four advanced LLMs on Rust-SWE-bench using Pass@1 resolution, cost, token usage, and reproduction success metrics.
- Agents: Four agents—SWE-agent, OpenHands+CodeAct v2.1, Agentless, and AutoCodeRover v2.0—are evaluated on Rust-SWE-bench.The agents represent distinct general-purpose approaches for software engineering tasks.
- LLMs: Four LLMs—Claude-Sonnet-3.7, GPT-4o, OpenAI o4-mini, and Qwen3—are adopted for the evaluation.Qwen3 is obtained from Hugging Face, while the other models are accessed through provider APIs.
- Metrics: Resolved Rate (%) is measured as Pass@1, requiring one generated patch to apply and pass held-out developer-written acceptance tests.The study also reports average API cost and token usage.
- Metrics: Reproduction Success Rate (%) measures the proportion of tasks whose agent-generated test replicates the behavior described in the original issue.This metric specifically evaluates the issue-reproduction stage.
4.2 Implementations
The Rust adaptations provide agents with Rust-specific prompting and shell tooling, while reproduction trajectories are re-executed in sandboxed containers for evaluation.
- Rust adaptations: The studied agents are adapted for Rust through modified prompts, custom shell tools, and Rust-specific program-structure APIs.These changes support Cargo environment and dependency management during issue reproduction.
- Reproduction evaluation: Reproduction commands and files are extracted from agent trajectories and re-executed inside sandboxed Docker containers.The process first checks syntactic validity before analyzing whether the reproduced behavior matches the issue.
4.3 Research Questions
The evaluation asks how agents perform on Rust-SWE-bench and what behavioral patterns and failure factors characterize their attempts to resolve real-world Rust issues.
- RQ1: performance: RQ1 measures the overall effectiveness of selected agent-model configurations on Rust-SWE-bench.The question compares how different agents perform across the benchmark.
- RQ2: behavioral analysis: RQ2 analyzes agents’ behavioral characteristics while resolving real-world Rust issues.The analysis examines edit scope, compilation errors, and reproduced tests.
4.4 Result Analysis
ReAct-style agents perform best overall, but their effectiveness declines on larger patches and is constrained by repository-wide structure, Rust type and trait semantics, and issue reproduction failures.
- RQ1: performance: OpenHands with Claude-Sonnet-3.7 resolves 106 tasks (21.2%), outperforming SWE-agent at 15.0% and OpenAI o4-mini at 6.8%.The result is the strongest agent-model configuration reported for Rust-SWE-bench.
- RQ1: task categories: Bug fixes (57%) and feature implementations (24%) dominate the resolved tasks, closely matching the benchmark’s task composition.The resolved-task distribution therefore concentrates on common Rust maintenance and evolution categories.
- RQ2: edit scope: Within a 15-line edit scope, OpenHands resolves 48 tasks and Agentless 43, but within 150 lines they resolve 105 and 57 tasks, respectively.The widening gap indicates stronger agents handle larger, more complex patches more effectively.
- RQ2: compilation errors: OpenHands encounters an average of three compilation errors per task, with 43.7% linked to repository-wide code organization.These failures include naming, scoping, and path-resolution errors during issue resolution and reproduction.
- RQ2: compilation errors: Rust’s type and trait system accounts for 32.6% of compilation errors, reflecting failures to satisfy strict semantic contracts despite syntactically plausible code.The reported error classes include E0599, E0308, E0277, and E0407.
- RQ2: issue reproduction: OpenHands with Claude-Sonnet-3.7 achieves a 55.5% reproduction success rate, leaving 44.5% of tasks unable to pass reproduction.For successfully reproduced issues, it reruns reproduction tests 4.7 times on average for validation and dynamic information gathering.
5 Approach
RustForger combines isolated issue-reproduction workspaces with cross-project dynamic tracing, then evaluates resolution, cost, reproduction, complexity, and ablation outcomes against baseline agents. Its strongest configuration resolves 28.6% of Rust-SWE-bench tasks and improves reproduction and cost efficiency across tested models.
- Framework: RustForger first creates an isolated testing workspace for reliable issue reproduction, then uses dynamic analysis for code analysis and patch generation.The framework separates workspace setup from subsequent tracing-assisted patch generation.
- Dynamic tracing: The Trace command automates AST instrumentation, test execution, runtime data collection, and cleanup through a unified agent-computer interface.Its tracing runtime captures serializable values as JSON and descriptive placeholders for non-serializable Rust types.
- Evaluation results: 28.6% of tasks were resolved by RustForger with Claude-Sonnet-3.7, outperforming the strongest baseline by 34.9%.The result corresponds to 143 of 500 tasks; the Trace command was used in 71.3% of these resolution processes, with p < 0.001 against OpenHands.
- Evaluation results: $3.0 was RustForger’s average cost with Claude-Sonnet-3.7, 21.3% below OpenHands and 13.8% below SWE-agent.With OpenAI o4-mini, its $0.6 cost was less than half of OpenHands’ $1.23 and 28.2% of SWE-agent’s $2.13.
- Evaluation results: RustForger’s success rate declined from 42.9% for issues resolved within a day to 15.0% for issues open longer than six months.The intermediate rates were 32.3% within a week, 31.5% within a month, and 22.0% within six months.
- Evaluation results: 67.3% was RustForger’s reproduction success rate with Claude-Sonnet-3.7, versus 55.5% for OpenHands; with OpenAI o4-mini, the rates were 45.4% and 32.6%.RustForger consistently outperformed all baselines in reproduction success across the tested LLMs.
- Ablation study: The isolated workspace added 3.0 resolved tasks on average, while Trace added 5.3 more in the ablation study.The study compared full RustForger with variants disabling Trace or skipping the isolated workspace.
- Ablation study: Trace helped resolve a representative macro issue by verifying distinct runtime dispatch for named fields and tuple indices.The example involved extending animated_field! while preserving named-field behavior.
6 Threats to Validity
The study addresses internal and external validity by controlling execution environments, reducing stochasticity, and using a diverse benchmark and representative agents. Its findings remain tied to the selected Rust-SWE-bench tasks and compared systems.
- Internal validity: Dedicated Docker containers preserve the repository state, toolchain, and dependencies across experimental runs.The authors use this control to support reproducibility and reduce system-level variation.
- External validity: The study’s external validity depends on Rust-SWE-bench and the selected agents, although the benchmark contains 500 tasks from 34 popular Rust repositories.Each task comes from an actual merged pull request, and the agents were selected for representative SWE-bench performance.
7 Conclusion
The paper introduces Rust-SWE-bench and finds that repository-wide comprehension and issue reproduction hinder current LLM agents on Rust tasks. RustForger addresses these challenges and achieves the strongest reported performance.
- Benchmark and findings: Rust-SWE-bench contains 500 repository-level Rust issue-resolution tasks from diverse and popular repositories.It provides the benchmark foundation for the paper’s empirical study.
- Benchmark and findings: Current LLM-based agents are primarily hindered by repository-wide code comprehension and issue reproduction.These challenges motivate RustForger’s design.
- RustForger: RustForger integrates an automated testing workspace with cross-project dynamic tracing to address those limitations.The framework targets both reliable reproduction and runtime analysis.
- Results: 28.6% of tasks were resolved with Claude-Sonnet-3.7, a 34.9% improvement over the strongest baseline, while 46 issues were uniquely solved across adopted LLMs.These are the paper’s reported headline outcomes.