Source-linked AI summary
Trae Agent: An LLM-based Agent for Software Engineering with Test-time Scaling
Trae Research Team, Pengfei Gao, Zhao Tian, Xiangxin Meng, Xinchen Wang, Ruida Hu, Yuanan Xiao, Yizhou Liu, Zhao Zhang, Junjie Chen, Cuiyun Gao, Yun Lin, Yingfei Xiong, Chao Peng, Xia Liu
TL;DR
Repository-level software issue resolution remains challenging, while prompting-based ensemble methods struggle with large candidate spaces and repository-wide understanding. Trae Agent uses modular agents for generation, pruning, and selection, and consistently outperforms four baselines on SWE-bench, including a 75.20% Pass@1 score on SWE-bench Verified.
Problem
Prompting-based ensemble methods struggle to search large ensemble spaces and lack repository-level understanding for complex software issues.
Method
Trae Agent formulates issue resolution as optimal solution search and uses modular agents for patch generation, pruning, and selection.
Results
Trae Agent consistently outperforms four ensemble reasoning baselines across evaluation settings, achieving a 75.20% Pass@1 score on SWE-bench Verified.
Takeaways & Limitations
The results support agent-based ensemble reasoning as an effective approach for repository-level software issue resolution.
Takeaways & Limitations
The evaluation addresses LLM randomness through repeated experiments, but reported conclusions remain tied to the studied models, benchmark, and settings.
Abstract
from arXiv · showhide
Software issue resolution is a critical challenge in software engineering and has garnered increasing attention in recent years. With the rapid advancement of large language models (LLMs), substantial progress has been made in addressing real-world software engineering tasks. Recent studies have introduced ensemble reasoning techniques to enhance the performance of LLM-based issue resolution. However, existing prompting-based methods still face limitations in effectively exploring large ensemble spaces and lack the capacity for repository-level understanding, both of which constrain their overall effectiveness. In this paper, we propose Trae Agent, the first agent-based ensemble reasoning approach for repository-level issue resolution. Trae Agent formulates our goal as an optimal solution search problem and addresses two key challenges, i.e., large ensemble spaces and repository-level understanding, through modular agents for generation, pruning, and selection. We conduct extensive experiments using three leading LLMs on the widely-adopted SWE-bench benchmark, comparing Trae Agent against four state-of-the-art ensemble reasoning techniques. Experimental results demonstrate that Trae Agent consistently achieves superior performance, with an average improvement of 10.22% over all baselines in terms of Pass@1. Trae Agent has achieved first place on the SWE-bench Verified leaderboard, with a notable Pass@1 score of 75.20%. We are pleased to release Trae Agent as an open-source project to support the research community, with all resources available at https://github.com/bytedance/trae-agent.
1 Introduction
Trae Agent addresses the difficulty of repository-level software issue resolution by using modular agents to search large ensembles of candidate patches. Experiments on SWE-bench show consistent improvements over prompting-based ensemble baselines.
- Repository-level issue resolution remains difficult because it requires global understanding of large codebases and cross-file reasoning.
- Prompting-based ensemble methods struggle to search large candidate spaces and lack repository-level understanding for complex issues.Their single-prompt selection can miss subtle semantic differences between patches.
- 5.83%–14.60%: Trae Agent improves Pass@1 over four state-of-the-art ensemble reasoning baselines across evaluation settings.The evaluation uses three leading LLMs on SWE-bench.
- Trae Agent is an agent-based ensemble reasoning framework that formulates issue resolution as optimal solution search.It uses modular agents for patch generation, pruning, and selection.
- 75.20% Pass@1: Trae Agent achieved first place on the SWE-bench Verified leaderboard.
- The project releases all resources as an open-source artifact for the research community.The repository is available at https://github.com/bytedance/trae-agent.
2 Motivation
Existing ensemble methods can benefit from multiple candidate patches, but increasing ensemble size also makes selection harder. These observations motivate pruning strategies that reduce redundant or faulty candidates while preserving promising solutions.
- As ensemble size increases, Average remains relatively stable while Oracle consistently improves.At ensemble size 10, Oracle improves over Average by 20.80% in Pass@1.
- At ensemble size 10, Adversary decreases by 27.01% compared with Average in Pass@1.The theoretical lower bound deteriorates as the ensemble grows.
- Augment and DeiBase initially improve with larger ensembles but subsequently degrade.This pattern motivates patch pruning to reduce the ensemble space by removing redundant and faulty patches.
- Augment and DeiBase outperform random selection by 3.17% and 1.06%, respectively, in Pass@1.
- Oracle improves over Augment and DeiBase by 29.49% and 36.29%, respectively, revealing substantial selection headroom.
3 Approach
Trae Agent treats repository-level issue resolution as optimal solution search over candidate patches, using modular agents for generation, pruning, and selection. Its pipeline targets ensemble diversity, search-space reduction, and repository-level understanding.
- Overview: Trae Agent generates a diverse candidate set, prunes redundant or faulty patches, and selects a final patch through three modular components.The components are patch generation, patch pruning, and patch selection.
- Patch Generation: Candidate patches are generated in parallel by coder agents equipped with file editing, Bash, sequential thinking, and task-completion tools.Tool interactions are translated into shell commands executed in a Docker-based environment, with JSON feedback returned to the agent.
- Patch Generation: High-temperature sampling, multiple independent runs, and round-robin use of three LLMs increase candidate diversity until ensemble size N is reached.The Mixture setting uses Gemini 2.5 Pro, Claude 3.7 Sonnet, and GPT-4.1 in round-robin fashion.
- Patch Pruning: Hierarchical pruning combines patch deduplication with regression testing to reduce the ensemble space while preserving promising candidates.Deduplication removes redundant patches, while regression testing removes patches that fail to preserve existing functionality.
- Patch Pruning: Regression testing removes faulty candidates but may introduce false positives because tests extracted from the original codebase can be inaccurate.The reported regression-testing error rate is 3.69%.
- Patch Selection: A selector agent builds repository-level understanding by analyzing relevant code and execution traces before identifying the correct patch among pruned candidates.The selector agent gathers code snippets linked by issue references, patch modifications, and dependencies, then collects generated-test traces.
4 Evaluation Design
The evaluation asks whether Trae Agent improves issue resolution, how its hyperparameters and components contribute, and how ensemble size affects selection. Experiments use SWE-bench Verified, Pass@1, three LLMs, and four ensemble baselines under a controlled comparison.
- Research Questions: The study evaluates Trae Agent against four state-of-the-art ensemble reasoning techniques across questions about performance, hyperparameters, components, and ensemble space.The research questions are RQ1 through RQ4.
- Dataset: Evaluation focuses on SWE-bench Verified, a 500-issue subset manually verified by professional software developers.The full SWE-bench contains 2,294 real-world GitHub issues and includes noisy instances according to prior studies.
- Metrics: Pass@1 counts an issue as resolved when the selected patch passes all associated golden tests.Each technique selects one patch per issue, and Pass@1 measures functionality correctness under this stringent criterion.
- Baselines: The comparison includes Augment, DeiBase, and versions of both methods augmented with the same patch-pruning component.Using the same candidate patches isolates differences in ensemble reasoning techniques.
- Models: Patch generation uses Gemini 2.5 Pro, Claude 3.7 Sonnet, and GPT-4.1, with Claude 3.7 Sonnet adopted for consistent comparison after showing the best performance.The models generate patches for the evaluation.
5 Results and Analysis
Trae Agent outperforms the evaluated ensemble reasoning baselines across settings and achieves the strongest Pass@1 and uniquely resolved-issue results. Patch pruning also improves both prompting-based baseline methods.
- Evaluation Scope: The evaluation compares five ensemble reasoning techniques on SWE-bench Verified using Pass@1 and uniquely resolved issues across multiple LLM settings.The settings include Gemini 2.5 Pro, Claude 3.7 Sonnet, GPT-4.1, and a round-robin Mixture setting.
- Effectiveness: Trae Agent achieves the highest number of uniquely resolved issues among the compared ensemble reasoning techniques.This result is reported in Figure 9 on SWE-bench Verified.
- Effectiveness: 5.01%∼12.86% improvement in Pass@1 separates Trae Agent from the baselines across the studied ensemble reasoning techniques.Wilcoxon signed-rank tests report all p-values below 8.00 × 10^-6.
5.2 RQ2: Influence of Hyper-parameter
RQ2 evaluates how ensemble size affects Pass@1. Trae Agent remains superior across sizes, while pruning improves competing ensemble methods and yields statistically significant gains.
- Results: 5.83%∼14.60% average Pass@1 improvement: Trae Agent outperforms four baselines across ensemble sizes from 1 to 10.Wilcoxon tests report p-values smaller than 3.74 × 10−12.
- Results: Baseline Pass@1 generally rises initially and then declines as ensemble size increases, unlike Trae Agent’s consistently upward trajectory.The authors attribute baseline decline to lower-bound degradation and context-length dilution, while noting a computational-cost trade-off for further scaling.
- Results: 3.91% and 3.74% average Pass@1 improvements: pruning benefits Augment and DeiBase, respectively, across ensemble sizes.The corresponding pruned variants consistently outperform their original counterparts, with p-values smaller than 1.95 × 10−11.
5.3 RQ3: Contribution of Main Components
RQ3 evaluates Trae Agent against ablations to measure the contributions of pruning, selection, and majority voting. Each main component improves Pass@1, with statistically significant advantages over the variants.
- Patch pruning: 5.57%, 3.73%, and 3.42% average Pass@1 improvements: Trae Agent surpasses variants without pruning, deduplication, and regression testing.These comparisons support contributions from the pruning component and both of its strategies.
- Patch selection: 4.08% average Pass@1 improvement: Trae Agent outperforms Trae AgentA, supporting the selector agent’s effectiveness.The result also demonstrates superiority over existing prompting-based ensemble reasoning techniques.
- Majority voting: 4.14% average Pass@1 improvement: Trae Agent outperforms Trae AgentwoM, highlighting the contribution of majority voting.The authors state that voting mitigates selection instability and reduces the impact of LLM hallucinations.
- Statistical validation: All Wilcoxon p-values are smaller than 7.32 × 10−3, indicating statistically significant advantages over the ablation variants.The paper concludes that each main component contributes substantially to Trae Agent’s effectiveness.
5.4 RQ4: Influence of Ensemble Space
RQ4 examines whether patch pruning reduces ensemble space and how that space relates to selection effectiveness. The analyses associate pruning with smaller candidate sets and strong correlations between ensemble space and Pass@1.
- Process: The study fixes ensemble size at 10 when comparing Trae Agent with variants that remove pruning, deduplication, or regression testing.This setup isolates the effect of the pruning component and its two strategies on ensemble space.
- Ensemble-space reduction: 27.80%, 30.22%, and 39.15% average ensemble-space reductions: Trae Agent, Trae AgentwoR, and Trae AgentwoD outperform Trae AgentwoP.These results support the effectiveness of pruning, deduplication, and regression testing in reducing remaining candidate patches.
- Correlation analysis: Absolute Pearson’s r values range from 0.73 to 0.91, Spearman’s ρ from 0.80 to 1.00, and Kendall’s τ from 0.67 to 1.00 between ensemble space and Pass@1.Across all three coefficients, the authors characterize the relationship as strong.
6 Discussion
The discussion situates Trae Agent within the growing SWE-bench agent ecosystem and examines regression-test quality and future extensions. Future work targets diversity, larger ensembles, and stronger deduplication.
- Benchmark context: Over 500,000 SWE-bench downloads and more than 90 leaderboard submissions illustrate the benchmark’s broad activity as of July 2025.SWE-bench includes real-world tasks such as bug fixing and feature implementation.
- Regression-test quality: 33.04% false positives versus 3.69% false negatives: regression tests more often preserve incorrect patches than discard correct ones.The paper characterizes false positives as conservative pruning, while false negatives are more detrimental to effectiveness.
- Future work: Trae Agent currently relies on a single coder agent with high-temperature sampling for diverse patch generation.Future work proposes integrating multiple issue-resolution agents to increase candidate diversity.
- Future work: Current experiments evaluate ensemble sizes from 1 to 10, leaving larger sizes and cost-efficiency for future investigation.The reported performance of Trae Agent consistently improves as ensemble size increases within this range.
- Future work: Future work also plans LLM-based patch deduplication techniques to enhance Trae Agent.The discussion connects this direction to studies on LLM-based program-equivalence detection.
7 Threats and Validity
The study addresses construct validity through repeated experiments and statistical testing, and external validity through broad evaluation across models and benchmark comparisons.
- Construct Validity: Three repeated RQ1 experiments produced low Pass@1 standard deviations, indicating robust results across runs.The reported standard deviations range from 0.0019 to 0.0052 across the evaluated methods.
- Construct Validity: p-values exceeding 0.38 across comparisons showed no statistically significant differences among the three repeated experimental results.The Wilcoxon Signed-Rank Test used a significance level of 0.05.
- External Validity: The evaluation uses SWE-bench, multiple issue-resolution metrics, four ensemble reasoning techniques, and three leading LLMs to address external-validity concerns.The authors plan to extend evaluation to additional benchmarks and LLMs to assess generalizability across diverse settings.
8 Related Work
Related work spans agent-based software issue resolution and ensemble reasoning, while Trae Agent extends these directions with a modular agent-based ensemble approach.
- Automatic Software Issue Resolution: Automatic software issue resolution is a growing research area, with agent-based approaches gaining particular prominence.The task is described as critical in software engineering and increasingly studied by both academia and industry.
- Automatic Software Issue Resolution: OpenHands, SWE-agent, and Moatless represent agent-based frameworks that use planning, custom interfaces, and iterative codebase interaction.These systems support issue resolution through tools or interfaces for viewing, editing, and working with repositories.
- Trae Agent: Trae Agent provides a modular foundation that can integrate with individual patch-generation techniques and broader software engineering tasks.Its generalizable design is presented as a potential basis for more complex ensemble reasoning systems.
- Ensemble Reasoning: Ensemble methods combine multiple model outputs to reduce single-model limitations and improve generalization across tasks.Examples include bagging, boosting, stacking, and newer LLM approaches such as Best-of-N and S*.
- Ensemble Reasoning: Augment and DeiBase apply LLM-based judging, explanations, and confidence scores to select among candidate patches.Both methods address ensemble selection for software issue resolution through prompting-based assessment.
9 Conclusion
Trae Agent is presented as an agent-based ensemble reasoning approach for repository-level issue resolution. It formulates ensemble reasoning as optimal solution search and outperforms four baselines across evaluation settings.
- Conclusion: Trae Agent is the first agent-based ensemble reasoning approach proposed for repository-level issue resolution.The approach is designed to improve the effectiveness of LLM-based issue resolution.
- Conclusion: The method addresses large ensemble spaces and repository-level understanding through modular generation, pruning, and selection agents.Its ensemble reasoning process is formulated as an optimal solution search problem.
- Conclusion: Trae Agent consistently outperforms four state-of-the-art ensemble reasoning baselines across all evaluation settings.The experiments use three leading LLMs on the widely used SWE-bench benchmark.