Source-linked AI summary
SE-Agent: Self-Evolution Trajectory Optimization in Multi-Step Reasoning with LLM-Based Agents
Jiaye Lin, Yifu Guo, Yuzhen Han, Sen Hu, Ziyi Ni, Licheng Wang, Mingguang Chen, Hongzhang Liu, Ronghao Chen, Yangfan He, Daxin Jiang, Binxing Jiao, Chen Hu, Huacan Wang
TL;DR
Multi-step agents generate rich reasoning trajectories, but existing methods often ignore relationships among trajectories and produce redundant paths. SE-Agent iteratively revises, recombines, and refines trajectories to expand search beyond local optima and leverage cross-trajectory insights. On SWE-bench Verified, it delivers up to 55% relative improvement and state-of-the-art performance among open-source agents.
Problem
Existing multi-step reasoning methods underuse interdependencies among trajectories, while diverse sampling often produces redundant solution paths and homogeneous outcomes.
Method
SE-Agent iteratively optimizes reasoning trajectories through revision, recombination, and refinement, selecting high-reward trajectories across evolution cycles.
Results
SE-Agent consistently outperforms strong baselines across five LLMs on SWE-bench Verified, achieving up to 55% relative improvement and state-of-the-art open-source-agent performance.
Takeaways & Limitations
Trajectory-level self-evolution expands the exploration space and uses cross-trajectory inspiration to improve reasoning quality in complex environments.
Abstract
from arXiv · showhide
Large Language Model (LLM)-based agents have recently shown impressive capabilities in complex reasoning and tool use via multi-step interactions with their environments. While these agents have the potential to tackle complicated tasks, their problem-solving process, i.e., agents' interaction trajectory leading to task completion, remains underexploited. These trajectories contain rich feedback that can navigate agents toward the right directions for solving problems correctly. Although prevailing approaches, such as Monte Carlo Tree Search (MCTS), can effectively balance exploration and exploitation, they ignore the interdependence among various trajectories and lack the diversity of search spaces, which leads to redundant reasoning and suboptimal outcomes. To address these challenges, we propose SE-Agent, a Self-Evolution framework that enables Agents to optimize their reasoning processes iteratively. Our approach revisits and enhances former pilot trajectories through three key operations: revision, recombination, and refinement. This evolutionary mechanism enables two critical advantages: (1) it expands the search space beyond local optima by intelligently exploring diverse solution paths guided by previous trajectories, and (2) it leverages cross-trajectory inspiration to efficiently enhance performance while mitigating the impact of suboptimal reasoning paths. Through these mechanisms, SE-Agent achieves continuous self-evolution that incrementally improves reasoning quality. We evaluate SE-Agent on SWE-bench Verified to resolve real-world GitHub issues. Experimental results across five strong LLMs show that integrating SE-Agent delivers up to 55% relative improvement, achieving state-of-the-art performance among all open-source agents on SWE-bench Verified. Our code and demonstration materials are publicly available at https://github.com/JARVIS-Xs/SE-Agent.
1 Introduction
LLM agents form reasoning trajectories through multi-step environment interactions, but existing methods underuse their cross-trajectory information and often produce redundant solution paths. SE-Agent addresses this by iteratively manipulating trajectories to expand solution diversity and improve task performance.
- Multi-step agent interactions form trajectories that encode complete problem-solving attempts and valuable reasoning strategies.
- MCTS balances exploration and exploitation but treats trajectories independently, missing interdependencies and synergies among solution paths.
- Varying sampling temperatures or prompts often produces structurally similar trajectories with homogeneous outcomes.
- SE-Agent uses revision, recombination, and refinement to generate diverse trajectories and expand the candidate solution space.
- Experiments on SWE-bench Verified show significant performance improvements across different LLMs, validating trajectory-level self-evolution for software engineering tasks.
2 Related Works
Related work covers code agents for autonomous repository-level software engineering and methods that enhance agent capabilities through planning, debugging, and multi-agent optimization. SE-Agent is evaluated in this broader context of increasingly complex code tasks.
- Code Agents: Code agents autonomously understand, generate, and manipulate source code for increasingly complex software engineering tasks.
- Code Agents: 180.0% is achieved with Claude-4-Sonnet using the latest SWE-Agent together with SE-Agent, against a 66.6% resolved baseline.
- Code Agents: SWE-bench evaluates agents on resolving real-world GitHub issues by automatically applying functional bug fixes.
- Agent Capability Enhancement: Recent enhancement methods include hierarchical planning, multi-agent debugging, and iterative optimization for improving agent performance.
3 Preliminaries and Problem Setup
The paper models complex tasks as reasoning environments with states, actions, transitions, and rewards, and represents agent behavior as rewarded state-action trajectories. SE-Agent seeks a policy that maximizes expected trajectory reward.
- Task-Oriented Reasoning Environment: A reasoning environment is modeled as E = (T, S, A, P, R), comprising tasks, states, actions, transition dynamics, and a reward function.
- Reasoning Trajectories: A reasoning trajectory is an ordered sequence of states and actions generated by repeatedly applying a policy to a task.
- Reasoning Trajectories: Trajectory quality is measured by the final reward R(τ, t), which evaluates how well the final state satisfies the task requirements.
- Objective of SE-Agent: The objective is to find a policy π* that maximizes expected reward across tasks.
- SE-Agent Framework: Figure 1 depicts SE-Agent beginning with diverse pilot trajectories and applying trajectory-level operators to reach a high-reward solution path.
4 SE-Agent
SE-Agent treats pilot trajectories as improvement operators and evolves reasoning paths through iterative refinement and cross-trajectory learning. It repeatedly generates improved trajectories and selects the best one.
- Overview: SE-Agent alternatively generates a series of improved trajectories and chooses the best trajectory to derive a high-reward solution.
- Overview: Pilot trajectories act as an improvement operator applied to an apprentice trajectory, enabling continuous evolution of reasoning paths.
- Overview: The framework evolves reasoning paths through iterative refinement and cross-trajectory learning.
4.1 Overview of SE-Agent
SE-Agent treats reasoning trajectories as evolving, collectively useful solution paths rather than isolated attempts. It iteratively improves them through revision, recombination, and refinement to expand exploration and leverage cross-trajectory insight.
- Overview: SE-Agent begins with diverse trajectories and iteratively generates improved generations instead of stopping after selecting the best initial path.The final output is the highest-rewarding trajectory after evolution.
- Overview: Its three operations are revision, recombination, and refinement, respectively improving individual paths, combining strengths, and removing redundancies.
- Overview: The framework escapes local optima by exploring solution spaces through prior experience while using cross-trajectory inspiration to improve reasoning efficiently.
- Overview: SE-Agent resembles evolutionary computation and expert iteration but explicitly manipulates complete reasoning trajectories rather than isolated state-action pairs.
4.2 Revision Operation
The revision operation creates a broad starting pool and improves each trajectory through reflection, targeted correction, and selective incorporation of alternative approaches.
- Revision Operation: SE-Agent builds its initial trajectory pool using complementary planning variation and controlled mutations of existing paths.Planning strategies, prompting techniques, reasoning approaches, and targeted changes to steps or conclusions diversify the pool.
- Revision Operation: For every initial trajectory, reflection analyzes strengths, weaknesses, logical inconsistencies, and underdeveloped reasoning before revision.
- Revision Operation: Revision removes redundant or circular reasoning and incorporates alternative perspectives when they are likely to improve trajectory effectiveness.
4.3 Recombination Operation
Recombination enables collective evolution by combining, transferring, and reorganizing useful information across trajectories to produce stronger reasoning paths.
- Recombination Operation: Recombination performs cross-trajectory learning through crossover, transfer learning, and restructuring.
- Recombination Operation: Crossover combines high-performing segments from different reasoning paths so hybrid trajectories inherit strengths from multiple parents.
- Recombination Operation: Transfer learning moves knowledge and effective strategies from successful trajectories into less-developed or suboptimal paths.
- Recombination Operation: Restructuring reorganizes trajectories using collective insights and global analysis of the entire trajectory pool.
4.4 Refinement Operation
Refinement evaluates trajectories across task completion, reasoning quality, efficiency, and diversity, then iteratively retains strong and distinct candidates until convergence. The final output is the highest-rewarding trajectory.
- Evaluation and Selection: SE-Agent scores trajectories with a multi-dimensional reward combining task completion, reasoning quality, and efficiency.The weights w1, w2, and w3 customize the relative importance of these dimensions.
- Evaluation and Selection: Task completion combines rule-based structural validation with LLM-based expert evaluation of solution quality.The expert-evaluation contribution is weighted by λ.
- Evaluation and Selection: Selection retains high-reward trajectories while preserving distinct reasoning approaches through trajectory-dissimilarity metrics.
- Evaluation and Selection: Evolution stops after a preset cycle count or when maximum-reward improvement remains below threshold ϵ for consecutive iterations.The final output is the highest-rewarding trajectory in the final pool.
5 Experiments
Experiments evaluate SE-Agent on SWE-bench Verified across multiple models and compare it with established baselines. Results show consistent gains, with ablations and case studies linking performance to trajectory diversity and optimization.
- Experimental Setup: SE-Agent is evaluated on SWE-bench Verified, a 500-issue benchmark of real-world GitHub bugs verified by developer-written unit tests.The evaluation uses resolution metrics including Pass@1 and Pass@5.
- Experimental Setup: The study compares SE-Agent with SWE-Agent and SWE-Search across three open-source and two closed-source LLMs.The framework is designed as a plug-and-play module within existing agent frameworks.
- Performance Comparison: SE-Agent consistently outperforms both baselines across all five evaluated LLMs, with relative gains of up to +112% over SWE-Agent and +30% on average over SWE-Search.The reported gains include +112% for Llama-3.1-70b-Instruct, +80% for GPT-4o, and +51% for Claude-3.7-Sonnet against SWE-Agent.
- Ablation Study: Removing any trajectory-optimization module decreases Pass@1, while revision supplies diverse trajectories for subsequent recombination.The ablation compares variants without revision, without recombination, and without all trajectory-optimization operations.
- Ablation Study: SE-Agent uniquely resolves 12 issue instances that none of the other evaluated models solve, while also substantially overlapping with leading baselines.The results indicate both competitive performance on shared tasks and complementary coverage of difficult issues.
- Hyperparameter Analysis: With 10 candidate trajectories, SE-Agent reaches near-optimal performance and consistently surpasses baselines under equal maximum API-cost budgets.The case studies illustrate trajectory optimization through revision, recombination, and refinement, including a root-level repair that passes the full test suite.
6 Conclusion
The paper introduces SE-Agent to improve multi-step reasoning through iterative trajectory optimization. By revising, recombining, and refining trajectories, it expands exploration and improves performance across multiple LLMs on SWE-bench Verified.
- Conclusion: SE-Agent enhances LLM-agent reasoning by iteratively optimizing trajectories through revision, recombination, and refinement.The framework is presented as a self-evolution approach for multi-step reasoning.
- Conclusion: Experiments on SWE-bench Verified show that SE-Agent consistently outperforms strong baselines across multiple LLMs.The conclusion positions self-evolution as valuable for more robust and adaptable reasoning frameworks.
A Prompts
The prompts guide agents to reproduce and investigate software issues, implement and verify fixes, and improve solutions through trajectory reflection, revision, and recombination.
- A.1.1 Multi-Planning: Multi-planning strategies repeatedly combine issue reproduction, code tracing, targeted modification, validation, and documentation of changes.Several strategy variants use search, IDE navigation, debugging, and explicit success indicators.
- A.1.1 Multi-Planning: Agents first reproduce the reported issue, investigate relevant code, implement aligned changes, verify the fix, and expand test coverage.The workflow includes regression checks and refinement when tests fail.
- A.1.2 Reflection and Revision: Reflection prompts identify assumptions, untouched components, and different perspectives before developing an alternative solution.The alternative should solve the same problem while avoiding the first solution’s techniques and challenging its assumptions.
- A.1.2 Reflection and Revision: Critical-decision analysis selects the single trajectory step with the greatest influence on the final solution.The requested output records the step, action, reasoning, and impact.
- A.1.2 Reflection and Revision: Locating the exact problem function is treated as a breakthrough because it converts unsuccessful searching into a concrete modification target.The example identifies the function’s file and line after earlier searches failed.
- A.1.2 Reflection and Revision: The reflection process concludes which features of a patch matter for constructing an alternative solution.The analysis captures modified files, key changes, strategy, techniques to avoid, and files or functions to avoid modifying.
- A.1.2 Reflection and Revision: Patch analysis abstracts an existing solution into its summary, modified files, key changes, strategy, and specific techniques.These fields are illustrated through a classifier-related example involving an estimators_ attribute.
- A.2 Recombination Operation: Recombination is introduced as a separate operation for analyzing and synthesizing agent trajectories.The supplied prompt identifies this operation but does not specify its full procedure.