Source-linked AI summary
Live-SWE-agent: Can Software Engineering Agents Self-Evolve on the Fly?
Chunqiu Steven Xia, Zhe Wang, Yan Yang, Yuxiang Wei, Lingming Zhang
TL;DR
Fixed agent scaffolds are costly to design, while existing self-improving agents rely on expensive offline training and may generalize poorly. LIVE-SWE-AGENT instead evolves its tooling during runtime from a bash-only scaffold, achieving strong results on two SWE-bench benchmarks without test-time scaling. The paper positions live self-evolution as a general direction, while its current implementation focuses primarily on custom tool creation.
Problem
Fixed scaffolds are difficult and costly to optimize, while existing self-improving agents require costly offline benchmark training and may not generalize across LLMs or benchmarks.
Method
LIVE-SWE-AGENT starts with bash-only access and autonomously creates, modifies, and uses custom executable tools during runtime issue solving.
Results
77.4% resolve rate on SWE-bench Verified and 45.8% on SWE-Bench Pro were achieved without test-time scaling, surpassing existing open-source and commercial systems.
Takeaways & Limitations
Live-SWE-agent demonstrates that software agents can continuously evolve during real-world issue solving and outperform state-of-the-art manually designed agents on the evaluated benchmarks.
Takeaways & Limitations
The current work focuses primarily on self-evolution through custom tool creation and usage rather than modifying the entire agent implementation.
Abstract
from arXiv · showhide
Large Language Models (LLMs) are reshaping almost all industries, including software engineering. In recent years, a number of LLM agents have been proposed to solve real-world software problems. Such software agents are typically equipped with a suite of coding tools and can autonomously decide the next actions to form complete trajectories to solve end-to-end software tasks. While promising, they typically require dedicated design and may still be suboptimal, since it can be extremely challenging and costly to exhaust the entire agent scaffold design space. Recognizing that software agents are inherently software themselves that can be further refined/modified, researchers have proposed a number of self-improving software agents recently, including the Darwin-Gödel Machine (DGM). Meanwhile, such self-improving agents require costly offline training on specific benchmarks and may not generalize well across different LLMs or benchmarks. In this paper, we propose Live-SWE-agent, the first live software agent that can autonomously and continuously evolve itself on-the-fly during runtime when solving real-world software problems. More specifically, Live-SWE-agent starts with the most basic agent scaffold with only access to bash tools (e.g., mini-SWE-agent), and autonomously evolves its own scaffold implementation while solving real-world software problems. Our evaluation on the widely studied SWE-bench Verified benchmark shows that LIVE-SWE-AGENT can achieve an impressive solve rate of 77.4% without test-time scaling, outperforming all existing software agents, including the best proprietary solution. Moreover, Live-SWE-agent outperforms state-of-the-art manually crafted software agents on the recent SWE-Bench Pro benchmark, achieving the best-known solve rate of 45.8%.
1 Introduction
LIVE-SWE-AGENT addresses the cost and rigidity of fixed or offline-trained software agents by evolving its tooling during runtime. It achieves 77.4% on SWE-bench Verified and 45.8% on SWE-Bench Pro without test-time scaling.
- Motivation: Existing agents use fixed scaffolds, while manually designing an optimal scaffold is challenging and costly because the design space is extremely large.Self-evolving alternatives modify scaffolds using offline benchmark evaluation but add substantial cost, including an approximately $22,000 DGM run on SWE-bench.
- Approach: LIVE-SWE-AGENT starts with bash-only access and continuously synthesizes, modifies, and executes custom tools while solving an issue.A reflection step asks whether creating or revising a tool would accelerate progress, without changing the underlying scaffold.
- Contribution: Custom tool creation adapts the agent’s action space to each problem and shifts improvement from offline training to online evolution.Tool synthesis is iterative and interleaved with problem solving, allowing tools to be refined as the failure mode becomes clearer.
- Results: 77.4% resolve rate on SWE-bench Verified and 45.8% on SWE-Bench Pro were achieved without test-time scaling, surpassing reported open-source and commercial systems.The analysis also reports benefits across LLM backends and both generic and issue-specific synthesized tools.
2 Approach
LIVE-SWE-AGENT lets an agent extend its capabilities during issue solving by creating executable custom tools. The process combines tool use, environmental feedback, and repeated reflection to produce general, efficient, and issue-specific utilities.
- Runtime self-evolution: The agent receives a codebase and issue description, then chooses at each step between issuing a command and creating a custom tool.The loop continues until the agent submits a solution.
- Tool interface: Custom tools are executable scripts, giving the agent a direct interface for creating and invoking task-specific capabilities.The interface requires only creating a script file and running it with arguments.
- Tool synthesis: Prompt instructions and post-feedback reflection guide the agent to create tools that help solve the current issue.The reflection step was necessary in experiments to remind agents to design useful, issue-specific tools.
- Design: The framework makes only simple prompt-level modifications, preserving the regular agentic loop and avoiding offline training.It does not impose a particular workflow or require changes to the agentic loop.
- General tools: Custom editing tools provide focused operations and explicit feedback that can prevent silent failures possible with commands such as sed.The example tool reports whether a replacement succeeded, helping inform subsequent actions.
- Efficiency and specialization: Custom search tools combine multi-step code navigation, reducing context length and time compared with separate grep, find, and cat commands.The framework also creates issue-specific tools, such as a MARC-file analyzer for inspecting binary test cases.
- Iterative synthesis: Tool creation is iterative because useful tools may become apparent only after the agent understands the issue and its failure modes.The MARC-file example illustrates why generating every possible tool at the beginning is impractical.
3 Experimental Setup
The experiments build LIVE-SWE-AGENT on mini-SWE-agent, evaluate it on SWE-bench Verified and SWE-Bench Pro, and compare it with representative agentic and self-evolving baselines.
- Implementation: LIVE-SWE-AGENT is implemented on mini-SWE-agent, retaining a maximum step limit of 250 and a maximum cost of $3 per issue.The base framework has approximately 100 lines of code and accesses only bash commands.
- Datasets: SWE-bench Verified contains 500 software development problems, while SWE-Bench Pro contains 731 publicly available problems targeting realistic, complex, enterprise-level issues.Verified problems are validated by human developers; Pro is intended to capture more challenging settings.
- Baselines: For SWE-bench Verified, the study compares against mini-SWE-agent and prior self-evolving agents including SICA, DGM, and HGM.The self-evolving comparison uses a subset of 60 SWE-bench Verified problems previously used to evaluate those baselines.
- Baselines: For SWE-Bench Pro, LIVE-SWE-AGENT is compared against SWE-agent, identified as the top-performing approach on that benchmark’s leaderboard.Baseline performance, cost, and backend LLM are reused from prior experimental results whenever possible.
4 Evaluation
LIVE-SWE-AGENT improves software-issue resolution by creating task-relevant tools during problem solving, achieving strong results across SWE-bench Verified and SWE-Bench Pro. Tool analyses and ablations indicate that custom tools improve efficiency and effectiveness, while gains depend on model capability.
- Main Results: 77.4% solve rate on SWE-bench Verified without test-time scaling surpassed all existing agents, including state-of-the-art commercial solutions.The result uses LIVE-SWE-AGENT with Gemini 3 Pro and single-attempt evaluation.
- Main Results: 45.8% resolve rate on SWE-Bench Pro established new state-of-the-art performance against manually designed and other top-performing agents.LIVE-SWE-AGENT outperformed SWE-agent, whose scaffold contains close to 7,000 lines of code.
- Tools Analysis: Custom tools cluster by common functionality while also exhibiting repository-specific variation across SWE-bench tasks.The t-SNE analysis identifies clusters for edit, view, and search tools, plus distinct repository-specific clusters such as openlibrary.
- Tools Analysis: Custom search and Go-analysis tools provide focused code search, contextual results, and structural information that basic bash command chains reproduce less conveniently.The examples include directory and pattern filtering, ignored folders, surrounding context, and Go definitions, references, and imports.
- Discussion and Future Work: LIVE-SWE-AGENT is positioned as a lightweight framework for jointly evaluating LLM tool creation and software-issue resolution.The approach requires no offline training and can be applied to other software-engineering domains or extended across tasks.
5 Related Work
Software-agent research has progressed from interactive bug fixing and multi-turn context gathering to self-improving agents. Existing self-improving approaches address scaffold-design difficulty but rely on costly offline training and may face generalization limits.
- Interactive Coding Agents: Interactive bug-fixing systems use environmental feedback and multi-turn conversations to provide LLMs with additional context for coding tasks.ChatRepair introduced an interactive bug-fixing approach inspired by iterative developer debugging.
- Self-Improving Agents: Self-improving agents modify their scaffold implementations and empirically validate changes, responding to the huge and costly software-agent design space.Examples include SICA, Darwin-Gödel Machine, and Huxley-Gödel Machine.
- Self-Improving Agents: Existing self-improving agents require costly offline training on known benchmarks, while LIVE-SWE-AGENT requires no offline training and can generalize across LLMs and issue domains.The comparison motivates runtime self-evolution as an alternative to benchmark-specific offline improvement.
- Software-Agent Benchmarks: SWE-bench and curated variants provide widely used datasets for evaluating software agents on real-world GitHub issues.SWE-bench Verified is described as a curated subset supporting faster and more reliable evaluation.
6 Conclusion
The paper presents LIVE-SWE-AGENT as a runtime self-evolving agent that continuously modifies its scaffold while solving real-world software problems. Evaluations on SWE-bench Verified and SWE-bench Pro show it outperforms manually designed software agents.
- LIVE-SWE-AGENT autonomously and continuously evolves its scaffold implementation during runtime while solving real-world software problems.
- The agent begins with only bash tools and develops its own scaffold while working on software problems.
- LIVE-SWE-AGENT outperforms state-of-the-art manually designed software agents on SWE-bench Verified and SWE-bench Pro.
A Additional experimental settings
The additional evaluation settings implement LIVE-SWE-AGENT on mini-SWE-agent with the framework's default step and cost limits. Model temperatures follow developer recommendations or mini-SWE-agent conventions.
- LIVE-SWE-AGENT is implemented on top of mini-SWE-agent for the additional evaluations.
- The experiments use a maximum step limit of 250 and a maximum cost of $3 per issue.
- Temperature settings vary by model family: Gemini 3 Pro uses 1, Anthropic models use 0.0, and OpenAI models use 1.
B Additional tool analysis
Additional tool analysis visualizes generated tools with t-SNE embeddings labeled by repository and tool name. The analysis uses PCA preprocessing and reports patterns similar to the main evaluation.
- Figure 6 visualizes generated tools with two-dimensional t-SNE embeddings from Claude 4.5 Sonnet on SWE-bench Verified and SWE-Bench Pro.
- The visualizations label embeddings by repository name and tool name, with two representative repository clusters shown in the legend.
- The additional analysis uses PCA with n_components 50 followed by sklearn t-SNE with max_iter 1000, and observes patterns similar to the main evaluation.
C.1 SWE-bench Verified ablation subset
The SWE-bench Verified ablation evaluations use 50 randomly selected problems. The listed subset spans multiple repositories, including Django, SymPy, scikit-learn, matplotlib, Sphinx, Astropy, xarray, and Requests.
- The ablation evaluations use 50 randomly selected problems from SWE-bench Verified.
- The listed ablation subset includes problems from Django, SymPy, scikit-learn, matplotlib, Sphinx, Astropy, xarray, and Requests repositories.
- Django appears repeatedly among the listed ablation problems, alongside issues from several scientific and Python software repositories.
C.2 SWE-bench Multilingual ablation subset
This subset comprises 50 randomly selected SWE-bench Multilingual problems, listed by instance ID across multiple repositories and projects.
- The subset includes Django, Xarray, Requests, Seaborn, Astropy, Matplotlib, scikit-learn, SymPy, and Pytest instances.
- 50 randomly selected problems form the SWE-bench Multilingual evaluation subset.The passage identifies the subset size but does not provide performance results here.
- Additional instances come from JavaParser, Laravel, PHPSpreadsheet, Project Lombok, Tokio, and uutils coreutils.
D Prompts Used
LIVE-SWE-AGENT uses minimally modified prompts built around a one-command bash interaction loop, with instructions enabling runtime creation of Python tools and feedback-guided refinement.
- D Prompts Used: The prompts are largely inherited from mini-SWE-agent, with minimal changes enabling on-the-fly tool creation and one SWE-Bench Pro directory-path adjustment.
- D.1 Initial prompt: The workflow covers repository analysis, issue reproduction, source editing, verification, and edge-case testing, while forbidding test and configuration-file modifications.
- D.1 Initial prompt: D.1 requires one command per response, formatted with a THOUGHT section and exactly one bash code block.
- D.1 Initial prompt: Multiple commands must be chained with && or ||, or issued sequentially after observing the previous command’s output.
- D.1 Initial prompt: The environment instructions emphasize non-interactive Linux commands, while examples cover file creation and numbered file viewing.
- D.1 Initial prompt: Agents are instructed to create informative Python tools, including a simple arbitrary-file editing tool and issue-specific tools when useful.
- D.1 Initial prompt: The tool-creation example writes an executable Python script containing custom tool logic.
- D.2 Feedback message: D.2 feeds return codes and outputs back to the agent and asks it to reflect on trajectories and consider useful tools for the current task.