Source-linked AI summary
EvoFSM: Controllable Self-Evolution for Deep Research with Finite State Machines
Shuo Zhang, Chaofa Yuan, Ryan Guo, Xiaomin Yu, Rui Xu, Zhangquan Chen, Zinuo Li, Zhi Yang, Shuhao Guan, Zhenheng Tang, Sen Hu, Liwen Zhang, Ronghao Chen, Huacan Wang
TL;DR
Deep research agents need to adapt beyond fixed workflows, while unconstrained self-evolution risks instability and instruction drift. EvoFSM evolves an explicit FSM through separated Flow and Skill operations guided by a critic, and transfers experience through self-evolving memory. The framework is evaluated on multi-hop QA and interactive decision-making tasks, with 58.0% accuracy on DeepSearch.
Problem
Fixed workflows struggle with open-ended query paths, while unconstrained self-evolution introduces stability risks such as hallucination and instruction drift.
Method
EvoFSM models research as an explicit FSM, separates Flow and Skill optimization, applies critic-guided atomic operations, and stores successful and failed trajectories for reuse.
Results
EvoFSM is evaluated on five multi-hop QA benchmarks and interactive decision-making tasks, reaching 58.0% accuracy on DeepSearch.
Takeaways & Limitations
EvoFSM provides structured, controllable adaptation while using accumulated experience to guide future task evolution.
Takeaways & Limitations
The framework relies on off-the-shelf proprietary LLMs without fine-tuning, limiting efficiency and responsiveness for complex FSM logic.
Abstract
from arXiv · showhide
While LLM-based agents have shown promise for deep research, most existing approaches rely on fixed workflows that struggle to adapt to real-world, open-ended queries. Recent work therefore explores self-evolution by allowing agents to rewrite their own code or prompts to improve problem-solving ability, but unconstrained optimization often triggers instability, hallucinations, and instruction drift. We propose EvoFSM, a structured self-evolving framework that achieves both adaptability and control by evolving an explicit Finite State Machine (FSM) instead of relying on free-form rewriting. EvoFSM decouples the optimization space into macroscopic Flow (state-transition logic) and microscopic Skill (state-specific behaviors), enabling targeted improvements under clear behavioral boundaries. Guided by a critic mechanism, EvoFSM refines the FSM through a small set of constrained operations, and further incorporates a self-evolving memory that distills successful trajectories as reusable priors and failure patterns as constraints for future queries. Extensive evaluations on five multi-hop QA benchmarks demonstrate the effectiveness of EvoFSM. In particular, EvoFSM reaches 58.0% accuracy on the DeepSearch benchmark. Additional results on interactive decision-making tasks further validate its generalization.
1 Introduction
EvoFSM addresses the rigidity of static research workflows and the instability of unconstrained self-evolution by structuring adaptation around an explicit FSM. It separates Flow and Skill optimization, adds memory across tasks, and evaluates generality across research settings.
- Static toolcall-generation-reflection pipelines struggle to adapt to dynamic query paths in open-ended research problems.
- Unconstrained rewriting can cause instruction drift and hallucination-related stability problems.
- Flow optimization targets macroscopic transition logic, while Skill optimization targets microscopic node-specific capabilities.
- EvoFSM models retrieval and reasoning as an explicit FSM with clear transition logic and deterministic behavioral boundaries.
- Its self-evolving memory stores successful strategies as priors and failure patterns as constraints for future queries.
- Evaluations cover five multi-hop QA benchmarks and two interactive decision-making datasets, supporting effectiveness and generality across settings.
2 Related Work
Related work develops deep research agents for long-horizon information seeking, web interaction, software engineering, and self-evolution. These systems use retrieval, browsing, reinforcement learning, tool synthesis, scaffolding updates, and multi-agent adaptation across varied settings.
- Search-o1 combines agentic RAG with Reason-in-Documents for on-demand knowledge acquisition during long-horizon reasoning.
- Search-R1 trains LLMs with reinforcement learning to generate search queries and interact with search engines over multiple turns.
- WebThinker supports autonomous browsing and report drafting, while WebAgent-R1 applies end-to-end reinforcement learning to dynamic web decision-making.
- RepoMaster extends agentic exploration to software engineering through repository structure graphs.
- Self-evolving agents update tools, scaffolding, defenses, profiles, or decentralized multi-agent behavior during deployment.
3 Methodology
EvoFSM represents deep research as a dynamic FSM and evolves it through critic-guided, atomic Flow and Skill operations. An experience pool transfers successful strategies and failure constraints across tasks.
- EvoFSM comprises FSM Initialization, Structured Self-Evolution, and a Self-Evolving Memory Mechanism.
- FSM Initialization: The FSM uses nodes for specialized action phases and edges for logical transitions, providing a deterministic yet dynamic behavioral graph.
- FSM Initialization: The transition function T routes the next state from the current state and runtime context, defining macroscopic Flow logic.
- FSM Initialization: Node-specific prompts define the microscopic Skill dimension by specifying each agent’s operational guidelines and expertise.
- FSM Initialization: The Critic Mechanism evaluates final outputs against user queries and triggers evolution by identifying specific failure modes.
- FSM Initialization: Initialization retrieves top-k historical strategies from the experience pool to construct an initial FSM configuration for a new query.
- Structured Self-Evolution: Flow Operators edit states or transitions, including adding, deleting, or modifying workflow elements without changing node instructions.
- Structured Self-Evolution: Skill Operators revise instructions for individual states while preserving the global topology.
4 Experiments
EvoFSM is evaluated against retrieval baselines and ablations on multi-hop QA, then tested for transfer to interactive decision-making tasks. Results indicate that structured evolution and FSM topology jointly improve accuracy, robustness, and cross-domain performance, with a modest reasoning-step trade-off.
- Main Results: Iterative retrieval-and-reasoning frameworks outperform Standard RAG across all benchmarks, while EvoFSM structures the loop into explicit phases and transitions.Under GPT-4o, Agentic RAG gains 15.0% absolute on DeepSearch over the single-shot baseline; EvoFSM is described as improving evidence accumulation reliability.
- Main Results: EvoFSM’s relative improvements remain largely stable across the evaluated backbone models.The authors attribute this robustness primarily to workflow and optimization mechanisms rather than model-specific quirks.
- Main Results: EvoFSM consistently outperforms Search-o1 across all five LLMs on DeepSearch, including gains of 11.0% with Claude-4, 10.0% with GPT-4o, and 4.0% with Llama-3-70B.The framework initializes queries with relevant historical strategies and refines the workflow to each instance under the same tool budget.
- Ablation Study: 15.0% accuracy is lost on DeepSearch when structured self-evolution and memory are removed, falling from 51.0% to 36.0%.The static FSM cannot dynamically apply flow and skill operations or recall successful priors, limiting adaptation to unforeseen query bottlenecks.
- Ablation Study: 9.0% performance degradation occurs on DeepSearch without FSM topology, while removing both topology and self-evolution produces a 17.0% drop.The authors report redundant looping and loss of the research objective without explicit flow logic, and a compounding benefit when structure and evolution are combined.
- Further Analysis: EvoFSM consistently outperforms ReAct and Reflexion on ALFWorld and WebShop, especially WebShop, while using slightly more reasoning steps than ReAct.The extra steps are allocated to validation and disciplined state transitions, trading modest efficiency for higher task success rates.
5 Conclusion
EvoFSM addresses static workflows and unconstrained self-evolving agents by structuring research as a dynamic FSM with controllable Flow and Skill evolution. Experiments report strong performance across multi-hop QA and interactive decision-making tasks.
- EvoFSM models research as a dynamic Finite State Machine that decouples optimization into macroscopic Flow and microscopic Skill.Its atomic operations target workflow transitions or node-specific capabilities rather than performing unstructured global rewriting.
- EvoFSM uses precise atomic operations and self-evolving memory to keep adaptation targeted and transfer successful exploration strategies across tasks.The memory also supports continuous learning by distilling experience for reuse.
- EvoFSM significantly outperforms strong baselines on five multi-hop QA benchmarks and exhibits superior generalization in interactive decision-making environments.
Limitations
The framework remains constrained by its reliance on off-the-shelf proprietary LLMs, its dependence on an LLM-based Critic, and an experience pool that grows without consolidation or forgetting.
- EvoFSM relies entirely on off-the-shelf proprietary LLMs through prompt engineering and in-context learning, without fine-tuning or specialized training.General-purpose models may struggle to internalize complex FSM logic without explicit weight updates, limiting efficiency and responsiveness.
- The self-evolving process depends on the Critic accurately diagnosing failures without external ground truth during deployment.Critic hallucinations or missed logical errors may cause incorrect patterns to be learned or prevent effective evolution.
- The self-evolving memory faces scalability problems because its experience pool currently grows indefinitely without consolidation or forgetting.
A Illustrative Examples
The illustrative cases show EvoFSM adapting either the workflow topology, a node’s instructions, or both, depending on the failure mode. Structural deadlocks call for Flow changes, while information loss calls for Skill refinement.
- Flow Evolution via ADD_STATE: Flow Evolution addresses a failed Three Gorges Dam research execution by injecting a new Verifier state to intervene structurally.The example presents ADD_STATE as the relevant operation for changing the reasoning path.
- Instruction Evolution: Instruction Evolution refines the Browse node’s system prompt when information loss reduces precision, leaving the overall workflow unchanged.This targets node-specific behavior rather than the FSM topology.
- Synergistic Evolution (Flow + Skill): Synergistic Evolution combines topology reconfiguration with sharper search expertise for EU AI Act analysis.The case adds a Verifier and improves individual search skill to move beyond superficial news summarization toward legal analysis.