PaperlayerView exact revision on arXiv

Source-linked AI summary

DeepAgent: A General Reasoning Agent with Scalable Toolsets

Xiaoxi Li, Wenxiang Jiao, Jiarui Jin, Guanting Dong, Jiajie Jin, Yinuo Wang, Hao Wang, Yutao Zhu, Ji-Rong Wen, Yuan Lu, Zhicheng Dou

arXiv:2510.21618v3cs.AIcs.CLcs.IRcs.LG

TL;DR

Real-world tasks require tools and long-horizon interaction, while existing agents are often constrained by predefined workflows and limited toolsets. DeepAgent unifies reasoning, dynamic tool discovery, execution, and structured memory folding, and trains tool use with ToolPO. Across eight benchmarks, it reports superior performance across labeled-tool and open-set scenarios.

  • Problem

    Existing agents often use predefined workflows or small fixed toolsets, limiting autonomous task completion, dynamic tool discovery, and coherent long-horizon reasoning.

  • Method

    DeepAgent unifies reasoning, dynamic tool discovery, action execution, and autonomous memory folding, while ToolPO uses simulated APIs and fine-grained tool-call credit assignment for end-to-end training.

  • Results

    DeepAgent consistently outperforms baseline agents across eight benchmarks spanning general tool use and downstream applications, including labeled-tool and open-set scenarios.

  • Takeaways & Limitations

    The results support DeepAgent as a general reasoning agent for scalable tool use and complex real-world tasks requiring dynamic tool discovery.

Abstract

from arXiv · show

Large reasoning models have demonstrated strong problem-solving abilities, yet real-world tasks often require external tools and long-horizon interactions. Existing agent frameworks typically follow predefined workflows, which limit autonomous and global task completion. In this paper, we introduce DeepAgent, an end-to-end deep reasoning agent that performs autonomous thinking, tool discovery, and action execution within a single, coherent reasoning process. To manage long-horizon interactions, we introduce an autonomous memory folding mechanism that compresses past interactions into structured episodic, working, and tool memories, reducing error accumulation while preserving critical information. To teach general-purpose tool use efficiently and stably, we develop an end-to-end reinforcement learning strategy, namely ToolPO, that leverages LLM-simulated APIs and applies tool-call advantage attribution to assign fine-grained credit to the tool invocation tokens. Extensive experiments on eight benchmarks, including general tool-use tasks (ToolBench, API-Bank, TMDB, Spotify, ToolHop) and downstream applications (ALFWorld, WebShop, GAIA, HLE), demonstrate that DeepAgent consistently outperforms baselines across both labeled-tool and open-set tool retrieval scenarios. The code and demo are available at https://github.com/RUC-NLPIR/DeepAgent.

1 Introduction

DeepAgent addresses limitations of workflow-based agents by unifying reasoning, dynamic tool discovery, action execution, and memory management in one coherent process. Its memory folding and ToolPO training support long-horizon tool use, and experiments across eight benchmarks report superior performance and adaptability.

  • Existing agents rely on predefined workflows, limiting execution autonomy, dynamic tool discovery, interactive memory management, and coherent task-level reasoning.
  • DeepAgent unifies autonomous thinking, as-needed tool retrieval, and action execution within a single coherent reasoning process.The agent maintains a global perspective instead of relying on predefined tools, planning, and isolated operation loops.
  • Autonomous Memory Folding consolidates reasoning and interaction history into structured episodic, working, and tool memories.The mechanism reduces token overhead and lets the agent reconsider strategies after unsuccessful or erroneous exploration paths.
  • ToolPO trains general-purpose tool use with LLM-simulated APIs and fine-grained advantage attribution for correct intermediate tool calls.It addresses training instability, slow execution, API costs, and sparse final-outcome rewards.
  • Experiments span eight benchmarks covering general tool-use tasks and downstream applications with domain-specific toolsets.The evaluation includes ToolBench, API-Bank, TMDB, Spotify, ToolHop, ALFWorld, WebShop, GAIA, and HLE.
  • DeepAgent demonstrates superior tool-use capabilities and high adaptability to real-world tasks across the evaluated scenarios.

2 Related Work

Prior work has advanced reasoning and tool-augmented agents, but many systems remain constrained by fixed workflows, limited toolsets, or one-time retrieval. DeepAgent targets dynamic discovery and invocation of helpful tools from scalable toolsets.

  • Large Reasoning Models: Large reasoning models improve complex problem solving through step-by-step slow thinking, but parametric knowledge alone cannot interact with the real world.
  • Large Reasoning Models: Existing tool-augmented reasoning approaches typically use small, fixed toolsets such as search, browsing, and coding, limiting generality.
  • Autonomous Agents: Current autonomous agents predominantly follow predefined workflows with fixed execution patterns, restricting advanced reasoning models’ autonomous decision-making.
  • Autonomous Agents: Most tool-invocation approaches perform a single upfront retrieval step, with limited exploration of dynamic tool discovery during task execution.
  • Autonomous Agents: DeepAgent aims to dynamically discover and invoke helpful tools from scalable toolsets for generalized real-world tasks.

3 Methodology

DeepAgent frames tool-using agents as sequential decision-makers whose reasoning, tool discovery, calls, and memory folding occur in one continuous process. Its framework combines structured memory, auxiliary-model support, and ToolPO training for general tool use.

  • Problem formulation: The agent interacts with an arbitrary-scale tool collection through internal thought, tool search, tool calls, and memory-fold actions.Its state records previous actions and observations, while memory folding replaces the interaction history with a compressed state.
  • Framework overview: DeepAgent’s main reasoning model autonomously reasons, discovers tools, executes actions, and manages memory in a single stream.An auxiliary LLM filters tool documentation, condenses tool outputs, and compresses long interaction histories.
  • Autonomous tool search and calling: Tool search retrieves top-k tools by cosine similarity between the query embedding and precomputed tool-documentation embeddings.Retrieved documentation is summarized when lengthy before being returned to the main reasoning model.
  • Autonomous memory folding and brain-inspired memory schema: Memory folding compresses the preceding interaction history into episodic, working, and tool memories that replace the raw history.The JSON schema is designed to preserve critical details through a controllable, predictable structure.
  • End-to-end RL training with ToolPO: ToolPO trains DeepAgent end-to-end using diverse data, an LLM-based tool simulator, and rewards for task success, tool calls, and efficient memory folding.The simulator provides a stable, efficient, and low-cost environment instead of requiring extensive real-world API interaction.
  • End-to-end RL training with ToolPO: Tool-call advantage attribution assigns action-level credit only to tokens corresponding to tool calls and memory-folding actions.This provides a targeted learning signal for correct and efficient tool use.

4 Experimental Settings

The experiments evaluate DeepAgent on general tool-use and downstream tasks spanning diverse, scalable toolsets and long-horizon interactions, using workflow-based and autonomous-tool-use baselines.

  • General Tool-Use: DeepAgent is evaluated on general tool-use benchmarks spanning tens to over 10k tools, testing planning, retrieval, and multi-step tool calling.The suite includes ToolBench, API-Bank, TMDB, Spotify, and ToolHop.
  • Downstream Applications: Downstream evaluations cover ALFWorld, WebShop, GAIA, and HLE with domain-specific tools and long-horizon interaction requirements.These environments require state tracking, error recovery, and coordination across heterogeneous tools.
  • Baselines: Baselines include workflow methods such as ReAct, CodeAct, Plan-and-Solve, Reflexion, and AgentLM, alongside autonomous tool-use systems such as WebThinker and HiRA.OpenAI Deep Research is also included among autonomous tool-use comparisons.
  • Implementation Details: ToolPO training uses 100 steps, batch size 64, λ1 = λ2 = 1, rollout size K = 8, and maximum sequence length 32,768.All experiments are conducted on 64 NVIDIA H20-141GB GPUs.

5 Experimental Results

DeepAgent outperforms workflow-based and other baselines across general tool-use and downstream benchmarks, with especially strong gains in open-set retrieval and long-horizon settings. Ablations and training analyses support contributions from ToolPO, memory folding, and dynamic tool discovery.

  • General Tool-Use: 89.0% on TMDB and 75.4% on Spotify exceed the best 32B baselines of 55.0% and 52.6%, respectively, on labeled-tool tasks.These results support end-to-end reasoning over rigid action loops.
  • General Tool-Use: 64.0% on ToolBench and 40.6% on ToolHop surpass the best baselines of 54.0% and 29.0%, respectively, in open-set settings.ToolPO further increases ToolBench success by up to 6.0% and labeled Spotify performance by 5.2%.
  • Downstream Applications: 53.3 on GAIA and 91.8% on ALFWorld are the best results among 32B models, exceeding HiRA’s 42.5 and the 84.3% baseline, respectively.On GAIA, ToolPO improves performance from 46.7 to 53.3 (+6.6); on ALFWorld, it improves 88.1% to 91.8% (+3.7).
  • Training Dynamics: ToolPO achieves higher reward and validation upper bounds than GRPO while producing less fluctuating training rewards.The analysis attributes stability to tool simulators and tool-call process supervision.
  • Ablation Studies: Removing ToolPO drops the ablation score from 48.1 to 44.3, while removing memory folding lowers the average to 44.2 and GAIA from 53.3 to 44.7.The GAIA result highlights the relevance of memory folding for long-horizon interaction.
  • Tool Retrieval: Autonomous retrieval outperforms pre-retrieved tools across frameworks, with DeepAgent scoring 52.6 on average versus 28.5 for the best workflow method.The largest gains occur on ToolBench’s 16k-tool and ToolHop’s 3.9k-tool settings.
  • Action Scaling: DeepAgent consistently outperforms ReAct across tested action limits on WebShop and GAIA, while performance generally improves as the maximum action count increases.The performance gap widens particularly on WebShop.
  • Backbone Scaling: With Qwen3-30B-A3B-Thinking and Qwen3-235B-A22B-Thinking backbones, DeepAgent maintains a significant margin over ReAct and Plan-and-Solve.The results indicate performance across both tested MoE reasoning-model backbones.

6 Conclusion

DeepAgent unifies thinking, tool discovery, and execution in one reasoning process, adding autonomous memory folding and ToolPO for long-horizon tool use. Across general and downstream applications, it outperforms baselines, particularly in open-set dynamic-retrieval scenarios.

  • Conclusion: DeepAgent unifies thinking, tool discovery, and execution within a single coherent agentic reasoning process.The design targets more general and scalable LLM agents.
  • Conclusion: Autonomous memory folding compresses interaction history into structured memory so the agent can reconsider its strategy during long-horizon interaction.ToolPO uses LLM-simulated APIs and fine-grained credit assignment for tool invocations.
  • Conclusion: Experiments show DeepAgent significantly outperforms baseline agents, especially in open-set scenarios requiring dynamic tool discovery over scalable toolsets.The conclusion frames these findings as support for broader real-world agent applications.

A.1 Training Data

The training dataset combines general tool use, real-world interaction, deep research, and mathematical reasoning examples to instill comprehensive agent capabilities.

  • Training Data: The dataset spans four task categories: general tool use, real-world interaction, deep research, and mathematical reasoning.This collection is intended to instill comprehensive agent capabilities.
  • General Tool-Use: General tool-use data includes 1k labeled-tool and 1k tool-retrieval instances sampled from ToolBench training data.The data targets diverse tool use and retrieval over large toolsets.
  • Real-World Interaction: Real-world interaction data includes 500 ALFWorld and 500 WebShop instances for state transitions and user-goal completion.Deep-research data includes 200 WebDancer and 500 WebShaperQA instances for web search and browsing.

A.2 Benchmarks

The benchmarks cover general tool use across toolsets ranging from tens to over ten thousand tools, alongside downstream applications requiring domain-specific tools. A ToolBench case study illustrates autonomous multi-tool selection, sequential execution, and result synthesis.

  • General Tool-Use: General tool-use benchmarks span toolsets from tens to over ten thousand distinct tools, testing approaches under different scaling conditions.The benchmark suite includes ToolBench, API-Bank, TMDB, Spotify, and ToolHop.
  • Downstream Applications: Downstream benchmarks evaluate complex real-world tasks requiring domain-specific toolsets across embodied interaction, shopping, web research, visual question answering, coding, and file reading.The suite includes ALFWorld, WebShop, GAIA, and HLE.
  • Case Study: Table 6 presents a ToolBench case study of a multi-step, multi-tool task, showing key actions in DeepAgent’s coordinated workflow.The example demonstrates autonomous tool selection, sequential actions, and synthesis of comprehensive answers.
  • Case Study: The case begins with searching for tools, then uses video search and tag-based retrieval before verifying a YouTube video identifier.The illustrated task combines documentary discovery, cinema-related person discovery, and YouTube-link construction.
  • Case Study: The final response consolidates documentary findings, cinema-related speakers with Vimeo profiles, and a validated YouTube streaming link.The YouTube search tool confirms that the video ID is valid before link construction.

B Implementation Details

The implementation uses specified reasoning and auxiliary language models, API-based web and browsing tools, embedding-based tool retrieval, standardized function documentation, and distributed ToolPO training.

  • Models: QwQ-32B is the main reasoning model for Tables 1 and 2, while Qwen3 models are used for Table 5.Qwen2.5-32B-Instruct serves auxiliary roles including filtering outputs, simulating RapidAPIs, and generating folded memories.
  • Tooling: Web search uses Google Serper, page browsing uses Jina Reader, visual question answering uses Qwen2.5-VL-32B-Instruct, and tool retrieval uses bge-large-en-v1.5.The VQA tool takes a question and image as input and returns a model-generated response.
  • Tool Documentation: Tool documentation follows the OpenAI function-definition format and is used to build the toolset index and populate agent prompts.The format specifies tool names, descriptions, parameters, property descriptions, and required arguments.
  • Training: Training runs 100 ToolPO steps with batch size 64, rollout size K=8, maximum sequence length 32,768, and at most 50 actions on 64 NVIDIA H20-141GB GPUs.The configuration sets λ1 = λ2 = 1 and uses VeRL for multi-node distributed training.

C Memory Schema

DeepAgent folds interaction history into a fixed-schema memory architecture with episodic, working, and tool components. These memories preserve task progress, maintain immediate continuity, and consolidate reusable tool-use experience for long-horizon reasoning.

  • Architecture: The memory architecture uses fixed JSON schemas for episodic, working, and tool memory to support stable folding and reduce information loss.The schemas enable reliable parsing and use of compressed memories during long-horizon reasoning.
  • Episodic Memory: Episodic memory records high-level task progression, including milestones, decisions, and outcomes, to preserve long-term context.Its schema summarizes the task and organizes key events by step.
  • Working Memory: Working memory captures the immediate goal, active challenges, and next actions to maintain continuity across folds.Its schema represents anticipated tool calls, planning, or decisions.
  • Tool Memory: Tool memory consolidates tool-use experience such as success rates, effective parameters, common errors, and response patterns into reusable rules.The design supports reflecting on both successful and failed tool interactions.
  • Operational Use: A case study demonstrates how the framework uses autonomous tool selection, sequential actions, and result synthesis for complex multi-step tasks.The example is presented as evidence of coordinated use of multiple tools.
Loading 2510.21618v3…