Source-linked AI summary
Agentic Reasoning: A Streamlined Framework for Enhancing LLM Reasoning with Agentic Tools
Junde Wu, Jiayuan Zhu, Yuyuan Liu, Min Xu, Yueming Jin
TL;DR
Knowledge-intensive tasks require retrieval, computation, verification, and organization beyond formal reasoning alone. Agentic Reasoning integrates specialized agents and structured memory, achieving strong expert-level and deep-research performance while incurring computational and reliability constraints.
Problem
Formal reasoning methods remain difficult to apply to less structured, knowledge-intensive tasks requiring factual verification, complex relationships, or moral reasoning.
Method
Agentic Reasoning integrates web-search, coding, and Mind-Map agents into LLM reasoning, using a knowledge graph to store and organize reasoning context.
Results
23.8% accuracy on Humanity’s Last Exam improved 14.4% over the raw model, while the approach achieved strong results across expert-level QA and real-world research tasks.
Takeaways & Limitations
External tools and structured memory support problem solving and deep research across expert-level questions and real-world tasks.
Takeaways & Limitations
Sequential invocation of multiple external agents increases computational costs and inference latency, limiting real-time applicability.
Abstract
from arXiv · showhide
We introduce Agentic Reasoning, a framework that enhances large language model (LLM) reasoning by integrating external tool-using agents. Agentic Reasoning dynamically leverages web search, code execution, and structured memory to address complex problems requiring deep research. A key innovation in our framework is the Mind-Map agent, which constructs a structured knowledge graph to store reasoning context and track logical relationships, ensuring coherence in long reasoning chains with extensive tool usage. Additionally, we conduct a comprehensive exploration of the Web-Search agent, leading to a highly effective search mechanism that surpasses all prior approaches. When deployed on DeepSeek-R1, our method achieves a new state-of-the-art (SOTA) among public models and delivers performance comparable to OpenAI Deep Research, the leading proprietary model in this domain. Extensive ablation studies validate the optimal selection of agentic tools and confirm the effectiveness of our Mind-Map and Web-Search agents in enhancing LLM reasoning. The code is at: https://github.com/theworldofagents/Agentic-Reasoning
1 Introduction
Agentic Reasoning addresses the difficulty of extending formal reasoning methods to knowledge-intensive tasks by integrating external LLM-based agents. It combines web search, coding, and Mind-Map memory, achieving strong results on expert-level questions and deep research tasks.
- Motivation: Reasoning methods that work well for math and code remain challenging to apply to less structured, knowledge-intensive domains.Such domains require factual verification, abstract concepts, complex logical relationships, and moral reasoning.
- Motivation: Deep answers to open-ended questions require research, verification, retrieval, computation, and organization of complex logical relationships.The paper asks whether reasoning LLMs can use external tools similarly to humans.
- Results: 23.8% accuracy on Humanity’s Last Exam improved 14.4% over the raw model and narrowed the gap to OpenAI Deep Research to 2.8%.Domain experts also found the system effective for automating complex manual investigation in real-world expert-level tasks.
- Framework: Agentic Reasoning integrates external LLM-based agents into reasoning and identifies web-search, coding, and Mind-Map agents as broadly effective tools.The framework delegates specialized tasks to auxiliary agents during problem solving.
- Contributions: The Web-Search agent uses a strategy reported to outperform previous search or RAG approaches, while Mind-Map organizes logical relationships in long reasoning chains.Mind-Map is knowledge-graph-based and supports reasoning coherence.
- Results: The approach achieves new SOTA results across several benchmarks and surpasses prior methods in human evaluations.The evaluation covers expert-level problem solving and deep research tasks.
2 Method
Agentic Reasoning dynamically invokes web-search and coding agents while using a knowledge-graph Mind-Map to preserve reasoning context. The agents retrieve information, perform computations, and support coherent long-chain reasoning.
- Agentic Reasoning Pipeline: The reasoning LLM dynamically calls WebSearch, Code, and Mind-Map agents as external tools during problem solving.The first two provide problem-solving assistance, while Mind-Map serves as structured memory.
- Agentic Reasoning Pipeline: Agent calls temporarily halt reasoning, dispatch queries with reasoning context to external agents, and reintegrate returned results into the reasoning chain.This enables iterative inference with updated knowledge.
- Mind-Map: Mind-Map transforms raw reasoning chains into a structured knowledge graph by extracting entities and semantic relationships.The construction process follows a procedure similar to GraphRAG.
- Mind-Map: Mind-Map clusters and summarizes reasoning context, and supports queries that retrieve relevant information from the knowledge graph.These functions provide context to external tools and memory for continued reasoning.
- Mind-Map: Mind-Map helps maintain long reasoning chains across multiple breakdown tasks and tool calls without missing critical information.It is queried when the model is uncertain or loses track during extended reasoning.
- Web-Search Agent: The Web-Search agent combines query breakdown, search, reranking, and RAG, iterating when top-page relevance falls below a threshold.It synthesizes retrieved responses into a snippet integrated into the main reasoning process.
- Coding Agent: The Coding agent delegates code writing and execution to a specialized coding LLM, returning natural-language results to preserve the reasoning model’s focus.This is intended to support longer and more coherent reasoning chains.
3 Experiments
The experiments evaluate Agentic Reasoning across expert-level question answering, deep research, and tool ablations. Results show strong performance from the web-search, coding, and Mind-Map combination, with Mind-Map especially helping long and logic-intensive reasoning.
- Evaluation Setup: The evaluation covers Humanity’s Last Exam, GPQA, GAIA, FreshWiki deep research, and 56 expert-authored open research questions reviewed by PhD-level experts.FreshWiki articles are assessed using ROUGE and entity recall, while open research articles are judged for interest, organization, relevance, and coverage.
- Ablation on Toolbox: Tool quality matters more than tool quantity: adding seven-tool or 109-tool external toolboxes can degrade GPQA performance through inappropriate tool selection.Some capabilities, including translation and code interpretation, are already embedded in the reasoning model, while inaccurate external outputs can also reduce response quality.
- Ablation on Toolbox: Web search performs best alone, coding and Mind-Map are comparable, and combining web search with either tool produces synergistic gains; all three perform best together.The ablation compares individual tools and combinations, finding that the three-tool configuration achieves the strongest performance.
- Ablation on Web-Search agent Design: Query Breakdown, Rerank, and Mind-Map Reasoning Context improve performance incrementally, whereas Knowledge Refinement becomes ineffective because it is redundant with Rerank.The final Web-Search agent therefore includes RAG, Query Breakdown, Rerank, and Mind-Map Reasoning Context.
- The Effect of Mind-Map: Mind-Map significantly improves extended reasoning by preserving prior steps, maintaining coherence across tool calls, and clarifying complex logical relationships.It is particularly effective for questions requiring long reasoning chains, many tool calls, or logic-based deductions.
4 Related Work
Related work has explored multi-agent frameworks and search augmentation, but existing systems often lack optimized delegation or remain proprietary. Agentic Reasoning combines structured tool integration with search, coding, and Mind-Map support.
- Multi-agent Collaboration: Frameworks such as Auto-GPT and LangChain Agents connect LLMs to external APIs, search engines, and computational environments but often lack optimized task delegation and structured integration.Related work also includes hierarchical planning and task-specific agents for structured cooperation.
- Agentic Reasoning: Agentic Reasoning differs by integrating Mind-Map, web search, and coding as coordinated tools for reasoning and deep research.The framework addresses structured integration rather than relying only on retrieval or web search.
- Search Augmentation: Agentic RAG systems allow models to determine when and what knowledge to retrieve, extending earlier work on search integration during reasoning.Examples include approaches involving QwQ and Web-Search agents.
5 Conclusion
The paper concludes that Agentic Reasoning enhances LLM reasoning through Mind-Map, web search, and coding. It reports gains over existing methods on quantitative benchmarks and human evaluations, with future work targeting task-specific tools and test-time computing.
- Contributions: Agentic Reasoning integrates Mind-Map, web search, and coding to improve problem-solving and deep research capabilities.The conclusion presents these three components as the framework’s core tools.
- Results: The approach outperforms existing models in expert-level question answering and real-world research tasks.The reported gains span both quantitative benchmarks and human evaluations.
- Future Work: Future work will explore task-specific tool integration and test-time computing to further enhance AI reasoning capabilities.
6 Limitations
The framework’s strong performance is accompanied by limitations involving efficiency, external knowledge quality, and trustworthiness in LLM-driven reasoning.
- Computational Overhead and Efficiency: Sequential invocation of web search, Mind-Map, and code agents increases computational costs and inference latency.These bottlenecks limit real-time applicability.
- Reliance on External Knowledge Sources: Agentic Reasoning depends on retrieved knowledge quality and lacks built-in source-credibility verification.This leaves the system susceptible to misinformation or biased content.
- Interpretability and Trustworthiness: The overall decision process remains highly dependent on LLMs despite the Mind-Map’s structured reasoning memory.This reliance introduces hallucination risks in complex, multi-step tasks.
7 Appendix
The appendix asks evaluators to rate each model-generated response on a five-point scale and optionally explain their judgments.
- Human Evaluation Survey: Evaluators rate each generated response on a scale from 1 to 5.A score of 1 is lowest and 5 is highest.
- Human Evaluation Survey: The survey requests an assessment of each response according to specified evaluation criteria.
- Human Evaluation Survey: Evaluators may leave optional comments to clarify the reasoning behind their ratings.
1. Interest Level (Int.)
The interest-level evaluation measures how engaging and intellectually stimulating each generated response is using a five-point rating scale and optional comments.
- 1. Interest Level (Int.): Interest Level (Int.) evaluates how engaging and intellectually stimulating the generated response is.
- 1. Interest Level (Int.): Ratings range from 1 for not engaging to 5 for highly engaging and thought-provoking.
- 1. Interest Level (Int.): Optional comments identify aspects that contributed to or detracted from the response’s interest level.
2. Organization (Org.)
The organization evaluation measures how clearly and logically a response is structured on a five-point scale, with comments identifying areas for improvement.
- 2. Organization (Org.): Organization (Org.) evaluates how well-structured and logically organized the response is.
- 2. Organization (Org.): Ratings range from 1 for very disorganized to 5 for exceptionally structured and logically ordered.
- 2. Organization (Org.): Optional comments ask evaluators to identify areas where the response could be better structured.
3. Relevance (Rel.)
Relevance measures how well a response addresses the research question, using a five-point scale from off-topic to fully on-topic.
- Relevance is rated from 1 to 5, with 1 indicating the lowest and 5 the highest relevance.
- A rating of 1 denotes an off-topic or misleading response.
- A rating of 5 denotes a response that fully addresses the research question.
4. Coverage (Cov.)
The evaluation compares deep-research performance and efficiency across proprietary systems, memory strategies, and agentic versus direct tool calls. Results indicate that Mind-Map achieves the strongest memory-strategy performance, while the agentic system trades some performance against GPT-Deep Research for substantially faster execution.
- Ablation Study on Memory Strategies: Mind-Map consistently achieves the highest performance across all evaluated memory-strategy settings on GAIA.
- Efficiency Analysis: 56 deep research questions were used to compare performance and time consumed across proprietary research systems.
- Efficiency Analysis: Agentic Reasoning outperforms Perplexity-Deep Research while being faster, matches Gemini-Deep Research with comparable latency, and trails GPT-Deep Research while running significantly faster.
- Agentic Tool Calls: Agentic tool calls are designed to overcome single-model token limits by enabling longer reasoning chains than one model call permits.
- Agentic Tool Calls: Agentic tool calls support uncertainty management because low-confidence outputs can be treated as tentative, reducing cascading errors across components.
- Task-Specific Model Modularity: Different agents can use different language models for coding and summarization, matching models to subtasks to improve performance and resource efficiency.