PaperlayerView exact revision on arXiv

Source-linked AI summary

MindSearch: Mimicking Human Minds Elicits Deep AI Searcher

Zehui Chen, Kuikun Liu, Qiuchen Wang, Jiangning Liu, Wenwei Zhang, Kai Chen, Feng Zhao

arXiv:2407.20183v2cs.CLcs.AI

TL;DR

Complex web questions are hard to answer because relevant evidence is dispersed across noisy pages and may exceed LLM context limits. MindSearch combines a graph-building planner with hierarchical retrieval agents; it improves response quality across closed- and open-set QA, and evaluators preferred its responses to existing search applications.

  • Problem

    Complex web questions require decomposition and integration of relevant information spread across noisy pages, which can exceed LLM context limits.

    Source: §1 · p. 2

  • Method

    MindSearch uses a WebPlanner to iteratively build a graph of sub-questions and WebSearchers to retrieve and summarize relevant web information hierarchically.

    Source: §1 · pp. 2–3

  • Results

    MindSearch improved response depth and breadth on closed- and open-set QA; its responses were preferred by human evaluators over ChatGPT-Web and Perplexity Pro.

    Source: §1 · p. 3, §5 · p. 10

  • Takeaways & Limitations

    MindSearch with open-source LLMs offers a competitive approach to AI-driven search engines.

    Source: §1 · p. 3

Abstract

from arXiv · show

Information seeking and integration is a complex cognitive task that consumes enormous time and effort. Inspired by the remarkable progress of Large Language Models, recent works attempt to solve this task by combining LLMs and search engines. However, these methods still obtain unsatisfying performance due to three challenges: (1) complex requests often cannot be accurately and completely retrieved by the search engine once (2) corresponding information to be integrated is spread over multiple web pages along with massive noise, and (3) a large number of web pages with long contents may quickly exceed the maximum context length of LLMs. Inspired by the cognitive process when humans solve these problems, we introduce MindSearch to mimic the human minds in web information seeking and integration, which can be instantiated by a simple yet effective LLM-based multi-agent framework. The WebPlanner models the human mind of multi-step information seeking as a dynamic graph construction process: it decomposes the user query into atomic sub-questions as nodes in the graph and progressively extends the graph based on the search result from WebSearcher. Tasked with each sub-question, WebSearcher performs hierarchical information retrieval with search engines and collects valuable information for WebPlanner. The multi-agent design of MindSearch enables the whole framework to seek and integrate information parallelly from larger-scale (e.g., more than 300) web pages in 3 minutes, which is worth 3 hours of human effort. MindSearch demonstrates significant improvement in the response quality in terms of depth and breadth, on both close-set and open-set QA problems. Besides, responses from MindSearch based on InternLM2.5-7B are preferable by humans to ChatGPT-Web and Perplexity.ai applications, which implies that MindSearch can already deliver a competitive solution to the proprietary AI search engine.

1 INTRODUCTION

MindSearch addresses the difficulty of integrating noisy, dispersed web information for complex queries with a multi-agent framework that decomposes questions and retrieves information hierarchically. Evaluations report improved response quality and human preference over existing search applications.

1 INTRODUCTION

  • Complex web information seeking is difficult because queries need decomposition, relevant evidence spans noisy pages, and retrieved content can exceed LLM context limits.Vanilla retrieval-augmented generation often engages too superficially with the depth and complexity of web retrieval.Source: §1 · p. 2
  • MindSearch pairs a WebPlanner that reasons through atomic sub-questions with WebSearchers that manage information seeking.The planner progressively constructs a graph of sequential or parallel sub-problems using code.Source: §1 · p. 2
  • WebSearcher uses hierarchical retrieval to extract valuable information efficiently from many search pages, while specialized agents divide reasoning and retrieval work.The division of work helps each agent handle long contexts more robustly.Source: §1 · p. 3
  • Evaluations on closed-set and open-set QA found improved response depth and breadth, with human evaluators preferring MindSearch responses to ChatGPT-Web and Perplexity Pro.The evaluations used GPT-4o and InternLM2.5-7B-Chat.Source: §1 · p. 3

2 MINDSEARCH

MindSearch divides complex web information seeking among a planner and specialized search agents. The planner constructs a reasoning graph, searchers retrieve and summarize relevant pages, and context is managed across agents.

2 MINDSEARCH

  • MindSearch coordinates a WebPlanner, which decomposes questions into sequential or parallel search tasks, with WebSearchers that retrieve information for sub-questions.The planner uses search feedback to determine subsequent steps.Source: §2 · p. 3

2.1 WEBPLANNER: PLANNING VIA GRAPH CONSTRUCTION

  • The WebPlanner represents a solution as a DAG of search nodes and reasoning edges, then writes executable code to add graph elements and invoke parallel searches.After collecting the search information, it adds an END node and produces the final response; code actions provide a unified way to construct the reasoning path.Source: §2.1 · p. 3, Figure 2 · p. 4

2.2 WEBSEARCHER: WEB BROWSING WITH HIERARCHICAL RETRIEVAL

  • WebSearcher rewrites each sub-query into similar queries, aggregates search results, selects pages for detailed reading, and summarizes answers from their full content.This coarse-to-fine pipeline extracts relevant information while reducing the burden of navigating massive web pages.Source: §2.2 · pp. 4–5, Figure 3 · p. 5

2.3 LLM CONTEXT MANAGEMENT IN MINDSEARCH

  • MindSearch manages long contexts by assigning decomposition and analysis to the planner and task-specific retrieval to searchers, transferring parent-node and root responses as context.The framework reports integrating information from more than 300 pages in less than 3 minutes, a workload estimated to take human experts about 3 hours.Source: §2.3 · p. 5

3 EXPERIMENTS

Experiments compare MindSearch on closed-set and open-set QA, using human judgments and benchmark tasks. Results show gains over vanilla baselines and competing search systems, while ablations examine planning, retrieval, and inference-time scaling.

3.1.2 RESULTS AND ANALYSIS

  • Human evaluators found absolute depth and breadth improvements on open-set QA, while facticity improved less than breadth (70% vs 83%).The authors attribute the responses’ detail to finer-grained search topics and identify hallucination mitigation as future work.Source: §3.1.1 · pp. 5–6, §3.1.2 · p. 6, Figure 4 · p. 6
  • On the same question, MindSearch produced more concrete, detailed responses than Perplexity.ai Pro, which the authors attribute to its planning search paths.Figure 5 provides a qualitative trajectory comparison.Source: Figure 5 · p. 6, §3.1.2 · p. 7

3.2.1 IMPLEMENTATION DETAILS

  • The evaluation spans closed-set and open-set QA, with GPT-4o and InternLM2.5-7b-chat as backends and BING as the sole search API.For open-set QA, five experts selected preferred responses from 100 real-world queries across depth, breadth, and factuality.Source: §3 · p. 5, §3.1.1 · pp. 5–6, §3.2.1 · p. 7

3.2.2 RESULTS AND ANALYSIS

  • MindSearch outperforms its vanilla baselines by 4.7% on GPT-4o and 6.3% on InternLM2-7b, with most HotpotQA gains on the hard set.The baselines are raw LLM without search and ReAct Search; Table 1 compares performance on closed-set QA tasks.Source: §3.2.2 · p. 7, Table 1 · p. 7

3.3.1 ABLATIONS ON WEBPLANNER

  • With WebSearcher held constant, code generation improves over ReAct, and graph construction further improves planning performance, especially on HotpotQA’s hard set.The study compares ReAct, CodeAct, and graph construction as WebPlanner strategies.Source: §3.3.1 · pp. 7–8, Table 2 · p. 8

3.3.2 ABLATIONS ON WEBSEARCHER

  • Each tested WebSearcher component contributes to performance, with multi-query generation and aggregation having the greatest influence.The ablation removes multi-query generation and aggregation, prefixing previous context, and page selection.Source: §3.3.2 · p. 8, Table 3 · p. 8

3.4 INFERENCE TIME SCALING VS. SEARCH PERFORMANCE

  • MindSearch scales search performance more efficiently than ReAct, requiring fewer inference steps for improvement, while single-turn search uses the least time.The comparison covers single-turn search, multi-turn ReAct, and multi-turn MindSearch on HotpotQA.Source: §3.4 · pp. 8–9, Figure 6 · p. 8

4 RELATED WORK

Related work covers tool integration and retrieval-augmented generation for LLMs, alongside web agents that have moved from QA toward more dynamic web interactions. MindSearch focuses on web information seeking and integration rather than web browsing.

4.1 TOOL UTILIZATION WITH LLM

  • Tool-learning research integrates LLMs with tools such as search engines, databases, and APIs, while improving retrieval and the use of retrieved information.Related work also connects tool integration with interpretability, trustworthiness, robustness, adaptability, and applications including question answering.Source: §4.1 · p. 9

4.2 RAG WITH LLM

  • RAG combines LLMs with retrieval for knowledge-intensive tasks, with prior work improving either retrieval or the language model’s reading and comprehension.The cited applications include open-domain question answering, hallucination reduction, and code generation.Source: §4.2 · p. 9

4.3 WEB AGENTS

  • Web agents have advanced from QA tools to dynamic web interactions, while MindSearch focuses on search-based information seeking and integration rather than browsing.The related-work discussion describes navigation and automation systems, including approaches using reinforcement learning and behavior cloning.Source: §4.3 · pp. 9–10

5 CONCLUSION AND FUTURE WORK

MindSearch combines iterative graph-based query decomposition with hierarchical retrieval by specialized agents to address complex web information-seeking and integration. Evaluations show response-quality advantages, while citation quality, visual inputs, and web-page interaction remain limited or unsupported.

5 CONCLUSION AND FUTURE WORK

  • MindSearch models problem solving as iterative graph construction, pairing effective query decomposition with hierarchical retrieval to improve precision and recall.Its multi-agent design distributes cognitive load to support lengthy contexts.Source: §5 · p. 10
  • Evaluations on closed-set and open-set QA with GPT-4o and InternLM2.5-7B show significant response-quality advantages.Human evaluators preferred MindSearch responses to ChatGPT-Web and Perplexity.ai.Source: §5 · p. 10
  • Citation quality of web-search references was not comprehensively evaluated.The paper cites the diversity and subjectivity of AI web-search evaluation compared with closed-set QA reference evaluation.Source: §5 · p. 10
  • MindSearch does not support visual inputs or interaction with web pages, which the paper identifies as future-work scenarios.Source: §5 · p. 10

A COMPARISON WITH OTHER STATE-OF-THE-ARTS

On HotpotQA, MindSearch outperforms the compared web-search approaches, while its multi-agent design is contrasted with Searchain's repeated regeneration of its reasoning chain. On 10 complex research questions, MindSearch completed the work much faster than human labelers.

A COMPARISON WITH OTHER STATE-OF-THE-ARTS

  • Searchain regenerates its full reasoning chain at each revised step, whereas MindSearch can reason immediately when needed.The passage describes Searchain's approach as time-consuming and fallible due to weaker long-context reasoning.Source: A COMPARISON WITH OTHER… · p. 15
  • MindSearch consistently outperforms self-ask, CodeAct, and Searchain by a large margin on HotpotQA.The comparison uses InternLM-7b-2.5; Table 4 presents the results.Source: Table 4 · p. 15, A COMPARISON WITH OTHER… · p. 15

B TIME EFFICIENCY ON MINDSEARCH VS. HUMAN LABELERS

  • 19h17min for human labelers versus 23 min for MindSearch on 10 complex research questions.One human labeler spent 47 minutes collecting information and 74 minutes writing a detailed answer for one question.Source: B TIME EFFICIENCY ON MINDSEARCH VS.… · p. 15

C.1 NUMBER OF HOPS VS DEPTH OF DAG

On Musique, DAG depth increases monotonically with question hop count but is not identical to it. Parallel execution and shortcuts or simplifications can produce shorter search paths.

C.1 NUMBER OF HOPS VS DEPTH OF DAG

  • DAG depth increases monotonically with question hop count on Musique, but the two measures are not identical.For example, questions with 3 hops have a reported DAG depth of 1.2.Source: §C.1 · p. 15, Table 5 · p. 15
  • Parallel execution can resolve multiple questions in one step, increasing DAG depth by only one.Source: §C.1 · p. 15
  • Shortcuts or simplifications in questions that claim two or more hops can shorten the search path.Source: §C.1 · p. 15

C.2 COST ANALYSIS ON THE NUMBER OF SEARCH QUERIES

MindSearch generates fewer search queries on average than ReAct, which the paper attributes to more accurate query formulation. It also adapts to several other language models with little adaptation.

C.2 COST ANALYSIS ON THE NUMBER OF SEARCH QUERIES

  • MindSearch generates 0.3 fewer queries on average than ReAct (3.2 vs 3.5).Source: §C.2 · p. 16
  • ReAct's weaker question decomposition leads to repeated keyword searches, while MindSearch uses LLM reasoning to form more accurate queries.The paper characterizes the repeated searches as useless and inefficient.Source: §C.2 · p. 16

D GENERALIZATION TO OTHER LANGUAGE MODELS

E ANALYSIS ON ERROR CORRECTION IN MINDSEARCH

MindSearch’s error-correction analysis examines how it recovers from mistakes, focusing on validation and feedback from code execution.

E ANALYSIS ON ERROR CORRECTION IN MINDSEARCH

E.1 ERROR CORRECTION WITH CODE EXECUTION FEEDBACK

  • Code execution provides a validation mechanism: Python interpreter exceptions can be returned to the LLM when generated code or a planning sentence is invalid.Source: §E.1 · p. 16
  • After a node-execution mistake, MindSearch uses the executor’s error message to regenerate and correct its code.Source: §E.1 · p. 16

E.2 ERROR CORRECTION WHEN WEBSEARCHER CANNOT FOUND RELATED INFORMATION

When search feedback indicates information was not found, MindSearch can revise its query or conclude after repeated unsuccessful attempts.

E.2 ERROR CORRECTION WHEN WEBSEARCHER CANNOT FOUND RELATED INFORMATION

  • To manage varied multi-agent replies, each WebSearcher response is limited to relevant information or a not-found result.Source: §E.2 · p. 16
  • After not-found feedback, MindSearch regenerates a query and retrieves information about UEFA Euro 2020 corner kicks from Wikipedia.Source: §E.2 · p. 16, Figure 8 · p. 17
  • After several failed attempts, MindSearch can add a response node and stop searching rather than enter a repetitive loop.Source: §E.2 · p. 17

F DETAILS ON OPEN-SET EVALUATION PHASE

The appendix describes open-set human evaluation and gives details of the prompts and graph API used by MindSearch.

F DETAILS ON OPEN-SET EVALUATION PHASE

G SYSTEM PROMPT

  • The WebSearchGraph API manages nodes and adjacency-list connections, with methods for adding a root question and sub-question nodes.Source: G SYSTEM PROMPT · p. 22
  • The WebPlanner prompt directs the model to build a search graph from focused, single-question nodes, allowing unrelated searches to run concurrently.Source: G SYSTEM PROMPT · p. 23
  • The WebSearcher prompt specifies tool-call formatting and requires source-marked, detailed replies based on the current problem’s search results.Source: G SYSTEM PROMPT · p. 23
Loading 2407.20183v2…