PaperlayerView exact revision on arXiv

Source-linked AI summary

WebWalker: Benchmarking LLMs in Web Traversal

Jialong Wu, Wenbiao Yin, Yong Jiang, Zhenglin Wang, Zekun Xi, Runnan Fang, Linhai Zhang, Yulan He, Deyu Zhou, Pengjun Xie, Fei Huang

arXiv:2501.07572v3cs.CLcs.AI

TL;DR

Traditional search may not trace deeper website content, a gap for LLMs handling complex, multi-step information seeking. This paper introduces WebWalkerQA and an explorer-critic WebWalker framework; experiments show the benchmark is challenging and that combining RAG with WebWalker improves web navigation.

  • Problem

    Traditional search engines perform horizontal searches and may not trace deeper content embedded within websites.

    Source: §1 · p. 1

  • Method

    WebWalkerQA evaluates LLMs on multi-step traversal of a root website, while WebWalker uses explorer and critic agents to navigate pages and update memory.

    Source: §1 · p. 1, §4 · p. 5

  • Results

    Experiments show WebWalkerQA challenges RAG systems, while combining RAG with WebWalker improves web navigation performance.

    Source: §7 · p. 10

  • Takeaways & Limitations

    Deep, vertical exploration is important for web-based tasks integrating information retrieval with RAG.

    Source: §7 · p. 10

    The supplied discussion describes WebWalker operating on a given webpage or integrated with RAG, where click actions are reported effective.

    Source: A Limitations and Discussion · p. 16

Abstract

from arXiv · show

Retrieval-augmented generation (RAG) demonstrates remarkable performance across tasks in open-domain question-answering. However, traditional search engines may retrieve shallow content, limiting the ability of LLMs to handle complex, multi-layered information. To address it, we introduce WebWalkerQA, a benchmark designed to assess the ability of LLMs to perform web traversal. It evaluates the capacity of LLMs to traverse a website's subpages to extract high-quality data systematically. We propose WebWalker, which is a multi-agent framework that mimics human-like web navigation through an explore-critic paradigm. Extensive experimental results show that WebWalkerQA is challenging and demonstrates the effectiveness of RAG combined with WebWalker, through the horizontal and vertical integration in real-world scenarios.

1 Introduction

WebWalkerQA evaluates whether LLMs can retrieve information through multi-step traversal of a website, addressing the limits of shallow search. The paper introduces WebWalker, a multi-agent navigation framework, and reports that combining it with RAG is effective while traversal remains challenging.

Motivation

  • Traditional search may miss deeper website content, while action-based benchmarks such as Mind2Web (Deng et al., 2023) and WebArena (Zhou et al., 2024a) face noisy inputs and do not capture multi-layer information seeking.The cited benchmarks use HTML-based instruction-action tasks that can strain long-context understanding.Source: §1 · p. 1

WebWalkerQA

  • WebWalkerQA defines web traversal as systematically exploring pages from a query-linked root website and evaluates text-based reasoning through question answering and click actions.The benchmark targets complex, multi-step interactions and navigation and information-seeking capabilities.Source: §1 · p. 1

WebWalker

  • WebWalker uses explorer and critic agents for vertical web exploration, with a ReAct-based explorer and a critic that maintains memory and generates responses.The explorer follows a thought-action-observation paradigm.Source: §1 · p. 2

Results

  • Even with powerful LLM backbones, WebWalker remains suboptimal on WebWalkerQA; combining RAG with WebWalker is effective, and vertical exploration is promising for scaling inference.The introduction also notes that navigation still requires planning and reasoning on information-seeking tasks.Source: §1 · p. 2

WebWalkerQA

  • WebWalkerQA contains 680 queries across four real-world scenarios and more than 1,373 webpages, with single-source and multi-source questions spanning education, conferences, organizations, and games.It emphasizes deep information in structured official sources, where answers may require clickable paths and reasoning.Source: §1 · p. 2

2 Related Work

WebWalkerQA differs from prior web benchmarks by framing tasks as QA pairs that cover both single-source and multi-source exploration, while WebWalker targets information-seeking through HTML-button reasoning and multi-agent navigation.

2.1 Web-Oriented Benchmark

  • Web-oriented benchmarks evolved from instruction-following tasks such as Mind2Web (Deng et al., 2023) toward increasingly human-like, vision-centric, broad, complex, and realistic evaluations.Several newer benchmarks incorporate screenshot information (Zheng et al., 2024a;b; He et al., 2024a; Koh et al., 2024a; Cheng et al., 2024).Source: §2.1 · pp. 2–3
  • WebWalkerQA uses QA pairs to cover single-source and multi-source queries, modeling two human-like page-exploration patterns; multi-source queries require information from distinct webpages.The closest prior benchmarks, MMInA (Zhang et al., 2024c) and AssistantBench, also focus on time-consuming, multi-page navigation tasks.Source: §2.1 · pp. 2–3

2.2 Agents on Web-Navigation

  • WebWalker specializes in information-seeking by reasoning over HTML button data and uses a multi-agent framework to navigate pages for reliable, authoritative information.Prior web agents commonly use small language models to filter actions or identify relevant HTML elements, prompt LLMs with agentic modules, or use screenshots as sensory input.Source: §2.2 · p. 3

3 WebWalkerQA

WebWalkerQA is a 680-pair bilingual benchmark built from recursively traversed official-site subpages, with single- and multi-source questions and explicit difficulty levels. It frames web traversal as finding information from a root URL and query, assessed by answer correctness and successful-action efficiency.

3.1 Data Collection

  • The dataset combines GPT-4o-generated annotations with human quality control and calibration of questions and answers.The pipeline recursively collects accessible subpages, constructs queries from page information and specified roles, then filters unnatural queries for short, entity-containing answers. Its multi-source questions require multiple pages and are not easily exploited by search-engine shortcuts.Source: §3.1 · p. 4

3.2 Data Statistics

  • Questions are categorized as single-source or multi-source, with difficulty determined by the depth labels assigned to their source pages.Single-source depths 2–4 correspond to easy, medium, and hard; multi-source summed depths 2–4, 4–6, and 6–8 correspond to those levels, respectively.Source: §3.2 · pp. 4–5
  • WebWalkerQA contains 680 bilingual QA pairs across four real-world domains; education accounts for 46.3% and Chinese for 60.5%.The remaining domain shares are conference 24.0%, organization 7.9%, and game 24.0%; English accounts for 39.5%.Source: §3.2 · pp. 4–5, Figure 3 · p. 5

3.3 Web Traversal Task and Evaluation

  • Web traversal starts from a root URL and query, and aims to find enough information through website page navigation to answer it.Evaluation measures answer accuracy and the action count of successful executions that answer correctly; GPT-4 judges correctness against the ground truth using a CoT prompting strategy (Wei et al., 2022).Source: §3.3 · p. 5

4 WebWalker

WebWalker uses explorer and critic agents to traverse web subpages and accumulate information until the critic can answer the query. Exploration selects subpage URLs rather than answering directly, while the critic updates memory and stops when information is sufficient or the step limit is reached.

4 WebWalker

  • WebWalker pairs an explorer that traverses web pages with a critic that updates memory until enough information is gathered to answer the query.The explorer follows a Thought-Action-Observation paradigm; the critic evaluates whether the accumulated information is sufficient.Source: §4 · p. 5

4.1 Think then Explore

  • The explorer uses page content and clickable HTML sublinks as observations, then selects a subpage URL to explore rather than answering the query.Its context includes the sequence of prior observations and actions, and exploration continues until the critic decides to answer or the maximum number of steps is reached.Source: §4.1 · pp. 5–6

4.2 Think then Critique

  • After each explorer action, the critic takes the query and current observation-action pair, incrementally updates memory, and answers when the gathered information is sufficiently complete.The critic is introduced to address the explorer’s implicit policy and potentially large history context.Source: §4.2 · p. 6

5 Experiment

Across tested backbones, WebWalker outperforms the baselines but remains below 40% accuracy with GPT-4o, while results also show performance differences by domain and language and reveal exploration and reasoning errors.

5.1 Experimental Setting

5.2 Main Results

  • WebWalker outperforms Reflexion and ReAct, closed-source models outperform open-source models, and even GPT-4o-backed WebWalker stays below 40% accuracy, highlighting WebWalkerQA’s difficulty.Accuracy declines as query depth or the number of required sources increases.Source: §5.2 · p. 7, Table 3 · p. 7
  • Larger open-source models improve in performance and correct-execution action count; greater depth or source requirements reduce accuracy, while larger models or reflection can support long-distance traversal.Action counts include only correct executions.Source: §5.2 · pp. 7–8

5.3 Results across Domains and Languages

  • WebWalker performs relatively better on conference queries and similarly in Chinese and English, which the authors relate to explicit conference-button information and bilingual model training.The reported domain and language performance is shown in Figure ??.Source: §5.3 · p. 8, Figure 5 · p. 8

5.4 Error Assessment

  • Incorrect executions arise from refusal or wrong-page localization, reasoning errors, or exceeding the step limit; small ReAct models may give up early, while memory and scale mitigate this tendency.Some reasoning errors occur even after the golden page has been visited, showing that interpreting page information can remain difficult.Source: §5.4 · p. 8

6 Discussion

The discussion finds that RAG systems struggle to retrieve deep information on WebWalkerQA, while adding WebWalker to RAG improves performance across difficulty levels; increasing action count K also improves performance within a tested range.

6.1 RAG Performance on WebWalkerQA

  • RAG systems struggle with WebWalkerQA’s traversal challenges, with performance worsening as information depth increases.Without retrieval, even the strongest tested models perform very poorly; among the evaluated RAG systems, Tongyi achieves the best result at 40%, and multi-source accuracy is lower than single-source accuracy.Source: §6.1 · p. 9

6.2 WebWalker Combined with RAG System

  • Integrating WebWalker into standard RAG improves performance across difficulty levels, especially for multi-source questions.WebWalker provides vertical exploration, and the critic agent’s memory M is appended to relevant documents to aid generation.Source: §6.2 · pp. 9–10, Figure 7 · p. 10

6.3 Scaling Up on Action Count K

  • Larger action counts K improve performance within the tested range of 5, 10, 15, 20, and 25.The authors present scaling the process of digging through links as a potential direction for vertical exploration in RAG systems.Source: §6.3 · p. 10, Figure 7 · p. 10

7 Conclusion

The paper introduces WebWalkerQA to evaluate LLMs’ web traversal in complex, multi-step information-seeking tasks and WebWalker, a multi-agent framework combining exploration and critique. Experiments show that WebWalkerQA challenges RAG systems and that combining RAG with WebWalker improves web navigation performance.

7 Conclusion

  • Combining RAG with WebWalker improves web navigation performance, while WebWalkerQA effectively challenges RAG systems.The work emphasizes deep, vertical exploration as important for web-based tasks and aims to support more scalable and reliable LLM-based information retrieval integrated with RAG.Source: §7 · p. 10
  • WebWalkerQA benchmarks LLMs’ web traversal abilities in complex, multi-step information-seeking tasks.Source: §7 · p. 10
  • WebWalker is a multi-agent framework that mimics human-like web navigation by combining exploration and critique.Source: §7 · p. 10

A Limitations and Discussion

WebWalker can retrieve information from a webpage independently or integrate with RAG, while the paper notes limitations in dataset size, HTML-only navigation, and prompt-only operation and outlines possible improvements.

B Implementation Details … E.3 Details Prompts for Agents

This material specifies the RAG comparison systems, dataset-source and annotation procedures, and the implementation and prompts used for WebWalker.

B Implementation Details

C Details for RAG Systems

C.1 Commercial Systems

  • The five commercial systems—Doubao, ERNIE-4.0-8K, Tongyi, Kimi, and Gemini-Search—are accessed through business-oriented APIs for reproducibility.Source: §C.1 · p. 16

C.2 Open-sourced Systems

  • Mindsearch (Chen et al., 2024a) mimics human web information seeking with WebPlanner and WebSearcher, while the from-scratch Naive RAG uses Google’s top 10 links with Qwen-Plus.Source: §C.2 · pp. 16–17

D Annotated Case

E.1 Sources of Root Page

  • The dataset’s root pages are found by Google searches and manual filtering; education sources are official computer science department websites at universities.Source: §E.1 · p. 17

E.2 Details on Prompts for Annotation

  • Multi-source annotation prompts require a coherent query combining at least two related sublinks into a concise entity answer, with strict checks that both documents are necessary and the answer is correct.A separate single-source judge checks whether current knowledge in doc2 is required, rejecting answers already supported by known knowledge in doc1.Source: §E.2 · pp. 17–18

E.3 Details Prompts for Agents

  • The Explorer prompt directs agents to search buttons for quality sources, while Critic prompts assess whether observations are useful and whether accumulated information suffices to answer.Source: §E.3 · pp. 19–20

F Details for Evaluation · F.1 Evaluator · G Case Study

The evaluator prompt is shown in Figure 9.

F.1 Evaluator

G.1 Reasoning Error · G.2 Time Cut-off

These sections identify two limits in web-based question answering: systems can find the relevant page but still fail to reason over its contents, and stale temporal data can prevent answers about newer information.

G.1 Reasoning Error

  • The Inclusive Connections Lounge question requires locating the relevant webpage before answering.Table 5 presents a case requiring reasoning capability in a web traversal task.Source: §G.1 · p. 20, Table 5 · p. 21
  • After finding the page, the system must understand its information comprehensively to calculate the required time.Source: §G.1 · p. 20
  • Even when the source page is found, errors can occur if the system fails to process the time correctly.The task therefore also depends on time-calculation or reasoning ability.Source: §G.1 · p. 20

G.2 Time Cut-off

  • o1’s temporal data cutoff is October 2023.Table 6 presents a case of time cutoff in o1 predictions.Source: §G.2 · p. 20, Table 6 · p. 21
  • Because of that cutoff, o1 cannot answer questions about web information published after October 2023.Source: §G.2 · p. 20

Related papers

Loading 2501.07572v3…