Source-linked AI summary
WebWalker: Benchmarking LLMs in Web Traversal
Jialong Wu, Wenbiao Yin, Yong Jiang, Zhenglin Wang, Zekun Xi, Runnan Fang, Linhai Zhang, Yulan He, Deyu Zhou, Pengjun Xie, Fei Huang
TL;DR
Traditional retrieval may miss information embedded deeply within websites, motivating evaluation of LLM web traversal. The paper introduces WebWalkerQA and the WebWalker explore-critic framework, finding that the benchmark challenges current systems and that combining RAG with WebWalker improves web navigation performance. The authors conclude that deep vertical exploration is important for web-based information retrieval integrated with RAG.
Problem
Traditional search may retrieve shallow content, leaving complex multi-layered web information insufficiently evaluated by standard RAG.
Method
The paper constructs WebWalkerQA and proposes WebWalker, a multi-agent framework with explorer and critic agents for human-like web navigation.
Results
WebWalkerQA remains challenging for strong LLMs, while combining RAG with WebWalker improves web navigation performance.
Takeaways & Limitations
Deep vertical exploration provides an important complement to horizontal retrieval for web-based information retrieval with RAG.
Takeaways & Limitations
The appendix discussion limits the described role of WebWalker to operating on a given webpage or integrating with RAG systems.
Abstract
from arXiv · showhide
Retrieval-augmented generation (RAG) demonstrates remarkable performance across tasks in open-domain question-answering. However, traditional search engines may retrieve shallow content, limiting the ability of LLMs to handle complex, multi-layered information. To address it, we introduce WebWalkerQA, a benchmark designed to assess the ability of LLMs to perform web traversal. It evaluates the capacity of LLMs to traverse a website's subpages to extract high-quality data systematically. We propose WebWalker, which is a multi-agent framework that mimics human-like web navigation through an explore-critic paradigm. Extensive experimental results show that WebWalkerQA is challenging and demonstrates the effectiveness of RAG combined with WebWalker, through the horizontal and vertical integration in real-world scenarios.
1 Introduction
The paper introduces WebWalkerQA to evaluate LLM web traversal when relevant information lies deep within websites, and proposes WebWalker for navigating such tasks. Experiments find the benchmark challenging and show benefits from combining horizontal RAG with vertical exploration.
- Traditional search performs horizontal query retrieval but may miss deeper website content needed for complex information-seeking tasks.
- WebWalkerQA evaluates systematic traversal from a root website through multi-step, text-based question answering.
- WebWalker uses explorer and critic agents: the explorer follows Thought-Action-Observation steps, while the critic maintains memory and generates responses.
- Even powerful LLM backbones perform suboptimally on WebWalkerQA, supporting its characterization as a challenging benchmark.
- Combining RAG with WebWalker coordinates horizontal retrieval and vertical exploration, while vertical exploration benefits information-seeking tasks.
- 680 queries across four real-world scenarios and more than 1373 webpages form the WebWalkerQA benchmark.
2 Related Work
Related benchmarks increasingly evaluate realistic web interaction, but WebWalkerQA focuses specifically on question answering over multi-page traversal. It distinguishes traversal depth, source breadth, and multi-step interaction in web information-seeking settings.
- Earlier web benchmarks evaluated web-oriented tasks, while newer LLM benchmarks increasingly support complex, realistic, and vision-centric interaction.
- In WebWalkerQA, multi-source means that answering requires information from multiple distinct web pages.
- Benchmark comparisons characterize depth as exploration extent, width as source multiplicity, hop as required steps, and #Pages as webpages involved.
- WebWalkerQA differs from closely related multi-page benchmarks by using question-answer pairs to evaluate web traversal and reasoning.
- Web-oriented agents generally either use smaller models to filter actions or HTML elements, or prompt LLMs with agentic modules for navigation.
3 WebWalkerQA
WebWalkerQA is a bilingual, multi-domain benchmark for web traversal that evaluates whether models can find information buried across linked pages. It contains 680 QA pairs built through recursive website traversal, LLM annotation, and human verification, with single- and multi-source difficulty levels.
- The dataset construction recursively traverses official websites, generates queries from gathered page information, filters them, and applies human quality control.
- WebWalkerQA contains 680 question-answer pairs spanning conference, organization, education, and game websites.
- Single-source questions require exploring one subpage, whereas multi-source questions require reading two associated pages simultaneously.
- Difficulty is determined by traversal depth: single_source2–4 and multi_source2–8 are grouped into easy, medium, and hard levels.
- WebWalkerQA includes Chinese and English data, with domain proportions of 24.0%, 7.9%, 46.3%, and 24.0% and language proportions of 60.5% and 39.5%, respectively.
- The Web Traversal task gives an initial website URL and query, evaluating answers with QA accuracy and successful-execution action count.
4 WebWalker
WebWalker is a multi-agent framework for navigating web pages through an explorer–critic process. The explorer selects subpages using page observations and interaction history, while the critic incrementally manages memory for answering the query.
- WebWalker consists of an explorer agent and a critic agent operating in a Thought-Action-Observation navigation paradigm.
- At each step, the explorer receives the current page and clickable sublinks, then selects a subpage URL according to its policy and history.
- The critic runs after explorer execution and incrementally accumulates relevant information from the query and current observation.
5 Experiment
Experiments compare WebWalker with ReAct and Reflexion across diverse LLM backbones and evaluate traversal performance, efficiency, domains, languages, and error patterns. WebWalker performs best among the agent frameworks, but WebWalkerQA remains difficult even for strong models.
- Experimental Setting: WebWalker, Reflexion, and ReAct are compared across closed-source and open-source LLM backbones using accuracy and successful-execution action count.The evaluation includes models with at least 128K context windows and at least 7B parameters, with zero-shot exploration capped at 15 actions.
- Main Results: WebWalker outperforms Reflexion, which outperforms ReAct across the evaluated backbone models.The comparison considers both performance and efficiency, while action counts are measured only for correct executions.
- Main Results: Even the best-performing WebWalker with GPT-4o remains below 40% accuracy, highlighting the challenge of WebWalkerQA.Performance declines as traversal depth or the number of required sources increases.
- Main Results: Increasing model size or adding reflection improves long-range web traversal, with larger models also showing higher successful-execution action counts.The results associate larger models and reflection with better handling of multi-step solutions.
- Results across Domains and Languages: WebWalkerQA spans Chinese and English across games, conferences, education, and organizations, with relatively stronger conference performance and similar results across languages.Conference pages may be easier because their button information is more explicit and directive; the models were bilingual.
- Error Assessment: Incorrect executions comprise refusal or wrong locating, reasoning errors, and exceeding the action limit.Some cases locate the golden page but still mark the answer incorrectly, underscoring the difficulty of reasoning over page information.
6 Discussion
The discussion finds that standard RAG systems struggle with WebWalkerQA because horizontal retrieval misses deep or distributed information. Integrating WebWalker adds vertical exploration, and increasing exploration steps improves performance within a tested range.
- RAG Performance on WebWalkerQA: RAG systems struggle with WebWalkerQA’s web-traversal challenges, while even strong close-book models perform poorly without retrieval.The benchmark uses dynamically updated official websites, whereas pretrained models rely on static knowledge limited by their cutoff dates.
- RAG Performance on WebWalkerQA: Standard RAG performs horizontal document search, whereas WebWalker performs vertical page exploration to acquire deeper information.The paper presents WebWalker as a module that can integrate with standard RAG systems.
- RAG Performance on WebWalkerQA: WebWalker can function as a module in agentic RAG, enabling vertical exploration and improving web navigation performance.The paper reports this integration using WebWalker built on Qwen-2.5-Plus within a naive RAG system.
- Scaling Web Traversal: Larger exploration limits K ∈ {5, 10, 15, 20, 25} lead to better performance within the tested range.The result supports vertical scaling during inference when tracing source information.
7 Conclusion
The paper introduces WebWalkerQA for complex, multi-step web-traversal evaluation and WebWalker, a multi-agent framework combining exploration and critique. Results show that combining RAG with WebWalker improves navigation and underscores the value of deep vertical exploration.
- Conclusion: WebWalkerQA evaluates LLM web-traversal abilities in complex, multi-step information-seeking tasks.WebWalker is proposed as a multi-agent framework that mimics human-like navigation through exploration and critique.
- Conclusion: Combining RAG with WebWalker improves web navigation performance and highlights deep vertical exploration as important for web-based information retrieval.The conclusion connects horizontal retrieval with WebWalker’s vertical exploration.
A Limitations and Discussion
The paper notes limitations in dataset size and describes WebWalker’s ability to operate independently or integrate with RAG systems. Under agentic RAG, click actions are reported as highly effective.
- Limitations: WebWalkerQA currently contains 680 high-quality QA pairs, while approximately 14k additional silver pairs remain unverified.The authors identify dataset size as a limitation and suggest the silver pairs as supplementary training data.
- Discussion: WebWalker can operate as a web information-retrieval assistant for a webpage or integrate with RAG systems to expand their scope.The paper frames this integration within the agentic RAG paradigm.
- Discussion: Within agentic RAG, the click action is reported as highly effective.
B Implementation Details
The implementation evaluates multiple commercial and open-source systems and defines structured annotation and agent prompts for web traversal. The Explorer uses tool-mediated navigation, while the Critic extracts useful information and judges answer sufficiency.
- Five commercial systems and two open-source systems are selected for evaluation.
- The dataset uses JSON records containing Question, Answer, Root_Url, and detailed Info metadata.Info includes Hop, Domain, Language, Difficulty_Level, Source Website, and Golden_Path.
- Annotation prompts: Multi-source annotation constructs concise queries requiring integrated information from at least two sublinks.The annotation procedure also requires accurate answers and explains why selected sublinks are interrelated.
- Annotation prompts: Strict judges reject answers that are incorrect, non-concise, single-document, or insufficiently multi-step.For multi-source cases, both documents must provide necessary information for the answer.
- WebWalker agents: The Explorer traverses pages through repeated Thought-Action-Observation steps, while the Critic extracts relevant information and evaluates answer sufficiency.The Critic returns usefulness information when observations help answer the query and otherwise marks them unhelpful.
F Details for Evaluation
The evaluator prompt is presented in Figure 9.
- Figure 9 presents the evaluator prompt used in the evaluation procedure.
- The evaluator prompt is shown as a figure rather than described in the surrounding text.
- The surrounding passage identifies Figure 9 as the location of the evaluator prompt.
G.1 Reasoning Error
Reasoning errors can arise after a relevant webpage is found when answering requires interpreting page information and calculating a result. Static knowledge cutoffs also prevent o1 from answering about later web information.
- Reasoning Error: A web-traversal question may require locating a relevant webpage, understanding its information, and calculating the requested time.
- Reasoning Error: Even after successfully locating the source page, systems may err if they fail to process the time correctly.
- Time Cut-off: October 2023 is o1’s temporal-data cutoff, preventing answers about web information published after that point.