Source-linked AI summary
Mind2Web 2: Evaluating Agentic Search with Agent-as-a-Judge
Boyu Gou, Zanming Huang, Yuting Ning, Yu Gu, Michael Lin, Weijian Qi, Andrei Kopanev, Botao Yu, Bernal Jiménez Gutiérrez, Yiheng Shu, Chan Hee Song, Jiaman Wu, Shijie Chen, Hanane Nour Moussa, Tianshu Zhang, Jian Xie, Yifei Li, Tianci Xue, Zeyi Liao, Kai Zhang, Boyuan Zheng, Zhaowei Cai, Viktor Rozgic, Morteza Ziyadi, Huan Sun, Yu Su
TL;DR
Existing benchmarks struggle to evaluate agentic search because they emphasize short horizons and static answers, despite systems producing complex, time-varying responses. Mind2Web 2 introduces 130 realistic long-horizon tasks and task-specific tree-rubric judge agents that assess correctness and attribution. Across ten systems and human performance, OpenAI Deep Research reaches 50-70% of human performance while spending half the time, while time-varying tasks remain difficult.
Problem
Existing benchmarks emphasize short-horizon tasks and predefined, time-invariant answers, limiting evaluation of complex agentic-search behavior.
Method
Mind2Web 2 uses 130 realistic long-horizon tasks and dedicated judge agents that evaluate fine-grained rubric criteria before aggregating scores to a root node.
Results
OpenAI Deep Research achieves 50-70% of human performance while spending half the time, while current systems still struggle with time-varying tasks.
Takeaways & Limitations
Mind2Web 2 provides a rigorous platform for advancing and benchmarking agentic search systems across open-ended long-horizon tasks.
Takeaways & Limitations
Automated attribution evaluation can fail when critical information is hidden in collapsed webpage sections.
Abstract
from arXiv · showhide
Agentic search such as Deep Research systems-where agents autonomously browse the web, synthesize information, and return comprehensive citation-backed answers-represents a major shift in how users interact with web-scale information. While promising greater efficiency and cognitive offloading, the growing complexity and open-endedness of agentic search have outpaced existing evaluation benchmarks and methodologies, which largely assume short search horizons and static answers. In this paper, we introduce Mind2Web 2, a benchmark of 130 realistic, high-quality, and long-horizon tasks that require real-time web browsing and extensive information synthesis, constructed with over 1000 hours of human labor. To address the challenge of evaluating time-varying and complex answers, we propose a novel Agent-as-a-Judge framework. Our method constructs task-specific judge agents based on a tree-structured rubric design to automatically assess both answer correctness and source attribution. We conduct a comprehensive evaluation of ten frontier agentic search systems and human performance, along with a detailed error analysis to draw insights for future development. The best-performing system, OpenAI Deep Research, can already achieve 50-70% of human performance while spending half the time, highlighting its great potential. Altogether, Mind2Web 2 provides a rigorous foundation for developing and benchmarking the next generation of agentic search systems.
1 Introduction
Agentic search systems autonomously plan, browse, and synthesize citation-backed answers, but their long-horizon, time-varying tasks exceed existing evaluation methods. Mind2Web 2 addresses this gap with a realistic benchmark and an Agent-as-a-Judge framework evaluating correctness and attribution.
- Agentic search systems decompose complex queries, iteratively browse dynamic websites, and synthesize citation-backed responses.
- Long-horizon agentic-search tasks can require hours, hundreds of actions, and interactions across dozens of websites, making reliable evaluation difficult.
- Existing benchmarks typically use moderate-horizon, single-website tasks or predefined, time-invariant answers that do not represent the full task spectrum.
- Mind2Web 2 contains 130 realistic, high-quality tasks across diverse practical domains, constructed with at least 1,000 hours of human labor.
- The Agent-as-a-Judge framework uses tree-structured rubrics to evaluate answer correctness and whether statements are attributable to cited sources.
- OpenAI Deep Research achieves 50-70% of human performance while spending half the time, although current systems still struggle with time-varying tasks.
2 Related Work
Related work spans agentic search systems, autonomous web-agent benchmarks, and LLM-based evaluation. Mind2Web 2 extends this literature toward realistic, complex tasks requiring comprehensive answers with real-time information.
- Agentic Search: Agentic search systems iteratively and autonomously tackle complex tasks using search APIs, retrieval, web browsing, planning, and interaction with live websites.
- Benchmarks and Evaluation Methodologies: Traditional web-agent benchmarks usually evaluate short, transactional processes on a single website rather than the complete agentic-search system.
- Benchmarks and Evaluation Methodologies: Mind2Web 2 uses Agent-as-a-Judge to support realistic and complex tasks requiring comprehensive answers with real-time information.
- Evaluation Methodologies: Although LLM-as-a-Judge is widely used, agentic-search complexity motivates Agent-as-a-Judge; unlike PaperBench's manually written rubric, this work largely automates rubric construction.
3 Mind2Web 2
Mind2Web 2 constructs a benchmark for realistic, long-horizon agentic-search tasks and addresses their evaluation through task-specific, tree-structured judge agents. Its benchmark combines fine-grained rubric scoring with extensive human task construction and validation.
- Overview: Mind2Web 2 addresses two challenges: collecting sufficiently complex, realistic tasks and automatically evaluating complex answers from different agentic-search systems.The benchmark emphasizes practical needs, long-horizon effort, objective verifiability, and time-varying answers.
- Rubric Design: Each rubric uses binary leaf judgments aggregated through a tree, with critical nodes acting as failure gates and non-critical nodes enabling partial scoring.Sequential nodes can short-circuit later evaluations when an earlier dependent criterion fails.
- Evaluation Metrics: The benchmark reports Partial Completion as the average root score and Success Rate as the percentage of tasks with a perfect root score of 1.These metrics distinguish partial satisfaction from full task completion.
- Judge Agents: A dedicated judge agent extracts structured information from an answer and verifies each rubric leaf before aggregating scores to the root.The Verifier can examine extracted answer text together with a screenshot of the corresponding webpage.
- Task Collection: 130 tasks were collected through task proposal, refinement, and validation by experienced annotators, with at least 1,000 hours of total construction labor.The labor covered both task collection and judge-agent development.
- Benchmark Statistics: Mind2Web 2 is described as the only agentic-search benchmark focusing on long-horizon, time-varying tasks, with dozens to hundreds of evaluation nodes per task.Its comparison table groups task horizons as Short (< 10), Medium (10−50), and Long (> 50) required actions.
4 Experiments
Experiments compare agentic search systems and humans across completion, efficiency, time-varying tasks, response styles, and error patterns. Deep Research systems generally outperform shallower search systems, but full task completion and live-information handling remain difficult.
- Up to 28% for agents and 54% for humans achieved success, showing that tedious tasks challenge both groups.
- Comparison Between Agent Types: Deep Research systems generally outperform search-augmented LLMs and web agents, reflecting stronger long-horizon information gathering and synthesis.ChatGPT Search and Perplexity Pro Search are limited by restricted search horizons, while Deep Research systems sustain more extensive task engagement.
- Test-Time Scaling: Increased inference time improves performance, and multiple independent trials raise Pass@3 scores by increasing the likelihood of task success.The paper links longer inference to more thorough retrieval, processing, and synthesis for complex tasks.
- Most systems perform worse on 57 explicitly time-varying tasks, whereas Operator and humans achieve relatively on-par or superior performance.These tasks involve relative dates, times, or frequently changing information such as product prices.
- Promises of Agentic Search: OpenAI Deep Research achieves 50-70% of human performance while spending less than half the time, indicating early promise despite current limitations.The paper attributes human failures partly to cognitive fatigue and limited working memory, while agents often handle detailed retrieval and interpretation more consistently.
- Error Analysis: Error analysis identifies incompleteness, invalid or missing attribution, unsupported answers, and evaluator discrepancies across agents and humans.Incompleteness is prevalent among all systems and is the most common human error; 7 of 720 nodes were actual verifier errors after excluding human mistakes and source inconsistency.
5 Conclusions
Mind2Web 2 is designed to comprehensively evaluate agentic search systems on long-horizon information-gathering tasks and time-varying answers. The paper presents it as a platform for assessing and advancing these systems.
- Mind2Web 2 evaluates agentic search systems on long-horizon information-gathering tasks and time-varying answers.
- The benchmark uses a scalable, automated, and reliable Agents-as-a-Judge framework to assess agent performance on open-ended long-horizon search tasks.
- A comprehensive empirical analysis spans AI-based search engines, deep research systems, and web agents, revealing their potential and current limitations.
- Mind2Web 2 provides a rigorous assessment platform for advancing agentic search systems.
A Limitations
The benchmark has scope, attribution, judgment, and system-access limitations, while its structured evaluation is intended to support transparent and accountable agentic-search development. Its broader-impact risks include benchmark overfitting and rubric-amplified bias.
- Task Coverage and Scope: Mind2Web 2 covers 130 curated tasks but excludes some vague or highly subjective information-seeking scenarios.The exclusions reflect its focus on realistic, tedious information gathering and practical evaluation.
- Evaluation Framework Assumptions: The framework assumes cited URLs are truthful and credible, and that critical information can be attributed to individual webpages.Source credibility and truthfulness evaluation are outside the work’s scope, and some tasks may not fit single-page attribution.
- Reliance on LLM-based Judgments: LLM-based extraction and verification may introduce errors or incorrect judgments.The authors use multi-stage validation and human evaluation to mitigate potential inaccuracies and assess reliability.
- Limited Analysis on Black-Box Systems: Focusing on capable commercial and research-grade systems, especially proprietary or closed-source ones, limits interpretation of performance differences and precise inference costs.The benchmark reports task completion time and answer length as relative practical-efficiency references.
- Broader Impacts: Structured rubrics and explicit source-attribution verification are intended to support transparent and accountable agentic-search development.
- Broader Impacts: Widespread rubric-based evaluation could enable benchmark-specific training, overfitting, and amplification of rubric or evaluation biases.The authors mitigate this risk by keeping the test set, rubric, evaluation script, and script-generation pipeline hidden.
C Details of Task Collection
Mind2Web 2 collects 130 tasks through explicit design principles and a three-stage proposal, refinement, and validation pipeline. The tasks emphasize realistic, laborious, clear, objective, verifiable web-search needs while excluding several difficult-to-evaluate settings.
- C.1 Domain Distribution: 130 tasks are organized across 6 broad domains and 24 sub-domains.
- Design Principles: Tasks must represent authentic practical needs rather than artificial combinations of unrelated steps.
- Design Principles: Tasks must require sustained web search, exploration, and information synthesis, with at least five minutes of human effort.Five minutes is a minimum; most Mind2Web 2 tasks take humans substantially longer.
- Design Principles: Task descriptions and answer criteria must be explicit, unambiguous, objective, and checked through manual and LLM-assisted ambiguity inspection.
- Design Principles: Tasks require practically verifiable criteria based mainly on the answer text and expected URL-based provenance.Minor portions may use external APIs or fixed ground-truth answers when necessary.
- Additional Constraints and Exclusions: The benchmark excludes video-understanding, non-English-website, complex-reasoning, and external-tool-dependent tasks.Evaluated systems themselves are not restricted from using helpful tools.
- Collection Pipeline: Task collection proceeds through proposal, refinement, and validation, with independent validators completing tasks and checking ambiguity, edge cases, feasibility, and evaluation assumptions.Only tasks passing validation from at least two validators are included.
C.4 Future Maintenance of the Benchmark
Mind2Web 2 remains exposed to website changes over time, but its broad information-seeking goals, final-answer evaluation, and Agent-as-a-Judge framework reduce this sensitivity. The authors also commit to periodic maintenance and task replacement when necessary.
- Website changes or updates may affect Mind2Web 2 tasks over time.
- Broad information-seeking goals let agents select sources flexibly rather than following fixed websites or action sequences.
- Evaluating final retrieved information instead of intermediate interactions, together with Agent-as-a-Judge, substantially reduces sensitivity to website changes.
- The authors will periodically review tasks and update or replace those whose website changes materially alter difficulty or solvability.Replacement tasks are intended to preserve similar complexity and scope.
D Details on Rubrics and Judge Agents
The framework uses tree-structured rubrics and judge-agent components to evaluate complex answers through extraction and verification. Judge agents are generated and refined with automated debugging and self-reflection.
- D.1 Rubric Design: Tree-structured rubrics provide unified, scalable, and practical scoring across Mind2Web 2 tasks and potential future tasks.The design emphasizes practical verification and partial scores that meaningfully reflect incremental progress and user utility.
- D.1 Rubric Design: The rubric design aims to produce rigorous and practically useful evaluations across varied and complex agentic search tasks.
- D.2 Details for Judge Agents: Judge-agent development uses rubric trees, extraction pipelines, and leaf-node verification processes supported by reusable helper functions and classes.The codebase includes rubric structures, scoring mechanisms, Verifier and Extractor components, and auxiliary utilities.
- D.2 Details for Judge Agents: A default short-circuit mechanism skips verification after critical or preceding sequential-node failures, but human evaluation disables it for comprehensive assessment.
- D.2 Details for Judge Agents: Simple verification gives binary correctness judgments for claims or statements using the task description, complete answer, and any additional instructions.The verifier is instructed to examine the claim carefully and provide explicit, concise reasoning.
- D.2 Details for Judge Agents: URL-based verification determines whether claims are supported by the actual text or screenshot of a referenced webpage or PDF.The verifier returns a binary supported-or-not-supported judgment with reasoning, using the original task and complete answer as context.
- D.3 Rubric and Judge Agent Generation: The automated generation pipeline uses Claude-3.7-Sonnet to create initial judge-agent scripts from task descriptions, rubric principles, evaluation strategies, and toolkit functionality.Examples of common mistakes and practical scripting tips are also provided to the code-generation model.
- D.3 Rubric and Judge Agent Generation: Two autonomous debugging strategies refine generated scripts through runtime-feedback correction and checklist-guided self-reflection.Self-debugging continues until runtime errors disappear, while self-reflection reviews correctness, logical coherence, rubric completeness, and overlooked edge cases.
E Experimental Details
The experiments compare a broad range of agentic search systems under repeated task runs, with explicit source-attribution requirements and cached webpages for stable verification. A human study supplies a performance reference point using logged browsing behavior and quality screening.
- The evaluation spans search-API systems, browser-interacting web agents, hybrid systems, and other agentic search forms.
- Systems lacking reliable source attribution or unlikely to show meaningful benchmark performance are excluded.
- Each agent system is independently run three times per task, with completion times recorded or manually measured when unavailable.Most systems are evaluated through web interfaces, while Hugging Face Open Deep Research is treated separately.
- Answers are collected between April and June 2025, and Hugging Face Open Deep Research uses OpenAI’s o3 as its base model.
- Webpages referenced in answers are pre-fetched and cached, creating stable verification references for time-varying online content.Playwright loads unique URLs and supports both normal webpages and PDFs; blocked pages can receive manual review and replacement.
- Human performance is measured on Subset-30 by completers who browse independently, provide URL-based sources, and log time and visited webpages.Three completers are assigned per task, and participants must pass two simplified trial tasks before the formal study.
F Details of Error Analysis and Additional Case Studies
Figure F.1 presents the workflow used to categorize errors in the error analysis.
- Figure F.1 shows the workflow for categorizing errors during error analysis.
- The figure organizes the process around error categorization.
- Figure F.1 serves as a visual guide to the error-analysis categorization procedure.
F.1 Error Analysis
The error analysis separates answer correctness from source attribution and identifies incompleteness, criteria violations, invalid or missing sources, and unsupported claims. Human annotators apply these categories to representative agent and human answers.
- Errors are categorized independently along correctness and attribution dimensions to analyze agent and human failure patterns.
- Incompleteness includes explicitly unavailable information and answers containing fewer requested items or steps.The categories are Information Not Found and Partial Missing.
- Criteria Violation covers answers that contradict explicit task criteria or contain directly identifiable factual errors.An example exceeds a specified $200–$600 budget with a $1,277.97 shopping list.
- Attribution errors include fabricated or invalid URLs, missing URLs, and URLs that fail to support the answer’s claims.
- Unsupported answers are divided into Synthesis Error, where useful source information is misrepresented, and Retrieval Error, where sources are irrelevant.A synthesis example contains inaccurate extracted details, while a retrieval example cites a URL unrelated to the claimed character abilities.
- Human annotators examine one randomly selected answer per system and human answers, using Figure F.1 to guide error identification and categorization.The representative systems include ChatGPT Search, Perplexity Pro Search, HF Open Deep Research, OpenAI Deep Research, and OpenAI Operator.
F.2 Additional Case Studies
The case studies show that both humans and agentic systems make distinct errors in long-horizon web search, including hallucination, carelessness, tool failures, and weak information synthesis. The examples span multiple frontier systems and illustrate failures in retrieval, attribution, and answer construction.
- Pervasive Hallucination Across All Systems: 23% of tasks contain hallucination for OpenAI Deep Research, while other systems reach at least 50%.The reported hallucination rate counts Invalid Attribution or Unsupported Answer and is likely an underestimate.
- Human Mistakes due to Carelessness: Human answers fulfill task requirements without omission or hallucinated URLs, but carelessness causes constraint, extraction, spelling, and common-sense errors.The paper attributes representative cases to overlooking requirements and misreading or incorrectly extracting webpage information.
- System Errors from Hugging Face Open Deep Research: Hugging Face Open Deep Research often produces Information Not Found errors after improper tool usage or generated-code mistakes prematurely terminate execution.The observed failures include incorrect input formats and coding errors that lead the agent to conclude that requested information is unavailable.
- Web Agents for Long-Horizon Information Seeking: Operator frequently performs poorly because it lacks sufficient long-term reasoning, planning, grounding, and memory for retrieved information and sources.The case studies also include invalid attribution and other system-level failures across frontier search systems.
- Additional Case Studies: OpenAI Deep Research can present claims without verifiable attribution, while Perplexity Pro Search may fail to find discoverable information within limited search steps.These examples indicate distinct failures in attribution and search horizon rather than a single common retrieval error.
- Additional Case Studies: ChatGPT Search retrieves relevant webpages but fails to synthesize a correct, accurately attributed answer for information spanning 20 years.The case highlights that finding relevant sources does not guarantee correct long-horizon synthesis.
H Instructions for Human Annotators
The human-annotator instructions define how Mind2Web 2 tasks are proposed, refined, validated, and evaluated. They prioritize realistic, sufficiently difficult, clear, objectively verifiable, publicly accessible, and single-round information-seeking tasks.
- H.1 Instructions for Task Collection: Task proposals should reflect realistic scenarios rather than artificially combining simple steps to increase complexity.The instructions allow some subjectivity about realism because practical needs differ across people.
- H.1 Instructions for Task Collection: Task descriptions must be clear, grammatically correct, explicit about required background knowledge, and free of vague or subjective criteria.Criteria should be objective, unambiguous, measurable, and resistant to alternative interpretations.
- H.1 Instructions for Task Collection: Tasks should take more than five minutes, require more than one or two simple searches, and use publicly accessible information without login or paywall requirements.The checklist also asks annotators to account for familiarity that could underestimate completion time.
- H.1 Instructions for Task Collection: Each task requires an outline of critical answer information so correctness and source attribution can be verified with judge and helper tools.The outline includes explicitly requested information and reasonable additions, while the verification tools include LLM and URL-based checks plus selected APIs.
- H.1 Instructions for Task Collection: The current framework does not support video or non-English websites, rapidly changing answers, extensive reasoning, complex calculations, or external tools.These constraints define the intended scope of information gathering through web browsing.
- H.1 Instructions for Task Collection: Validation requires completing each task end-to-end and confirming that critical information can be located and verified from URL sources.Validators assess realism, clarity, tediousness, feasibility, and whether tasks are information-seeking and sufficiently complex.
- H.1 Instructions for Task Collection: URL-based verification requires sources to directly support statements and excludes dynamic, unstable, multi-page, or login-gated information that the framework cannot reliably verify.The instructions specifically exclude content hidden behind interactions, unstable search-result URLs, and tasks requiring credentials.
- H.2 Instructions for Human Performance Study: Human participants browse relevant websites, record answers in Google Docs, and include URLs that allow others to verify each claim.Tasks must also provide explicit ground truth and evaluation notes, remain single-round, and contain all necessary information in the description.