Mind2Web 2: Evaluating Agentic Search with Agent-as-a-Judge

Motivation

• Agentic search systems plan and decompose complex queries, browse dynamically, and synthesize citation-backed responses, offloading much of the search work from users.The intended benefit is to let users focus more on oversight and critical decisions.Source: §1 · p. 2

Evaluation need

• As agentic search grows more complex, reliable evaluation is needed to assess long-horizon outputs and verify that synthesized answers are correct and supported by their sources.Automated evaluation can help detect plausible-sounding hallucinations and establish trustworthiness.Source: §1 · p. 2

Evaluation gap

• Existing benchmarks often limit tasks to moderate horizons or predefined, time-invariant answers, leaving long-horizon tasks requiring live information poorly covered.Such tasks may involve dozens to hundreds of actions on one website, or span many websites.Source: §1 · p. 2

Benchmark

• Mind2Web 2 contains 130 realistic, high-quality tasks across practical domains, constructed and validated with at least 1,000 hours of human labor.Each task undergoes multiple stages of expert polishing and validation to support realism, complexity, and verifiability.Source: §1 · p. 2

Evaluation framework

• Its Agent-as-a-Judge framework uses task-specific rubric trees to assess answer correctness and whether statements are supported by cited sources.Judge agents interleave information extraction, LLM judging, and tool calls; rubric trees average 50 nodes and reach 603, and human evaluation found 99% correctness.Source: §1 · pp. 2–3

Results

• OpenAI Deep Research reaches 50–70% of human performance in half the time, beats humans on some detail-heavy tasks, and current systems still struggle with time-varying tasks.Deep Research systems also show a clear advantage over search-augmented LLMs and web agents such as Operator, attributed to effective tool use and sustained focus over long horizons.Source: §1 · p. 3

Agentic Search

• Agentic search spans API-augmented LLMs, live-web agents, and longer-horizon Deep Research systems, while conventional search misses dynamically rendered information and no benchmark covers the spectrum.Visual web agents [11] [17] [31] [44] and OpenAI’s Operator represent advances in autonomous browsing; Operator is described as the current frontier [39].Source: §2 · p. 3

Benchmarks and Evaluation Methodologies

• Existing benchmarks cover short, transactional single-site tasks or open-web tasks with predefined, time-invariant answers, often a single string [8] [13] [17] [21] [39] [40] [46] [23] [34] [36] [37] [41].BrowseComp [36] uses generation-verification asymmetry for hard-to-solve but easy-to-verify answers, sidestepping complex, time-varying evaluation at the cost of deviating from the true user-query distribution; Mind2Web 2 instead uses Agent-as-a-Judge for realistic, complex tasks requiring comprehensive answers with real-time information.Source: §2 · pp. 3–4

LLM-as-a-Judge

• Because agentic-search complexity exceeds what a few LLM calls can evaluate, the paper uses Agent-as-a-Judge [35] [47].Unlike PaperBench [35], whose tree-structured rubric is manually written for AI-research replication, Mind2Web 2 largely automates rubric generation and uses score aggregation beyond simple weighted averaging; attribution evaluation is also related to prior attribution work [9] [19] [42].Source: §2 · p. 4

3.1 Overview

• Mind2Web 2 targets two challenges: collecting sufficiently complex, realistic tasks and reliably evaluating complex answers from agentic-search systems.Its Agent-as-a-Judge framework assesses whether answers meet task requirements and whether each statement is supported by its cited sources.Source: §3.1 · p. 4

3.2 Task Collection

• Tasks are designed to be realistic, diverse, long-horizon, laborious, objective, verifiable, and potentially time-varying.Proposers generate tasks, refinement experts revise or filter them, and at least two validation experts must independently validate each included task.Source: §3.2 · p. 4

3.3 Rubric Tree

• Rubric trees decompose task goals into binary-checkable leaf criteria, with critical nodes gating partial scores and sequential nodes short-circuiting dependent checks.Non-critical nodes allow partial credit, while a parent with only critical children passes only if all pass. Partial Completion averages root scores across tasks; Success Rate is the percentage of tasks with a perfect root score.Source: §3.3 · pp. 4–5, Figure 2 · p. 5

3.4 Rubric-based Judge Agent

• Each task has a dedicated judge-agent workflow that extracts answer details, verifies them against webpages, and aggregates criterion scores up the rubric tree.A reusable Python toolkit and LLM-based code-generation pipeline reduce script-writing effort; generated scripts are iteratively refined and then validated through two stages of human refinement.Source: §3.4 · pp. 5–6

3.5 Benchmark Statistics

• Mind2Web 2 collects 130 carefully curated tasks, and benchmark construction involved at least 1,000 human-labor hours; rubric trees reached up to 6 layers and 603 evaluation nodes.In a study of 30 tasks, seven participants attempted them, with each task completed by three different participants. A task could take up to one hour and require visiting as many as 31 websites and 375 webpages; these figures were underestimated because participants could make mistakes or omit steps, and could stop after one hour or if unable to find a clear path.Source: §3.5 · p. 6
• Mind2Web 2 is the only agentic-search benchmark described here that focuses on long-horizon, time-varying tasks, with dozens to hundreds of evaluation nodes per task.The authors argue that this fine-grained evaluation provides sufficient differentiation despite the benchmark’s 130 tasks.Source: §3.5 · p. 6

4.1 Experimental Setup

• The evaluation compares commercial search products, Deep Research systems, and a browser-interaction agent using Partial Completion, Success Rate, and Pass@3 across three runs per task.It also reports task time and answer length and includes a human performance study on Subset-30.Source: §4.1 · p. 7

4.2 Main Results

• ChatGPT Search [25] and Perplexity Pro Search [30] are weakest, most Deep Research systems perform better, and Operator [27] performs below Deep Research systems.The authors attribute these differences to restricted search horizons and shallow synthesis for commercial search products, and the planning and memory challenges of browser interaction for web agents.Source: §4.2 · p. 8
• Deep Research systems range from concise answers to reports of thousands of words, but greater length does not necessarily improve task completion and may burden users.Multiple independent attempts can also raise the likelihood of success, as reflected in improved Pass@3 scores.Source: §4.2 · p. 8
• Performance improves with increased inference time and multiple trials, while most systems perform worse on 57 explicitly time-varying tasks than on other tasks.OpenAI Operator and human participants, both adept at live website interaction, achieve relatively on-par or superior performance on time-varying tasks; this supports the value of integrating browsing into agentic search.Source: §4.2 · pp. 8–9, Figure 3 · p. 8, Figure 4 · p. 8
• OpenAI Deep Research achieves 50–70% of human performance while spending less than half the time; success rates reach up to 28% for agents and 54% for humans.The authors also find a gap between partial completion and success, indicating that systems may start tasks competently but fail to complete them accurately.Source: §4.2 · pp. 7–9

4.3 Error Analysis

• Error analysis identifies incompleteness and unsupported answers, including missing requested items, incorrect synthesis from relevant sources, and retrieval of irrelevant sources.Systems not optimized for Deep Research may terminate early or lack the capabilities to complete long-horizon tasks; ChatGPT Search and Perplexity Pro Search also struggle to synthesize extensive sources accurately.Source: §4.3 · pp. 9–10
• Criteria violations are common across agents and humans, while attribution errors include fabricated or incorrect URLs and claims lacking sources.Deep Research systems surpass humans on criteria violations, whereas Operator often struggles to provide attribution despite accessing websites.Source: §4.3 · pp. 9–10

4.4 Human Evaluation of Judge Agents

• After excluding human mistakes and a source-inconsistency case, only 7 of 720 nodes were actual Verifier errors, yielding 99.03% correctness for the judge agents.The evaluator agreed with all 15 rubrics, and validation identified 27 of 35 discrepancies as human evaluator errors; four remaining cases involved attribution evidence hidden in collapsed webpage sections.Source: §4.4 · pp. 10–11

5 Conclusions

• Mind2Web 2 is a benchmark for comprehensively evaluating agentic search on long-horizon information-gathering tasks and time-varying answers.The authors present it as a rigorous assessment platform for advancing agentic search systems.Source: §5 · p. 11
• The proposed Agents-as-a-Judge framework is scalable, automated, and reliable for evaluating open-ended, long-horizon search tasks.It systematically assesses agent performance.Source: §5 · p. 11
• The empirical analysis spans AI-based search engines, deep research systems, and web agents, revealing both their potential and current limitations.Source: §5 · p. 11

A Limitations

• Mind2Web 2 covers 130 diverse practical tasks, but excludes some scenarios, including vague or highly subjective queries, so it cannot represent every real-world information-seeking task.The authors argue that the diversity of included domains, websites, and scenarios still provides reasonable coverage and that the exclusions do not significantly diminish the benchmark’s utility.Source: A Limitations · p. 16
• The evaluation assumes cited URLs are truthful and credible and that critical information can be attributed to individual webpages, leaving source credibility assessment beyond its scope.The authors note these constraints have not prevented them from developing a large, diverse, and meaningful benchmark.Source: A Limitations · p. 16
• LLM-based extraction and verification can err, so the authors use human validation, script refinements, and human review of judge-agent outputs to support reliability.They report that OpenAI o4-mini is sufficiently capable for the benchmark’s extraction and verification tasks.Source: A Limitations · p. 16
• Focusing on meaningful commercial and research-grade systems limits interpretation of performance differences and precise inference costs, though task time and answer length offer relative efficiency references.The answer-based evaluation still assesses black-box capabilities and failure modes, including pervasive hallucinations.Source: A Limitations · p. 16

B Broader Impacts and Ethical Considerations

• Agentic search could reduce cognitive load, improve efficiency, broaden access to sophisticated search, and support decisions, but may also spread misinformation, enable disinformation or unauthorized data extraction, and perpetuate bias.The authors present reliable, scalable evaluation as a first line of defense against these risks.Source: B Broader Impacts and Ethical… · p. 16
• Mind2Web 2’s structured rubrics and explicit source-attribution verification aim to foster transparent, accountable systems and standardize capability assessments across research and industry.Source: B Broader Impacts and Ethical… · p. 16
• Widespread rubric-based evaluation could enable mass production of reinforcement-learning data, risking benchmark overfitting and amplified biases; the authors mitigate this by keeping test tasks, rubrics, scripts, and pipeline hidden.Such overfitting could leave agents weak on broader, unstructured real-world scenarios or produce systematic biases.Source: B Broader Impacts and Ethical… · p. 17

C.1 Domain Distribution

• Mind2Web 2 contains 130 tasks across 6 broad domains and 24 sub-domains, with expert review and consolidation used to refine domain assignments.Proposers categorized tasks using domains derived from prior work and expanded with GPT-4o; experts later reviewed and consolidated the categories.Source: Figure C.1 · p. 17, §C.1 · p. 17

C.2 Design Principles of Tasks

• Tasks must address authentic user needs and require sustained web search and synthesis, with at least five minutes of human effort.The five-minute threshold is a minimum; most tasks take humans longer.Source: §C.2 · p. 18
• Task descriptions and answer criteria must be explicit, precise, and unambiguous, with manual and LLM-assisted ambiguity checks.Domain-specific knowledge must be defined or explained in the task description.Source: §C.2 · p. 18
• Tasks need verifiable criteria, primarily through answer text and expected URL provenance, and attribution checks must be decomposable into single-webpage verification.Only a minor part of criteria may rely on alternatives such as external APIs or fixed ground-truth answers.Source: §C.2 · p. 18
• The benchmark excludes video-understanding, non-English, complex-reasoning, external-tool-dependent, rapidly changing, and unverifiable global-qualifier tasks.Systems are not restricted from using tools; the exclusion concerns tasks that explicitly require external tools, and global qualifiers are allowed when verifiable from fixed sources or ground truth.Source: §C.2 · p. 18

C.3 Task Collection Pipeline

• Proposers independently generate task ideas, self-check them against major and minor principles, and provide draft answers or relevant URLs.Self-checks include realism, tediousness, clarity, verifiability, grammar, and clarity.Source: §C.3 · p. 18
• Experts iteratively refine each proposed task for practicality, clarity, and adherence to the design principles.Refinement may adjust task descriptions, verification criteria, or expected answers while preserving realism, challenge, and objective verifiability.Source: §C.3 · p. 18
• Only tasks passing validation by at least two independent annotators enter the benchmark.Validators fully complete each task and check feasibility, ambiguity, edge cases, and URL-based evaluation assumptions; failing tasks are revised or rejected.Source: §C.3 · p. 19

C.4 Future Maintenance of the Benchmark

• Broad goals, source flexibility, final-information verification, and judging of time-varying answers make Mind2Web 2 less sensitive to website changes than prior benchmarks [29] [39].Unlike benchmarks tied to specific websites or action sequences, it evaluates final retrieved information rather than intermediate interactions.Source: §C.4 · p. 19
• The authors will periodically review tasks and update or replace those substantially affected by website changes or unavailability to preserve their intended difficulty and scope.They will also solicit feedback from benchmark users.Source: §C.4 · p. 19

D.1 Rubric Design

• Rubrics permit partial credit only when it reflects useful incremental progress, and require URL attribution as critical when such citations are reasonably expected.The framework aims to provide a unified, scalable evaluation method across Mind2Web 2 tasks and potential future tasks.Source: §D.1 · p. 19

D.2 Details for Judge Agents

• Judge agents use reusable rubric-tree, scoring, Extractor, and Verifier components, with short-circuiting that skips blocked or sequentially subsequent checks.Short-circuiting is disabled during human evaluation so all nodes can be compared with human annotations.Source: §D.2 · p. 19
• Simple verification returns a binary correct-or-incorrect judgment, while URL-based verification checks whether a claim is supported by the cited webpage text or screenshot.For URL-based checks, an irrelevant, invalid, or inaccessible source means the claim is unsupported; judgments should rely primarily on explicit webpage evidence.Source: §D.2 · pp. 20–21

D.3 Rubric and Judge Agent Generation

• Claude-3.7-Sonnet generates initial judge-agent scripts from task descriptions, rubric and evaluation guidance, toolkit functions, and examples of common mistakes.The authors report that iterative debugging and self-reflection are indispensable because initial scripts often need multiple refinement rounds for correctness and completeness.Source: §D.3 · pp. 21–22
• The generation pipeline iteratively runs scripts to find runtime errors, then uses checklist-guided self-reflection to review correctness, coherence, rubric completeness, and edge cases.During runtime debugging, OpenAI Deep Research answers provide extractor inputs while verification steps are omitted.Source: §D.3 · p. 22

D.4 Two-Stage Validation of Judge Agents

• Validation first has trained annotators inspect each script’s rubric for correctness, completeness, and feasibility, with complex sequential and parallel criteria sometimes needing manual adjustment.Source: §D.4 · p. 22
• Next, annotators review outcomes on one randomly selected answer from each of six randomly selected systems per task, make targeted critical fixes, and check generalization on held-out answers.The held-out answers are also used in the human agreement study; prompt and edge-case adjustments were often needed.Source: §D.4 · pp. 22–23

D.5 Human Evaluation of Judge Agents

• A human evaluator familiar with the tasks but not the judge agents assesses rubric-level agreement and manually assigns binary scores to leaf nodes across 15 sampled tasks.The evaluator had participated in error analysis; o4-mini was found capable of serving as Extractor and Verifier, and the agents had already undergone two-stage validation and refinement.Source: §D.5 · p. 23

E.1 System Selection and Settings

• The evaluation spans search-API systems, browser-interacting web agents, hybrid systems, and other forms, while excluding systems unable to reliably attribute sources and systems deemed too weak for the benchmark.Reliable source attribution is treated as integral to the evaluation.Source: §E.1 · p. 23
• Each agent is run three times per task; answers are collected manually from web UIs except for Hugging Face Open Deep Research, and its base model is OpenAI o3.Answers were collected between April and June 2025; completion times are recorded when available, with manual timing for Perplexity Pro Deep Research and Gemini Deep Research only on Subset-30.Source: §E.1 · p. 23
• OpenAI Operator and Gemini Deep Research occasionally omit sources for retrieved information, so their prompts are modified to mitigate this issue.The general prompt emphasizes source attribution; the two systems’ prompts specifically require a unique source URL for each list item.Source: §E.1 · pp. 23–24

E.2 Webpage Pre-caching for Evaluation

• For evaluation, the team pre-fetches and caches answer-linked webpages, including PDFs, to provide quick, consistent verification against source content as it appeared when answers were generated.URLs are aggregated per task and unique URLs are loaded using Playwright; human annotators can manually retrieve blocked pages and replace incorrect cached content.Source: §E.2 · p. 24

E.3 Human Performance on Subset-30

• Human completers independently solve Subset-30 tasks by browsing relevant websites and providing explicit URL-based sources for each claim or statement.Each task is assigned to three completers without prior task knowledge, excluding task creators and reviewers.Source: §E.3 · pp. 24–25
• Completers may give up if they have no clear solution path after 30 minutes, or after more than an hour of continued effort on a tedious path.Seven completers participate in total, and an open-source Chrome extension logs their time and visited webpages for subsequent analysis.Source: §E.3 · p. 25
• Only completers who meet quality expectations after two simplified Mind2Web 2 trial tasks participate in the formal human study.The trials check that completers can follow the instructions.Source: §E.3 · p. 25

• Figure F.1 shows the workflow for categorizing errors in the error analysis.Source: Figure F.1 · p. 26

F.1 Error Analysis

• Correctness failures include incompleteness—unfound information or fewer requested items or steps—and criteria violations that contradict task criteria or provide incorrect facts.Source: §F.1 · p. 26
• Attribution failures include invalid URLs that are expired, malformed, or fabricated, and missing URLs that leave claims unsupported.Source: §F.1 · p. 26
• Unsupported answers arise when URLs fail to substantiate claims: synthesis errors misrepresent useful information in a source, while retrieval errors use irrelevant URLs.Source: §F.1 · p. 26
• Annotators reviewed one randomly selected answer per task from five representative agent systems and from humans.The systems were ChatGPT Search, Perplexity Pro Search, HF Open Deep Research, OpenAI Deep Research, and OpenAI Operator.Source: §F.1 · p. 26

F.2 Additional Case Studies

• OpenAI Deep Research has a 23% hallucination rate, versus at least 50% for other systems; the rate likely underestimates hallucination because it counts only two error types.The counted types are Invalid Attribution and Unsupported Answer; hallucination may also contribute to other errors.Source: §F.2 · p. 31
• Human answers fully met completeness requirements and had no hallucinated webpage URLs, yet annotators made careless constraint, extraction, spelling, and common-sense errors.Examples include misclassifying the University of Waterloo as a U.S. university and misspelling a Pritzker Prize winner’s name.Source: §F.2 · p. 31, Figure F.9 · p. 32, Figure F.10 · p. 32
• Hugging Face Open Deep Research often ended prematurely after tool-input or generated-code mistakes, and its reliance on untailored off-the-shelf models may contribute to these failures.The illustrated case is an Information Not Found error after failure to invoke the search tool as instructed.Source: §F.2 · p. 31, Figure F.11 · p. 33
• Operator’s poor performance may reflect weak long-term reasoning, planning, or grounding and insufficient long-term memory for retrieved information and sources.The case study also shows an invalid attribution despite visiting the correct source.Source: §F.2 · p. 31, Figure F.12 · p. 34

G Example of Judge-Agent Scripts

• The public development-set script asks for a repository commit’s short ID, date, and authors’ GitHub profile links and displayed real names.Its extraction prompts specify source URLs for commit information and require extracting author information as stated in the answer.Source: G Example of Judge-Agent Scripts · pp. 38–39
• The judge verifies commit information in sequence, with parallel checks for ID existence, correctness, and provenance, and a separate date check.ID existence and provenance are marked critical, while the sequential verification node is non-critical to permit partial scoring.Source: G Example of Judge-Agent Scripts · pp. 39–40
• The commit-date check allows format variations, including Dec 7, 2023, December 7, 2023, and 2023-12-07, but the core date must match the expected Dec 7, 2023.Source: G Example of Judge-Agent Scripts · p. 41
• The script verifies up to five author positions in parallel, padding missing positions with empty author records.Each position is then passed to a single-author verification routine.Source: G Example of Judge-Agent Scripts · pp. 41–42
• Author names are critical checks, while profile-page verification is non-critical and permits partial scoring across authors.The script also marks missing author information as non-critical, but treats a missing name as critical for that author slot.Source: G Example of Judge-Agent Scripts · pp. 42–43
• Commit information is checked before author information, which is skipped when the commit check fails.The commit stage covers the commit ID, date, and provenance.Source: G Example of Judge-Agent Scripts · p. 43

H.1 Instructions for Task Collection

• Task proposals should describe realistic, information-seeking, multi-step tasks taking at least five minutes, with clear single-round requirements and verifiable answers.Proposals should avoid logins, quickly resolvable tasks, uncheckable global qualifiers, and vague or subjective elements.Source: §H.1 · p. 45
• Refinement checks that tasks are realistic rather than artificially complex, clearly worded, objective, feasible on public sites, and supported by an answer outline for correctness and attribution checks.The outline should capture critical and reasonably expected information; each fact should be supported by at least one URL.Source: §H.1 · p. 45
• Validation requires completing tasks end-to-end and confirming sources directly support expected statements while excluding inaccessible, unstable, or login-gated information.Annotators should document sources, necessary ground truth, and evaluation subtleties, and discuss borderline tasks with the team.Source: §H.1 · pp. 46–47

H.2 Instructions for Human Performance Study

• Human performance participants browse for answers, link each critical claim to a verifiable URL, and record their task-completion process for review.They use a Google Doc as a text pad and must actively search rather than rely on prior knowledge.Source: §H.2 · pp. 47–48
• The human study restricts task work to browser-based research without AI tools and requires exporting browsing-time data and submitting the answer and recording.Participants should notify the team when a task takes under five minutes or over one hour; tasks may be stopped after 60 minutes.Source: §H.2 · p. 48

H.3 Instructions for Error Analysis

• Annotators first read the task and response, then assess correctness and attribution, comment on each identified error, and record its type in the annotation sheet.The workflow is Read, Evaluate, Comment, Collect.Source: §H.3 · p. 49
• Incompleteness covers noticeable omissions of requested content, not subtle or long-range reasoning failures, and includes explicit failure to find information or providing fewer items or steps than requested.The instructions distinguish “Information Not Found” from “Partial Completion.”Source: §H.3 · p. 49
• Criteria Violation applies when an answer breaks explicit task constraints or gives an incorrect answer where the task specifies a ground-truth reference.Examples include exceeding a price limit or naming the wrong follow-up paper.Source: §H.3 · p. 49
• Attribution checks whether each key claim has a valid source, whether the cited page is relevant, and whether it supports the claim.A URL may be invalid if expired, malformed, or fabricated; attribution is missing when no source URL is supplied for a claim.Source: §H.3 · pp. 49–50
• Unsupported Answer covers reachable sources that fail to support a claim: Retrieval Error uses irrelevant sources, while Synthesis Error misquotes or misinterprets useful source information.Examples contrast a source lacking the requested information with an answer misstating a source’s price or tracklist.Source: §H.3 · p. 50

H.4 Instructions for Human Evaluation

• Human evaluators independently assess rubric quality and judge-agent reliability in two phases: rubric-level and node-level assessment.Source: §H.4 · p. 50