PaperlayerView exact revision on arXiv

Source-linked AI summary

WANDR: A Benchmark for Wide and Deep Research

Vitaliy Polshkov, Marcin Pitera, Jeremy Yang, Kirill Priemko, Maksim Gaiduk, Aleksandr Nikolenko, Denis Bykov, Clare Southern, Denis Yarats, Jerry Ma

arXiv:2608.14747v1cs.LG

TL;DR

Research agents increasingly need to discover many entities and deeply verify structured facts, but existing systems struggle with breadth, depth, and complete evidence. WANDR benchmarks these capabilities across 500 tasks and finds performance degrades sharply with scale, with the leading system reaching only 0.363 soft F1 and 0.133 hard F1.

  • Problem

    Research agents must produce broad, deeply verified collections of structured facts, yet production runs show recurring failures in discovery, constraint application, corroboration, and evidence support.

  • Method

    WANDR encodes tasks as compositional qualification hierarchies and grades citation-backed records with judges that refetch pages and verify submitted excerpts.

  • Results

    Performance degrades sharply with larger targets and deeper hierarchies; across six systems, the leader reaches 0.363 soft F1 and 0.133 hard F1.

  • Takeaways & Limitations

    WANDR makes large-scale evidence-backed research measurable and identifies substantive page support and complete excerpt support as major bottlenecks.

  • Takeaways & Limitations

    WANDR’s tasks represent recurring de-identified production requests rather than a random sample of all research work, so scores should not represent deployment-wide difficulty.

Abstract

from arXiv · show

WANDR (Wide ANd Deep Research) is a benchmark of 500 realistic, challenging data-collection tasks for research agents. Each task requires a system to discover a large set of entities that satisfy specified criteria (breadth), investigate each entity through multiple coordinated web searches (depth), and return independently verifiable records with supporting sources and excerpts. Tasks are represented as qualification key hierarchies that specify the entities, relationships, evidence, and required count at each level; a hierarchy with n companies, m employees per company, and k sources per employee requires n x m x k records. This structure supports diverse workflows such as market mapping, due diligence, literature review, product comparison, and talent sourcing, with targets ranging from dozens to thousands of records. WANDR replaces static gold answer sets with task-specific judges that refetch cited pages and verify each record against its evidence, allowing evaluation of current and changing facts. Record verdicts are aggregated into soft and hard precision, recall, and F1 scores that distinguish factual quality, coverage, and hierarchical completeness. The tasks are derived from de-identified product-usage logs and produced through a semi-automated pipeline with automated checks, empirical audits, and human review where needed. We evaluate six production research systems and find that the benchmark is far from saturated: at high effort, the strongest system reaches only 0.363 soft F1 and 0.133 hard F1. Performance degrades as target volume and hierarchy depth increase, with incomplete discovery, missing enrichment, and incomplete evidence construction remaining major bottlenecks. The benchmark and evaluation harness are available at https://github.com/perplexityai/wandr.

1 Introduction

WANDR benchmarks wide-and-deep research tasks that require discovering structured collections of entities and verifying each fact through coordinated web evidence. Its hierarchical task representation, semi-automated construction, and reference-free grading expose where research systems fail and support scalable evaluation and reinforcement learning.

  • Motivation: WANDR captures production knowledge-work tasks requiring both broad entity discovery and deep, independently verifiable investigation.The benchmark targets competitive-landscape mapping, deal due diligence, literature review, market and product comparisons, and talent sourcing.
  • Task representation: Qualification key hierarchies encode task structure as trees of identifying keys and filters, with n × m × k required records when levels require multiple children and sources.Each record cites a live page and includes verbatim excerpts, making it independently checkable.
  • Benchmark construction: WANDR uses production-grounded tasks and a semi-automated pipeline with authoring checks, feasibility audits, judge audits, and optional human quality review.Human effort supports quality review rather than exhaustive gold-answer construction.
  • Failure analysis: The benchmark localizes failures across discovery, enrichment, page identification, evidence extraction, and identity disambiguation.Flat lists and fixed-schema tables are special cases of the compositional hierarchy.

2 Related Work

Prior benchmarks differ by output shape, grading method, and domain specialization. WANDR targets evidence-backed, open-set collections with compositional qualification hierarchies, combining broad discovery with per-record accuracy.

  • 2 Related Work: Prior information-seeking benchmarks are organized by compact-answer versus collection outputs, grading methods, and domain specialization.Both output formats may restrict tasks to a single expert field.
  • 2 Related Work: Closed-ended benchmarks require bounded browsing or question answering that produces a compact response, often involving obscure questions, expert knowledge, or multi-hop reasoning.Examples include BrowseComp, Humanity’s Last Exam, FRAMES, GAIA, and AssistantBench.
  • 2 Related Work: Collection benchmarks most closely related to WANDR grade discovered items individually, but typically emphasize broad collection, shallow fields, or matching gold answer sets.WideSearch grades populated tables cell by cell, while DeepSearchQA supports single- and set-valued answers against gold sets.
  • 2 Related Work: Prior grading paradigms include gold-label matching, rubric-based LLM judging of reports, and gold-collection matching; WANDR is the breadth-oriented counterpart to DRACO.Some related work also makes the evaluator itself agentic.
  • 2 Related Work: Relative to collection benchmarks, WANDR adds evidence-backed records, reference-free verification, and flexible qualification-key hierarchies for discovery and per-entity enrichment.Its tasks derive from real professional usage and are admitted only when high-volume, feasible, and discriminative against internal baselines.
  • 2 Related Work: WANDR challenges agents to sustain breadth and per-record accuracy across large, structured collections, the regime in which current agents struggle.This challenge combines wide discovery with accurate records throughout the collection.

3 Task Structure

WANDR models wide-and-deep research tasks as composable trees whose levels, relationships, quotas, subtasks, and URL leaves define the records to collect and verify. This structure covers recurring workflows through patterns such as nesting, fan-out, dispatch, anchors, partitions, corroboration, and triangulation.

  • Composability: A hierarchy can combine discovery, enrichment, containment, related-entity fan-out, fixed labels, repeated child sets, and subtasks at shared keys.These variations are composable rather than mutually exclusive, allowing one representation to cover diverse workflows.
  • Core representation: WANDR represents each task as a tree whose levels define collected items, edges define parent–child relationships, counts specify required coverage, and branches end in supporting URLs.A root-to-leaf path forms one record containing an identifying item, source URL, excerpts, and an answer.
  • Illustrative task: The illustrated task requires 140 records: 70 appointment records plus 70 listing records aligned on the same company key.Its subtask adds one listing-authority URL per company alongside the primary appointment branch.
  • Hierarchy patterns: Seven recurring structural patterns include flat lists, nesting, fan-out, matrices, dispatch, anchors, and partitions.Examples range from countries containing cities to fixed facet labels, shared intermediate products, and geographic partitions before company collection.
  • Evidence structure: Corroboration requires multiple independent pages to support the same claim, whereas triangulation uses several pages to jointly describe a quantity or market surface.Tasks may also separately require source or domain independence for corroboration.

4 Task Construction

Task construction uses a four-stage semi-automated pipeline that seeds wide-research requests from de-identified logs, iteratively authors and critiques task packages, audits feasibility and judging, and curates the released set. Automated gates, optional human review, and rollback to the responsible stage enforce runnability, cleanliness, difficulty, discrimination, and quality.

  • Pipeline overview: The pipeline comprises seeding, iterative authoring, admission into a candidate pool, and curation into the released set.Failures return to the stage that owns them.
  • Seeding: Seeds are real requests mined from de-identified product-usage logs, filtered for wide-research structure, stable entities and sources, checkable answers, and difficult discovery.Rejected seeds remain negative examples, while deduplication and coverage steering promote diverse, economically useful domains.
  • Authoring: Authoring interleaves separate author and critic agents across design, task, and fixture drafts until checks pass or the work returns to an earlier stage.The package includes a key hierarchy, solver-facing description, judge specification, schema settings, and fixtures grounded in real pages.
  • Quality gates: Four automated checks require runnability, cleanliness, difficulty, and discrimination before admission.These checks verify package execution and alignment, prevent primitive baselines from solving the task, and require stronger rollouts to outperform weaker ones.
  • Admission: 10–12 authoring rollouts are merged into a best-known solution that must recover near-full requested volume during the feasibility audit.The merged output establishes attainability rather than serving as an answer key; submissions remain graded against their own cited pages.
  • Admission: Blind regrading forms an independent verdict, and an automated-judge error rate above approximately 10% of the remaining sample blocks the task.Optional human review scores four pass/fail axes covering hierarchy and volume, description, grader encoding, and identity handling.

5 Grading

WANDR grades independently verified leaf records with task-specific and universal checks, then aggregates quality and coverage bottom-up using soft and hard precision, recall, and F1. Identity resolution, confidence thresholds, and page refetching make evaluation robust but sensitive to missing evidence and changing URLs.

  • Record grading: Each leaf record receives a binary verdict after the grader refetches its cited page and checks page usability, claim clarity, excerpt faithfulness, eligibility, and evidence support.A complete record passes only when every applicable universal and task-specific condition holds.
  • Limitations: Confidence below 2 excludes a leaf from precision and can reduce recall, while URL changes between solving and grading can cause otherwise correct records to fail.Grading re-fetches cited pages, making close temporal proximity between solving and grading important.
  • Identity resolution: Identity resolution canonicalizes closed axes and semantically deduplicates open-ended axes before aggregation, preventing cosmetic variants from satisfying volume floors repeatedly.Resolution is performed independently along each key axis, including across parent tasks and subtasks.
  • Metrics: Leaf scores aggregate bottom-up into soft and hard precision, recall, and F1, with duplicate identities collapsed by worst score, top-k retention, and zero-padding for shortfalls.Precision averages supplied child scores; recall is normalized by the required count, and F1 is the harmonic mean of precision and recall.
  • Metrics: The benchmark reports an unweighted mean of each per-task metric, so every task contributes equally regardless of its required record count.Soft metrics award partial credit, whereas hard metrics require fully correct submitted subtrees or members.
  • Failure localization: Precision captures submitted-member quality, recall captures breadth relative to the target, and the verdict_retrieval–verdict_full gap captures losses from evaluating the complete record.Low precision can reflect depth failures such as incorrect facts, unfaithful excerpts, or missing corroboration.

6 Experiments and Results

Across systems and effort settings, performance remains unsaturated: Perplexity leads overall, while increasing target volume and hierarchy depth exposes major discovery, enrichment, completeness, and evidence-selection bottlenecks. Search as Code supports the repeated retrieval structure, but identity and rendering failures remain unevenly concentrated across systems.

  • Overall comparison: Perplexity leads at 0.363 soft F1 and 0.133 hard F1, with midrange resource use, while no system dominates both performance and resource use.Perplexity’s cost is $5.20 per task, with 14.9-minute median latency and 3.82M total reported tokens per task; Exa is fastest at 5.3 minutes.
  • Effort settings: Perplexity xhigh reaches 0.447 soft F1 and 0.224 hard F1, while available systems span costs from $0.03 per task for Exa low upward.Perplexity, Gemini, and Exa improve monotonically across available settings; OpenAI peaks at high, while Parallel’s hard F1 dips slightly between ultra2x and ultra4x.
  • Scaling and bottlenecks: Soft recall is lower than soft precision for every system, with Perplexity’s gap smallest and Anthropic’s largest.Perplexity retains 0.357 recall from 0.389 precision, whereas Anthropic falls from 0.354 to 0.222; recall penalizes duplicate identities and shortfalls against required counts.
  • Scaling and bottlenecks: Perplexity falls from 0.363 soft F1 to 0.133 hard F1, showing that many submitted members have incomplete required descendants and checks.Anthropic falls from 0.249 to 0.072, while remaining systems fall from 0.055–0.121 to 0.009–0.035; SaC leads every hard metric.
  • Evidence construction: Complete evidence support is the dominant rendering weakness: faithful excerpts support every requirement on only 55.6%–80.4% of records.After pages are fetched and judged plausible, 33.6%–68.3% of records still cite pages failing a substantive requirement; complete support also falls 6.8–21.8 percentage points as checks increase from two to five.

7 Discussion

WANDR measures large-scale, independently verified research and shows that current systems remain far from solving it, while exposing coverage, evidence, scaling, and grading limitations. The benchmark also offers a potential reinforcement-learning substrate and motivates controlled future experiments.

  • Benchmark contribution: WANDR combines open-ended breadth, per-record depth, and qualification hierarchies that separately expose discovery, enrichment, and evidence failures.Its 500 tasks target structured collections whose members require independent verification.
  • Evaluation results: 0.363 soft F1 and 0.133 hard F1 are the leading scores of Perplexity Search as Code across six production systems.Its hard recall is 0.134, showing that even the leading system remains far from solving the benchmark.
  • Failure analysis: Complete performance drops sharply with larger targets and deeper hierarchies, while discovery, enrichment, extraction, identity disambiguation, and excerpt support reveal distinct losses.The framework exposes both precision–recall and soft–hard gaps, and substantive page support and complete excerpt support remain bottlenecks.
  • Future directions: WANDR could support reinforcement learning through held-out training packages, quota-based curricula, dense record- and hierarchy-level rewards, and shared grading work across batched rollouts.Appendix J describes this design and its caveats.
  • Limitations: The released tasks are a stress test rather than a random sample, and one-run cross-system comparisons cannot estimate variance or isolate causal mechanisms.Reference-free grading also faces judge, retrieval, page-drift, extraction, canonicalization, and shared-backend alignment uncertainties.
  • Future directions: Future studies should vary workload, models, interfaces, orchestration, and retrieval components independently through matched comparisons and controlled ablations.They should also test reinforcement-learning reward designs, quota curricula, and grading savings while preserving held-out evaluation.

Appendices … C Task Package Format

The appendices define WANDR’s record-level grading, recurring evidence requirements, identity-resolution policies, and self-contained task-package format. Together, these mechanisms determine how submitted records are judged, deduplicated, scored, and distributed for reproducible evaluation.

  • A Per-Record Grading Criteria: Each record is judged through universal checks, task-specific validity checks, and substantive requirements evaluated from the submitted item, answer, URL, excerpts, and fetched page.Substantive requirements separately assess full-page satisfaction and whether submitted excerpts independently support the claim.
  • A.1 Layered Checks: Page-level satisfaction and excerpt-level support are distinct: faithful excerpts can omit required facts, while stitched text can be complete but unfaithful.Universal checks also require a usable page and a clear answer; invalid records may short-circuit substantive judgment.
  • A.1 Layered Checks: The record-level verdict distinguishes full-page requirement retrieval from complete-record validity, usability, answer clarity, faithful excerpts, and complete excerpt support.A task-specific validity failure can short-circuit an ineligible record, while page-level satisfiability is not simply a mechanical conjunction of every preceding check.
  • A.2 Common Evidence Patterns: Common evidence patterns require plausible entities, verified eligibility, appropriate and dedicated sources, and sometimes long-tail or heavy artifacts such as reports, registries, filings, and PDFs.Other tasks require substantive, broad, scattered, or structured extraction rather than ratings, isolated mentions, or incorrectly read table cells.
  • B Identity Resolution: Identity rules prevent cosmetic variants from satisfying volume floors repeatedly, while keeping leaf verdicts independent and applying one sameness policy per key axis for cross-record scoring.Identity resolution makes required volumes, coverage, and score rollups well defined.
  • B.1 Canonicalization and Deduplication: Canonicalization standardizes predictable values, whereas deduplication determines whether open-ended entities such as people and companies refer to the same entity.Both mechanisms are configured per key axis, with composite keys supplying context when a single field is ambiguous.
  • B.2 Common Identity Patterns: Identity policy defines the task’s counted entities: granularity can merge or separate locations and subsidiaries, while one referent can span renames, abbreviations, rebrands, or event stages.Finite canonical sets can require exact-set coverage, with aliases mapped to allowed members and out-of-roster values excluded.
  • C Task Package Format: Each released WANDR task is a self-contained package containing package files, prompts, fixtures, and verifier code in the public repository.The package format provides the materials needed to run and verify a task.

D Task Admission Gates and Review

Task packages enter the released set only after passing implementation, difficulty, feasibility, discrimination, and judgeability gates, followed by curation from the admitted pool. Human reviewers inspect authoring rollouts and trajectories to detect shortcuts and request hardening before admission.

  • Admission gates: A task must be runnable, convention-clean, sufficiently difficult, discriminative, empirically feasible, and judgeable before release.These requirements are specified in the authoring prompts and linter instructions.
  • Admission gates: Curation follows admission and selects tasks from the admitted pool to match release-level distributions.
  • Human review: Human review is governed by a dedicated rubric for task packages.
  • Human review: Reviewers inspect authoring rollouts and trajectories to expose degenerate strategies and shortcuts not apparent from packages or aggregate scores.Tasks can be returned for hardening before admission.

E Empirical Task Statistics

WANDR’s empirical task statistics characterize difficulty across discovery, enrichment, and extraction, while also measuring citation reuse, source ecology, and authoring effort. Across 500 tasks and 490 retained trajectories, extraction and enrichment difficulty co-vary, effective citation volume varies widely, and exploration scope is noisily associated with cost.

  • Task-ontology difficulty: Task difficulty is decomposed into discovery, enrichment, and extraction based on delivered-member fractions and retrieval-to-full-recall conversions.Figure 14 reports these dimensions over 500 tasks using rollout means.
  • Task-ontology difficulty: r = −0.185 links discovery and enrichment difficulty, while exact-zero cases occur for 24 discovery tasks and one enrichment task.The inverse association is measured in log-odds space among positive pairs; no task has exact-zero values for both.
  • Citation volume and source ecology: 1.97 effective records per member is the median, with the central 90% spanning 0.73–5.92; median domain dispersion is 53.4%.Effective records multiply required records by the square root of URL nonreuse, while domain dispersion is unique domains divided by unique URLs.
  • Trajectory effort and execution: 496 unique URLs is the median surfaced by retained authoring trajectories, compared with a median task mean cost of $4.00.The analysis retains 490 non-stub trajectories and excludes ten with fewer than 80 surfaced URLs.
  • Trajectory effort and execution: r = 0.280 is the positive but noisy association between exploration scope and effort in log–log space.Exploration scope is measured using surfaced URLs, including URLs touched through tool calls but absent from final submissions.

F Exemplar Task Coverage

The five exemplar tasks illustrate varied benchmark shapes, spanning GPU-game performance and enrichment, Qatar dessert-brand analysis, jurisdictional guardianship research, reciprocal partnerships, and time-indexed community sentiment. Each task defines structured records with source, validity, corroboration, or role-specific evidence requirements.

  • GPU-game benchmarking: The GPU task requires at least 20 GPUs and 10 games per GPU, with FPS evidence plus current prices and review ratings, totaling 290 records.Each GPU-game result needs one benchmark URL; each price and rating requires three URLs.
  • Qatar dessert brands: The Qatar dessert task covers 100+ brands across four facets—social identity, sentiment, delivery commerce, and positioning—for 400 required records.Each facet must use a focused, facet-appropriate source tied to a real locally operating Qatar brand.
  • Adult guardianship programs: The guardianship task covers all 30 listed jurisdictions, identifying each institutional fallback arrangement and five legal signals across comparison areas.Root records require authoritative statute, agency, or program evidence, while each legal signal needs its own dispatch-specific source.
  • Reciprocal partnerships: The partnership task targets at least 100 developer-tools or DevOps companies and three reciprocal partners per company, requiring quote and backquote URLs for each ordered pair.References must appear on officially controlled surfaces, be substantive, and involve distinct genuine companies.
  • Community-sentiment panels: The LLM-sentiment task identifies 10+ companies and supplies two first-hand discussion pages per company-month-site cell across three sites and a fixed 12-month window.Pages must be dated to the target month, hosted on the claimed site, standalone, and independently corroborated.

G Stylized Metric Rollup Algorithm · H Worked Rollup Walkthrough

The metric rollup recursively scores hierarchical tasks using configurable aggregation, continuity, and verdict choices, while the walkthrough shows how completeness, surplus entities, identity collapse, and failed subtasks affect final scores. The toy composite task yields soft F1 0.57 and hard F1 0.33.

  • G Stylized Metric Rollup Algorithm: The rollup recursively follows task topology, selecting leaf criteria scores and composing subtask scores according to entity identity and required volume.Its parameters choose precision or recall aggregation, soft or hard continuity, and the verdict used for leaf judgments.
  • G Stylized Metric Rollup Algorithm: At each hierarchy level, duplicate entity values are clustered, the worst-scoring required-volume entries are retained, missing entries are zero-padded, and scores are averaged.Subtask scores are multiplied into the parent score, while overall-task averaging omits thresholding and composition.
  • H Worked Rollup Walkthrough: The toy task requests three countries, two qualifying cities per country, one supporting page per city, and additional heritage-site and country-level evidence branches.Its main hierarchy is country (3) →city (2) →url (1), with a per-city heritage_site (2) →url (1) branch and two per-country URL-only branches.
  • H Worked Rollup Walkthrough: soft F1 0.57 and hard F1 0.33 summarize the toy composite task: France is complete, India receives partial credit, and Turkey fails a required subtask.These values are the overall scores produced by the released grader.
  • H Worked Rollup Walkthrough: France is fully complete because both qualifying cities have two heritage sites and both country-level subtasks pass, making all displayed precision and recall values one.The walkthrough explicitly reports that all displayed precision and recall values equal one.
  • H Worked Rollup Walkthrough: India’s Delhi supplies one of two required heritage sites, yielding soft recall 0.5 and soft precision 1 after zero-padding.When three cities are supplied for a required count of two, recall retains the two highest-scoring cities, excluding Bangalore because its empty heritage subtask scores zero.
  • H Worked Rollup Walkthrough: Turkey’s merged Selimiye Mosque variants retain an ambiguous zero-scoring citation, lowering Istanbul’s recall; merged city aliases propagate it, while a failed evidence subtask sets the country score to zero.Multiplicative composition gates the country score because the evidence concerns a continental final rather than a worldwide sporting event.

I Output-Delivery Ablation

The output-delivery ablation descriptively associates output-token delivery with lower soft and hard F1 than file delivery across Perplexity, Anthropic, and OpenAI. The soft F1 loss decreases as file-delivery performance declines, though the comparison is not causal.

  • Experimental Design: 45 tasks: all output-delivery ablation rows use the same task set and score and operational columns as the full sweep.The rollouts are independent rather than paired deterministic executions, so differences are descriptive associations with delivery configuration and particular rollout.
  • Output-Delivery Comparison: Output-token runs have lower soft F1 than file-delivery runs for Perplexity (0.397 versus 0.238), Anthropic (0.262 versus 0.202), and OpenAI (0.153 versus 0.122).The same ordering holds for hard F1: 0.156 versus 0.121, 0.099 versus 0.080, and 0.073 versus 0.068.
  • Output-Delivery Comparison: 0.159, 0.060, and 0.031: the absolute soft F1 loss decreases monotonically with file-delivery score across Perplexity, Anthropic, and OpenAI.The passage characterizes this three-system association as descriptive rather than causal.

J Suitability for Reinforcement Learning

Although WANDR is released primarily as an evaluation benchmark, its task-generation and grading design supports future reinforcement-learning training splits and harnesses. The design provides scalable task variation, explicit curricula, decomposable rewards, and amortized batch grading.

  • Scalable task supply and diversity: The semi-automated author–critic–linter pipeline can generate new tasks and controlled siblings across domains, hierarchy shapes, evidence requirements, traps, and breadth/depth settings.Curation can steer the resulting distribution, creating a path toward training-scale data with broader variation than repeatedly sampling the 500 released tasks.
  • A natural curriculum: Explicit count requirements enable curricula that increase task workload across epochs and stage breadth and depth separately.For example, a task can require 5 rather than 20 qualifying entities, or one rather than three corroborating sources, while holding semantic criteria fixed.
  • Dense, decomposable reward: Per-record retrieval and full-verdict scores aggregate into soft and hard precision, recall, and F1 across entities and hierarchy levels.Soft metrics reward correct partial progress, while hard metrics maintain pressure to complete whole members; the verdict gap separates page-level task satisfaction from full-record correctness.
  • Amortized batch grading: Reference-free verification enables amortized batch grading by processing multiple task roots through a streaming queue graph with stage-specific concurrency.The grader batches fetches and performs fetch, triage, canonicalization, deduplication, and judgment before score rollup.
Loading 2608.14747v1…