Source-linked AI summary
Q2D-Web: A Large-Scale Benchmark for Retrieval in Agentic RAG Systems
Maximilian Schall, Sedigheh Eslami, Markus Krimmel, Antoine Chaffin, Louis Milliken, Bo Wang, Denis Bykov
TL;DR
Existing benchmarks do not jointly provide web-scale collections, many production agent-reformulated queries, and deep relevance judgments for first-stage retrievers. Q2D-Web addresses this gap with a 190M-document corpus, 70k queries, three judgment sets, and subcorpus evaluation, finding stable model rankings across judgment sets and preserving full-corpus ordering with a one-third RRF subcorpus.
Problem
Existing benchmarks lack the combined scale, production agent-reformulated queries, and deep relevance supervision needed to evaluate first-stage retrievers in large-scale agentic RAG.
Method
Q2D-Web pairs a ∼190M-document web corpus with ∼70k agent-reformulated queries in ten languages and three judgment sets from citations, production rankings, and LLM judgments.
Results
A ∼30% RRF-selected subcorpus preserves full-corpus ranking order under Combined judgments, while 13 retrievers show largely stable rankings across judgment sets.
Takeaways & Limitations
Q2D-Web supports large-scale evaluation and faster retriever iteration while retaining the full-corpus system ordering under Combined judgments.
Takeaways & Limitations
Subcorpus sampling inherits bias from the retrievers used to construct it, so temporal holdout is used to assess generalization to non-contributing models.
Abstract
from arXiv · showhide
Evaluating first-stage retrievers in large-scale production RAG requires a benchmark that pairs a large-scale corpus with a large set of agent-reformulated search queries based on real user queries and their conversation threads, and that labels many relevant documents per query. No existing public benchmark evaluates this setting: large-scale collections typically provide only a small number of evaluation queries, whereas benchmarks with many queries generally contain only millions of documents. Moreover, most benchmarks assess human-written queries, while the first-stage retrievers in agentic RAG pipelines serve machine-written reformulations whose distribution differs from human search behavior. To overcome these evaluation gaps, we introduce Q2D-Web (Query2Doc-Web), a large-scale agentic retrieval benchmark consisting of a 190M-document web corpus and 70k agentic search queries in ten languages, reformulated from real-world user queries in production systems. Q2D-Web provides three sets of fixed relevance judgments: agent citations, production rankings, and a combined set that unions both signals and adds LLM-based judgments of unlabeled pooled documents to reduce false negatives. We benchmark 13 retrievers including lexical, dense, and late-interaction models and find that their relative ordering is largely insensitive to the choice of judgment set, while diverging substantially across topical domains, query languages, and query types. To enable fast evaluation, we also study subcorpus sampling as an approximation to full-corpus evaluations. Retaining a third of the corpus, selected by reciprocal rank fusion over pooled retriever runs, preserves the full-corpus model ranking under the combined judgments while raising absolute Recall@1000 only by 3 to 7 points. The public leaderboard is accessible under: https://huggingface.co/spaces/perplexity-ai/q2d-web-leaderboard
1. Introduction
Q2D-Web addresses gaps in scale, query realism, and relevance supervision for evaluating first-stage retrievers in production agentic RAG. It combines a large web corpus with agent-reformulated queries, multiple judgment sources, and subcorpus sampling for efficient evaluation.
- First-stage retrieval bounds the evidence that an agent can read and ultimately cite in production RAG.
- Existing benchmarks lack the combination of large document collections, many diverse queries, and deep relevance judgments needed for large-scale evaluation.MS MARCO Web Search has 100M documents but only 9,374 test queries, while sparse labels can leave relevant documents unjudged.
- Agentic production queries are machine-generated reformulations that use conversation context, and reformulation can reduce retrieval nDCG@10 by about 20% on average.
- Q2D-Web combines a ∼190M-document web corpus with ∼70k agent-reformulated queries in ten languages from nine months of PII-free production traffic.
- Three judgment sets derive relevance from agent citations, production rankings, and LLM judgments of previously unjudged candidates to reduce false negatives without relying on one label source.
- A ∼30% RRF-selected subcorpus preserves full-corpus system ordering under Combined judgments while enabling a two-stage evaluation protocol.Promising retrievers can be screened on the sampled subcorpus before full-corpus assessment.
- Across 13 retrievers, BM25 has the lowest Recall@1000 overall but recovers the largest set of relevant documents that no other tested retriever recovers.
2. Related Work
Prior retrieval benchmarks trade off corpus scale, query volume, query realism, or label depth. Q2D-Web targets these gaps with production agent-reformulated queries, many relevance judgments, and evaluation at web scale.
- Existing web collections can reach billions of documents, but traditional TREC Web evaluation provides only 50 pooled, human-judged queries per year.
- MS MARCO Web Search provides 100M documents and 9,374 labeled test queries, with only one clicked document judged per query.
- Q2D-Web provides ∼190M documents, ∼70k judged queries, and 99.6 positive documents per query, exceeding MS MARCO Web Search on query and label scale.
- Unlike human-authored agent-evaluation benchmarks, Q2D-Web isolates first-stage retrieval on the agent-reformulated queries used by production pipelines.
- Q2D-Web provides three reusable judgment sets over the same corpus and queries, combining agent citations, production rankings, and LLM judgments of previously unjudged candidates.
- Earlier corpus-subsampling methods operate at TREC scale, whereas Q2D-Web subsamples a ∼190M-document production index with tens of thousands of citation-labeled agentic queries.
3. Q2D-Web
Q2D-Web is built through a four-stage pipeline that samples agent-reformulated production queries, constructs a large web corpus, creates multiple relevance-judgment sets, and studies efficient subcorpus evaluation.
- Query sampling: Q2D-Web samples approximately 70k LLM-reformulated queries from approximately 23k production searches over nine months.Queries are generated from user messages, preceding conversation, and results retrieved within the same search.
- Query sampling: Queries are stratified by month and matched to production topic and language distributions before filtering duplicates, short queries, operators, and PII.The dataset retains the ten most common languages.
- Corpus assembly: The corpus pools the top 5,000 documents for each query, deduplicates near-duplicates with MinHash–LSH, and yields approximately 190M canonical documents.Documents are clustered at a token 5-gram Jaccard similarity threshold of at least 0.975, retaining the longest document per cluster.
- Query sampling: Each search can contain a primary query and support queries, including multilingual reformulations and multi-hop searches that broaden coverage or resolve specific models.The examples include context resolution, bilingual support queries, and overview-to-specific query sequences.
- Corpus assembly: Because pooled results come from production retrieval, the corpus concentrates plausible distractors, including hard negatives shared across related reformulations.This construction targets challenging alternatives rather than randomly sampled unrelated web documents.
- Relevance judgments: Q2D-Web derives binary query–document judgments from citation, web-ranking, and additional LLM-judged sources, whose union reduces false negatives and source-specific bias.Citation is high-precision but low-recall, while web ranking labels documents that citation behavior may omit; the combined pass adds judgments for pooled documents.
- Subcorpus sampling: Full-corpus evaluation is costly, so the benchmark studies subcorpus sampling that retains positively judged documents and selects challenging distractors from pooled retriever rankings.A full-corpus pass with pplx-embed-v1-4b takes 4,608 H200 GPU-hours, motivating an evaluation-preserving approximation.
4. Benchmarking First-Stage Retrievers
Q2D-Web benchmarks 13 lexical, dense, and late-interaction retrievers on a 190M-document corpus using three metrics and multiple relevance sets. RRF subsampling preserves system ordering at roughly one-third of the corpus, while results vary by model family, domain, language, and query type.
- Experimental Setup: 13 retrievers spanning lexical, dense, and late-interaction models are evaluated with Recall@1000, Recall@100, and nDCG@10 on full and sampled corpora.The three metrics are reported across three relevance-judgment sets.
- Comparison of Sampling Strategies: RRF preserves full-corpus model ranking from 31.7% of the corpus, while depth-k pooling requires 43.4%.Both RRF and uniform random achieve Kendall’s τ_b = 1.00 at 31.7%; depth-k pooling reaches the same agreement at 43.4%.
- Comparison of Sampling Strategies: RRF inflates mean Recall@1000 by 5.1 points versus 11.1 points for uniform random, while depth-k pooling reaches 3.9 points only on a larger subcorpus.RRF with k = 1000 is adopted as the best trade-off between ranking fidelity, absolute error, and evaluation cost.
- Results: Subsampling raises every Recall@1000 score by 3–7 points but leaves the winner unchanged for each relevance-judgment set.Nemotron-3-Embed-8B wins Citation, while pplx-embed-v1-4b wins Web Ranking and Combined.
- Results: pplx-embed-v1-4b wins Recall@1000, whereas Nemotron-3-Embed-8B leads Recall@100 and nDCG@10 under Combined judgments.The metric-dependent leadership reflects a difference between recovering more relevant documents overall and placing more near the top.
- Results: Performance varies across domains, languages, and query types: neural retrievers are weaker on supporting queries, while BM25 performs slightly better on them.BM25 remains much weaker overall, and multilingual performance differs by language and model.
5. Insights and Analysis
Q2D-Web shows that aggregate recall does not capture the distinct relevant documents recovered by different retrievers. Performance also varies across query slices, and hard cases expose both labeling errors and systematic retrieval gaps.
- Unique positive coverage: 44.77 Recall@1000 is BM25-tantivy’s lowest score, yet it contributes 14,621 unique positives, nearly six times the next-highest contributor.EmbeddingGemma-300M contributes 2,451 unique positives, while the strongest retrievers contribute 884 and 705.
- Query slices: Recall@1000 varies across query domains and languages, with rows representing retrievers and columns representing query slices.The figure uses a shared color scale in which darker cells indicate higher recall.
- Unique positive coverage: Different retriever families recover different parts of the relevant set, so aggregate recall does not determine unique positive coverage.The weakest aggregate-recall retriever recovers by far the largest set that no other retriever recovers.
- Query types: Neural retrievers perform worse on supporting agent-reformulated queries, whereas BM25 improves slightly.Figure 8 connects each retriever’s primary-query and supporting-query scores.
- Hard negatives: 15.8% of sampled hard negatives are actually relevant, while 84.2% are genuine hard negatives.Wrong-aspect cases dominate the genuine hard negatives at 39.0%, followed by partial at 17.8% and related-entity at 13.8%.
- Hard positives: 51.1% of 6,424 hard positives are lexical, with multilingual and truncation cases accounting for 15.4% and 17.7%.These relevant documents are missed by all thirteen retrievers at Recall@1000.
6. Conclusion
Q2D-Web is a production-derived benchmark combining web-scale documents, agent-reformulated queries, and multiple relevance-label sets. Its evaluations show stable system rankings across labels, preserve rankings with a roughly 30% RRF subcorpus, and support measuring retriever diversity.
- Benchmark: Q2D-Web contains ∼190M documents, ∼70k agent-reformulated queries, and three relevance-label sets from nine months of production traffic.The benchmark evaluates thirteen lexical, dense, and late-interaction retrievers.
- Findings: Retriever rankings are largely stable across label sets, while the lowest-aggregate-recall retriever contributes the most unique labeled positives.This shows that aggregate ranking and unique positive contribution capture different properties.
- Efficient evaluation: A ∼30% RRF subcorpus preserves ranking order under Combined judgments while slightly inflating absolute recall.For a 4B embedding model, evaluation cost falls from 4,608 to roughly 1,500 H200 GPU-hours.
- Ensembles: Many labeled positives per query enable measuring which relevant documents each retriever adds to an ensemble.Future work can test whether diverse ensembles recover more relevant documents than equally sized ensembles of similar retrievers.
Ethics and Privacy Statement
Q2D-Web uses privacy safeguards for production search traffic but remains a private benchmark, with only aggregate results shared externally.
- Privacy: Q2D-Web applies PII detection during query sampling and excludes queries flagged as containing PII.The authors state that the traffic use is consistent with applicable agreements, privacy policy, and data protection law.
- Release policy: The corpus, queries, and relevance judgments remain private, while only aggregate results such as leaderboard scores are shared externally.The three relevance-judgment sources are maintained separately rather than treating one pass as ground truth.
A. Retrievers for Relevance Pooling and Subcorpus Sampling
Table 4 lists the retrievers used to pool previously unjudged documents for the Combined+LLM-Judged pass and specifies their model and indexing characteristics.
- Pooling retrievers: Table 4 organizes pooling retrievers by embedding dimension and maximum indexed input length, with late-interaction dimensions reported per token.It also identifies models supporting smaller dimensions and records the January 1, 2025 release cutoff.