Source-linked AI summary
Total Recall QA: A Verifiable Evaluation Suite for Deep Research Agents
Mahta Rafiee, Heydar Soudani, Zahra Abbasiantaeb, Mohammad Aliannejadi, Faegheh Hasibi, Hamed Zamani
TL;DR
Deep research agents require multi-step information seeking and reasoning, yet their evaluation remains challenging and existing benchmarks do not meet all identified criteria. This paper introduces Total Recall QA and a framework for building verifiable datasets with relevance judgments from paired structured knowledge bases and text corpora. Competitive deep research methods and LLMs struggle even with oracle retrieval, leaving substantial room to improve retrieval and reasoning models.
Problem
Existing deep research benchmarks do not satisfy all identified evaluation requirements, limiting reliable and reproducible assessment of systems that synthesize information across sources.
Source: §1 · pp. 1–2Method
The paper introduces Total Recall QA and an entity-centric framework that builds verifiable queries and relevance judgments from aligned structured knowledge bases and text corpora.
Source: §1 · p. 2, §2.2 · p. 4, §2.3 · p. 4Results
Takeaways & Limitations
TRQA provides datasets, relevance annotations, and benchmark results for evaluating both intermediate retrieval and end-to-end deep research performance.
Source: §6 · p. 7
Abstract
from arXiv · showhide
Deep research agents have emerged as LLM-based systems designed to perform multi-step information seeking and reasoning over large, open-domain sources to answer complex questions by synthesizing information from multiple information sources. Given the complexity of the task and despite various recent efforts, evaluation of deep research agents remains fundamentally challenging. This paper identifies a list of requirements and optional properties for evaluating deep research agents. We observe that existing benchmarks do not satisfy all identified requirements. Inspired by prior research on TREC Total Recall Tracks, we introduce the task of Total Recall Question Answering and develop a framework for deep research agents evaluation that satisfies the identified criteria. Our framework constructs single-answer, total recall queries with precise evaluation and relevance judgments derived from a structured knowledge base paired with a text corpus, enabling large-scale data construction. Using this framework, we build TRQA, a deep research benchmark constructed from Wikidata-Wikipedia as a real-world source and a synthetically generated e-commerce knowledge base and corpus to mitigate the effects of data contamination. We benchmark the collection with representative retriever and deep research models and establish baseline retrieval and end-to-end results for future comparative evaluation.
1 Introduction
The paper introduces Total Recall QA to address gaps in deep research agent evaluation, including limited verifiability, reproducibility, and retrieval assessment. Its framework constructs datasets with verifiable answers and relevance judgments, and benchmark results show that strong systems still struggle.
1 Introduction
- Existing benchmarks often use rubric- or LLM-scored long-form answers, limited citation checks, or live search, and may overlook contamination risks.The paper argues these practices can conflict with verifiability and reproducibility requirements.Source: §1 · p. 2
- The proposed task and evaluation framework target reproducible component-level assessment, precise scoring, explicit relevance judgments, and fixed-corpus retrieval evaluation.The framework is inspired by the TREC Total Recall Tracks.Source: §1 · p. 2
- Total Recall QA requires retrieving all relevant documents for a question and reasoning across them to synthesize an accurate answer.The task is designed around multi-source information seeking and synthesis.Source: §1 · pp. 1–2
- The suite includes single-answer numerical queries, two Wikipedia-based datasets, and a synthetic e-commerce dataset intended to mitigate contamination effects.The queries retain total-recall retrieval and multi-step reasoning challenges while supporting verifiable evaluation.Source: §1 · p. 2
- Competitive deep research methods and LLMs struggle to answer Total Recall QA questions even with oracle retrieval.The paper reports that most errors with oracle retrieval are reasoning errors rather than parametric-knowledge bias.Source: §1 · p. 2
2 TRQA Framework and Data Construction
TRQA uses an entity-centric framework pairing a structured knowledge base with aligned documents to generate total-recall queries, answers, and relevance judgments across three datasets.
2 TRQA Framework and Data Construction
- The framework pairs each knowledge-base entity with a corpus document, enabling automatic query construction grounded in structured data and application across three settings.The datasets are TRQA-Wiki1, TRQA-Wiki2, and TRQA-Ecommerce.Source: §2 · p. 2
2.1 The Data Generation Framework
- Queries filter entity sets with property constraints, then aggregate a target property across the remaining entities to produce a single answer.Correctly answering requires retrieving the full relevant entity set and its property values.Source: §2.1 · p. 3
- The framework segments documents into passages and uses paired Wikidata–Wikipedia sources and synthetic e-commerce data to construct its datasets.The Wikipedia preprocessing preserves infobox and document structure while chunking to passages of 20–100 words.Source: §2.1 · p. 3, §2.2 · p. 4
- Passage relevance labels are generated by checking whether the knowledge-base target value appears in each passage, then rewriting conflicting or missing values to match.Entity-to-document links provide document-level relevance; an LLM labels passages and a second LLM makes value-only corrections when needed.Source: §2.1 · p. 3
2.2 Real Data: Wikidata-Wikipedia
- TRQA-Wiki1 selects Wikidata entity classes whose declared quantity matches the retrievable instance count, yielding sets with explicit completeness guarantees.For example, the U.S. states class has a specified quantity of 50.Source: §2.2 · p. 4
- TRQA-Wiki2 derives complete entity lists from QALD-10 SPARQL queries and QUEST annotations, although those source benchmarks do not themselves measure total-recall QA.QALD-10 is often answerable from one document, while QUEST lacks a single final answer.Source: §2.2 · p. 4
2.3 Synthetic Data: E-Commerce
- TRQA-Ecommerce uses synthetic products and generated blog posts to evaluate settings where models cannot rely on parametric memory.Its construction starts from MAVE, expands category metadata, generates product records, and creates product-specific blogs.Source: §2.3 · pp. 4–5
3 TRQA Statistics
The datasets vary in scale and query characteristics, with Wiki2 containing a larger share of long-tail topics than Wiki1.
3 TRQA Statistics
- TRQA-Wiki1 has fewer queries because its strict entity-validity and completeness requirements filter eligible query sets.The shared Wikipedia corpus has approximately 57 million documents, and a subsampled variant is also provided to reduce computational overhead.Source: §3 · p. 5
- TRQA-Wiki2 contains a larger proportion of long-tail entities than TRQA-Wiki1, partly reflecting their different entity-set selection strategies.Wiki1 draws on complete real-world sets such as U.S. states, while Wiki2 uses QALD-10 and QUEST entity lists from less popular domains including books, plants, and animals.Source: §3 · p. 5
- Aggregation operations vary by dataset: Wiki1 includes distance-based operations, while Ecommerce includes percentage operations.These operations reflect geospatial entities in Wiki1 and fractions of products satisfying attribute constraints in Ecommerce.Source: §3 · p. 5
4 Experimental Setup
The experiments assess retrieval and numerical answer quality across retrievers, deep research agents, closed-book language models, and single-retrieval RAG baselines.
4 Experimental Setup
- Six retrieval models are evaluated, spanning sparse lexical, learned sparse, reranked, and dense retrieval approaches.The set includes BM25, SPLADE++, a BM25-plus-MiniLM reranker, Contriever, E5, and BGE.Source: §4 · p. 5
- End-to-end comparisons include three deep research agents, closed-book LLMs, and single-retrieval RAG using the top 50 E5 results.The agent experiments use Qwen2.5-7B and E5; the RAG baseline is also tested with GPT-5.2.Source: §4 · p. 5
- Retrieval is measured by recall at multiple depths, while numerical answers use normalized Exact Match and Soft Match at four tolerance levels.Soft Match is reported at 10%, 20%, 50%, and 90% tolerance.Source: §4 · p. 5
5 Results
Across TRQA, results show dataset-dependent end-to-end performance, gaps in retrieval coverage, and persistent reasoning errors even with oracle retrieval.
Single-Turn Retrieval Evaluation
- BM25 is relatively strong at shallow ranks, neural retrieval does better at deeper ranks, and SPLADE++ leads at every depth on the synthetic dataset.A single retrieval round often has poor recall even with a deep rank list.Source: §5 · p. 5, Table 4 · p. 6
End-to-End Evaluation
- On TRQA-Wiki1 and Wiki2, closed-book LLMs—especially GPT-5.2—dominate, but on Ecommerce LLM-only performance drops sharply, often below strong DRAs.GPT-5.2 significantly outperforms other LLMs across all metrics.Source: Table 5 · p. 6, §5 · p. 6
- DRAs generally outperform single-step RAG with the same LLM; on Ecommerce, E5 & GPT-5.2 perform relatively well, where queries average twice as many relevant documents as elsewhere.The paper suggests this may require larger models for accurate reasoning.Source: §5 · p. 6
Sub-query Analysis
- Sub-query counts stay flat as gold entities increase, while added calls retrieve mainly distractors; Search-R1 adds no new entities.The figure analyzes average sub-queries against query gold entities and newly retrieved gold and distracting entities as calls increase.Source: §5 · p. 6, Figure 4 · p. 7
Intermediate Multi-Turn Retrieval Evaluation
- Search-R1 has higher accumulated intermediate recall than ReAct and Search-o1, but recall for all three remains far below 100%.Table 6 reports recall across all intermediate retrieval steps on the full corpus.Source: §5 · p. 6, Table 6 · p. 7
Oracle Retrieval
- Oracle retrieval substantially improves LLM performance over no retrieval, yet models still struggle; over 90% of oracle-retrieval errors are reasoning errors, and Ecommerce is weakest.The error analysis combines all TRQA datasets.Source: §5 · p. 7, Table 7 · p. 7
Partial Wikipedia Corpus
- The 5.9-million-document partial Wikipedia corpus yields nearly identical single-step retriever results to the full corpus under a fixed seed.The partial corpus is formed by pooling relevant passages and retrieval runs.Source: §5 · p. 7, Table 8 · p. 7
6 Conclusions
The paper introduces Total Recall QA and builds TRQA datasets with verifiable answers and retrieval judgments. Its benchmarks show substantial room to improve retrieval and reasoning models.
6 Conclusions
- Total Recall QA is introduced as a task for evaluating deep research agents.The paper develops the TRQA framework for automatic construction from aligned structured and unstructured data.Source: §6 · p. 7
- Three datasets contain questions with verifiable answers, and query-document relevance annotations enable intermediate retrieval evaluation.The datasets are constructed using the framework's paired structured and unstructured sources.Source: §6 · p. 7
- Benchmark results on the created datasets highlight the challenge of Total Recall QA and significant room for improvement in retrieval and reasoning models.Source: §6 · p. 7