Characterizing Deep Research: A Benchmark and Formal Definition

1 INTRODUCTION

• OpenAI’s DR system achieved the highest overall average F1, 0.55, while category-level F1 ranged from 0.02 to 0.72.The comparison evaluates systems from OpenAI, Perplexity AI, and Google.Source: Figure 2 · p. 2, Figure 3 · p. 2, §1 · p. 3
• LiveDRBench evaluates claim synthesis—the discovery of query-relevant information—rather than treating long-form report writing as deep research’s defining task.The paper separates claim synthesis from the subsequent task of writing a report based on those claims.Source: §1 · p. 2
• A deep research query requires high search intensity and non-trivial reasoning in finding, processing, or combining information units.The paper proposes that, assuming the entire web as the corpus, such a query takes an ideal human expert more than 10 minutes.Source: §1 · pp. 2–3
• Deep research combines high search and reasoning intensity, unlike tasks that are intensive mainly in reasoning or have limited search.The paper contrasts deep research with multi-hop QA, document-based scientific or legal QA, and explain-like-im-5 tasks.Source: §1 · p. 3
• The evaluation represents a solution as claims and recursive subclaims, scoring claims as incorrect when the claim or all its supporting subclaims are incorrect.This intermediate representation is intended to make substantive claim evaluation more objective than judging detailed reports for both style and substance.Source: §1 · p. 3
• LiveDRBench contains 100 queries across scientific and world-events domains, including dataset and material discovery, novelty identification, flight crashes, books, and movies.The benchmark spans varied query types, from needle-in-a-haystack searches to listing entities.Source: §1 · p. 3, Table 1 · p. 4

2 BACKGROUND & RELATED WORK

• Commercial deep research systems emerged in late 2024 and early 2025, while academic precursors explored literature search, planning, conversational prompts, agentic frameworks, and reinforcement-learning fine-tuning.The paper notes that details on commercial system architecture and evaluation were unavailable beyond qualitative examples.Source: §2 · pp. 4–5
• Despite extensive surveys, the paper identifies a missing formal definition of deep research.It positions its definition as addressing this gap.Source: §2 · p. 5
• Prior evaluations use LLM judges, question-specific human criteria, or nonpublic data; LiveDRBench instead offers a public benchmark with a mechanism to refresh questions.The paper also notes that a contemporaneous public benchmark follows problem inversion but focuses only on needle-in-a-haystack searches.Source: §2 · p. 5
• Unlike benchmarks built around static retrieval corpora, LiveDRBench allows retrieval from the entire web and uses specialized query-construction techniques to retain challenge.The paper frames this choice as an effort to approximate real-world deep research use.Source: §2 · p. 5

3.1 DEEP RESEARCH: TASKS INVOLVING BOTH SEARCH AND REASONING INTENSITY

• The required search and reasoning depend on the corpus, and a single large document may contain multiple separately retrieved information units.An information unit is an atomic piece such as a paragraph or retrieval chunk.Source: §3 · p. 6
• Reasoning intensity can lie in finding information, processing retrieved results, or combining results into claims.The paper gives examples ranging from iterative query selection to comparing information across searches.Source: §3.1 · p. 6
• Deep research requires processing many information units and non-trivial reasoning in finding, processing, or combining them.The authors suggest more than ten minutes of expert work as a reasoning-intensity threshold and roughly 20 information units across at least 10 searches as a typical search-intensity threshold.Source: §3.1 · p. 7
• The paper represents a DR answer as independent claims that may contain recursively nested subclaims, separating claim synthesis from report writing.Writing a report from claims is treated as an independently studyable long-form generation task.Source: §3 · pp. 5–6, §3.1 · p. 7

3.2 EVALUATION: PRECISION AND RECALL OVER GROUND-TRUTH CLAIMS’ STRUCTURE

• Claim-level precision and recall account for agreement on each claim and the precision or recall of its subclaims.An incorrect parent claim receives zero contribution regardless of its subclaims; correct answers with unsupported subclaims are also penalized.Source: §3.2 · p. 7
• Standard scores average contributions across independent claims, while strict scores take their minimum and reach one only for an exact claim-list match.Optional claim weights can prioritize important claims; strict scoring is motivated by user trust when a derivatory subclaim is incorrect.Source: §3.2 · pp. 7–8

4.1 CRITERIA FOR TASKS

• Benchmark tasks should resist single-article answers, require broad Web sourcing, support objective reproducible evaluation and updates, and cover varied domains and users.The criteria also aim to ground responses rather than permit answers directly from parametric memory.Source: §4.1 · p. 8
• The construction strategy inverts long-context reasoning questions, turning given keys and values into a search for the matching entity and corroborating sources.The original problem asks for values consistent with a document; the inverted task asks for the entity consistent with supplied properties.Source: §4.1 · p. 9
• Questions are manually checked for unique answers, with additional correct answers added to the ground truth when needed.The benchmark covers science, innovation and global culture or events, and uses claims as intermediate outputs for objective precision and recall evaluation.Source: §4.1 · p. 9

4.2.1 SCIFACTS: SOURCING SCIENTIFIC FACTS

• SCIFACTS contains material identification and geospatial paper identification tasks that invert long-context questions into searches for materials or papers from specified properties or datasets.Material tasks require a material and supporting paper; geo tasks seek papers using all datasets in the query.Source: §4.2.1 · pp. 9–10

4.2.2 NOVELDS: IDENTIFYING DATASET PAPERS

• NOVELDS includes dataset identification with metadata, dataset identification plus paper findings, and peer-dataset retrieval from high-level descriptions.Extraction may require reasoning across tables or interpreting figures; peer retrieval asks for distinguishing features without naming a dataset.Source: §4.2.2 · p. 10

4.2.3 PRIORART: DECONSTRUCTING PAPER ABSTRACTS

• PRIORART uses 17 synthetic abstracts to test whether models can identify source papers for abstract claims and explain how those papers support them.Tasks draw on reviewer discussions and related papers, including sources not already cited by the paper at hand, so following its references alone is insufficient.Source: §4.2.3 · pp. 10–11

4.2.4 FLIGHTS: IDENTIFYING FLIGHT INCIDENTS

• FLIGHTS contains seven questions based on official investigation reports, asking systems to identify incidents from distinctive descriptions and provide causes or timelines.The reports may span hundreds of pages, requiring synthesis across multiple sections.Source: §4.2.4 · p. 11

4.2.5 ENTITIES: ENUMERATING ENTITIES WITH CONSTRAINTS

• ENTITIES asks for exhaustive lists meeting constraints, typically by examining 80–140 items to return 8–30 qualifying answers.Questions cover recurring world events; ground truth is scraped, manually checked, and finalized with valid missing answers before precision, recall, and F1 evaluation.Source: §4.2.5 · pp. 11–12

4.3 LIVEDRBENCH: ENABLING BENCHMARK UPDATION WITH NOVEL QUESTIONS

• SCIFACTS queries can be generated from newly published scientific papers by extracting key information and using it to retrieve the source paper.The authors describe having used this approach with CURIE-derived queries.Source: §4.3 · p. 12
• NOVELDS peer-retrieval questions can be extended by extracting datasets and distinguishing features from literature-survey tables in new papers.Source: §4.3 · p. 12
• ENTITIES can add queries by selecting a similar event category, defining criteria, and iteratively scraping qualifying items.Existing categories include movies, book publications, international math Olympiads, and the Olympics.Source: §4.3 · p. 12

4.4 EVALUATION

• Each task expects a list of dictionaries, with top-level entries representing claims and keys representing supporting sub-claims.For SCIFACTS material identification, the material name is the claim and the source paper is its supporting sub-claim.Source: §4.4 · p. 12
• Prompts request JSON output, but reports from systems that do not follow the instruction are manually parsed; the JSON requirement can be omitted.The authors chose manual parsing rather than LLM parsing to avoid possible errors.Source: §4.4 · p. 12
• Precision and recall are computed from GPT-4o agreement scores between generated claims and ground-truth counterparts, then combined as F1.Agreement scores measure claim accuracy on a scale from 0 to 1.Source: §4.4 · p. 12
• Because multiple correct answers may be absent from the rubric, the authors manually add correct claims from evaluated systems before automated scoring against the expanded ground truth.The final benchmark data includes the expanded ground truth for each question.Source: §4.4 · p. 12

5.1 SETUP

• The evaluation covered three proprietary DR systems, three reasoning-enabled baselines, and three non-reasoning baselines, all with web search enabled.DR systems were tested manually in chat interfaces; baseline API settings were mostly defaults, with reasoning budgets corresponding to about 30 seconds.Source: §5.1 · p. 13

5.2 OVERALL RESULTS

• OpenAI DR achieved the highest overall average F1 at 0.55; category F1 ranged from 0.02 to 0.72.Table 2 reports precision, recall, and F1 across all task categories.Source: Table 2 · p. 13, §5.2 · pp. 13–14
• The weakest categories included SCIFACTS Materials, NOVELDS Identification and Extraction, and FLIGHTS, which require extracting findings across multiple document sections.On SCIFACTS Materials, systems often retrieved either the paper title or material name, but not both.Source: §5.2 · p. 14
• Reasoning-enabled baselines scored zero on SCIFACTS Materials, while o4-mini matched or surpassed Perplexity DR and Gemini DR on many categories but trailed OpenAI DR.Non-reasoning baselines performed worse, with limited planning, iterative search, and backtracking.Source: §5.2 · p. 14

5.3 ANALYSIS OF DEEP RESEARCH MODELS

• On the NOVELDS subset, OpenAI DR achieved the highest average F1 (0.55), outperforming Perplexity DR (0.41) and Gemini DR (0.35); on average, all DR models branched more than they backtracked.Figure 5 plots sources, branches, and backtracking against F1 and shows F1 efficiency per event.Source: §5.3 · p. 15, Figure 5 · p. 16
• Gemini DR had the highest F1 per source but the lowest overall F1, while OpenAI DR's frequent branching and backtracking raised costs and reduced overall efficiency.Perplexity DR gathered many sources but backtracked relatively little, indicating an exploratory yet shallow approach.Source: §5.3 · p. 16

6 DISCUSSION AND CONCLUSION

• Deep research is defined as discovering claims and their supporting evidence that ultimately form the final long-form answer.The characterization separates search and reasoning from long-form generation.Source: §6 · p. 16
• The benchmark contains 100 problems and was used to evaluate existing deep research systems.The evaluation is intended to be amenable to objective assessment.Source: §6 · p. 16
• ENTITIES results suggest training or interleaved programmatic control may help when the algorithm is straightforward but search is laborious.For SCIFACTS, systems may get the answer right but ground it incorrectly, or vice versa; the authors suggest tools that update reports using reasoning traces.Source: §6 · p. 17
• The analysis examines branching and backtracking, whose tradeoffs the authors identify as important for inference-time efficiency.The authors point to understanding these operations as a direction for improving deep research systems.Source: §6 · p. 17

A EXAMPLE TASKS FROM LIVEDRBENCH

• The examples include scientific tasks such as identifying materials from measured properties, locating papers using specified datasets, and finding prior art.The materials prompt requests an inference basis and source locations for the reported properties.Source: A EXAMPLE TASKS FROM LIVEDRBENCH · p. 20
• Other examples ask about diffusion-based lighting methods, GPT-4 value-dimension bias, and acquisition methods for 3D urban segmentation datasets.The prompts request paper details, framework-specific bias dimensions, or dataset attributes such as area and scene count.Source: A EXAMPLE TASKS FROM LIVEDRBENCH · pp. 21–22
• Public-interest examples ask for Oscar-winning adaptations by women authors and a timeline of anomalies in a high-profile flight investigation.Both prompts specify structured output fields; the flight prompt also asks for expected behavior and reasons for doubt.Source: A EXAMPLE TASKS FROM LIVEDRBENCH · p. 22
• The examples also include a task about generating readable optimized attack suffixes and assessing their behavior and internal embeddings.The prompt describes suffix generation as left-to-right optimization balancing a search objective with readability.Source: A EXAMPLE TASKS FROM LIVEDRBENCH · p. 23

B EVALUATION PROMPTS

• GPT-4o serves as the judge, while human checks and valid answers missing from the initial set are used to enrich the ground truth.The benchmark is built on this enriched ground-truth set.Source: B EVALUATION PROMPTS · p. 23
• For SCIFACTS Materials and Geo, the prompts evaluate paper titles and material names using key-level matching and semantic equivalence rules.The Materials task combines key-level evaluations into a reported metric, while Geo uses the same prompt for paper titles.Source: B EVALUATION PROMPTS · pp. 23–24
• A separate SCIFACTS judging prompt scores each value from 0 to 3 based on how closely it preserves the ground-truth information.The rubric ranges from close agreement with minimal meaning loss to an incorrect or unrelated answer.Source: B EVALUATION PROMPTS · pp. 23, 25
• For NOVELDS and FLIGHTS, predicted dictionaries are matched to ground-truth dictionaries by primary keys, then judged as whole dictionaries.Whole-dictionary evaluation preserves context among related sub-claims; numerical values are considered equivalent within a 1% margin.Source: B EVALUATION PROMPTS · pp. 24–25
• PRIORART and ENTITIES use string matching against ground-truth lists, allowing minor formatting differences.A match returns the corresponding ground-truth string; otherwise, the prompt returns an empty dictionary.Source: B EVALUATION PROMPTS · pp. 25–26

C TRACE ANALYSIS PROMPTS

• A branch is a distinct hypothesis, search direction, or subgoal, counted when the researcher changes focus or explores parallel possibilities.The prompt asks for the number of branches as an integer.Source: C TRACE ANALYSIS PROMPTS · p. 26
• Backtracking means revisiting or reconsidering earlier ideas, changing direction after an approach proves suboptimal, or discarding a line of inquiry.The prompt counts changed ideas or directions even when no explicit transition phrase appears.Source: C TRACE ANALYSIS PROMPTS · p. 26

• Table 11 reports precision, recall, and F1 for Deep Research models on LiveDRBench, with claim-agreement scores evaluated by the authors.Source: Table 11 · p. 27