PaperlayerView exact revision on arXiv

Source-linked AI summary

LiveResearchBench: A Live Benchmark for User-Centric Deep Research in the Wild

Jiayu Wang, Yifei Ming, Riya Dulepet, Qinglin Chen, Austin Xu, Zixuan Ke, Frederic Sala, Aws Albarghouthi, Caiming Xiong, Shafiq Joty

arXiv:2510.14240v5cs.AI

TL;DR

Deep research is difficult to benchmark because open-ended, time-sensitive reports require assessment across multiple quality dimensions. LiveResearchBench and DeepEval provide curated tasks and a multifaceted evaluation suite; across 17 systems, the authors find recurring vulnerabilities and that most models cannot yet write insightful reports.

  • Problem

    Deep research lacks representative benchmarks and reliable ways to assess open-ended reports across coverage, reasoning, evidence use, and presentation.

    Source: §1 · pp. 1–2

  • Method

    The authors construct LiveResearchBench, a benchmark of 100 expert-curated tasks, and DeepEval, a suite of tailored protocols for assessing content- and report-level quality.

    Source: §1 · p. 2

  • Results

    Evaluation of 17 systems reveals systematic vulnerabilities and error patterns, with most models still unable to write insightful reports and struggling with citation reliability and analytical depth.

    Source: §1 · p. 3, §6 · p. 10

  • Takeaways & Limitations

    The benchmark and evaluation suite establish a foundation for systematic evaluation and point to memory, compression, and synthesis as priorities for more insightful deep research.

    Source: §6 · p. 10

Abstract

from arXiv · show

Deep research -- producing comprehensive, citation-grounded reports by searching and synthesizing information from hundreds of live web sources -- marks an important frontier for agentic systems. To rigorously evaluate this ability, four principles are essential: tasks should be (1) user-centric, reflecting realistic information needs, (2) dynamic, requiring up-to-date information beyond parametric knowledge, (3) unambiguous, ensuring consistent interpretation across users, and (4) multi-faceted and search-intensive, requiring search over numerous web sources and in-depth analysis. Existing benchmarks fall short of these principles, often focusing on narrow domains or posing ambiguous questions that hinder fair comparison. Guided by these principles, we introduce LiveResearchBench, a benchmark of 100 expert-curated tasks spanning daily life, enterprise, and academia, each requiring extensive, dynamic, real-time web search and synthesis. Built with over 1,500 hours of human labor, LiveResearchBench provides a rigorous basis for systematic evaluation. To evaluate citation-grounded long-form reports, we introduce DeepEval, a comprehensive suite covering both content- and report-level quality, including coverage, presentation, citation accuracy and association, consistency and depth of analysis. DeepEval integrates four complementary evaluation protocols, each designed to ensure stable assessment and high agreement with human judgments. Using LiveResearchBench and DeepEval, we conduct a comprehensive evaluation of 17 frontier deep research systems, including single-agent web search, single-agent deep research, and multi-agent systems. Our analysis reveals current strengths, recurring failure modes, and key system components needed to advance reliable, insightful deep research. Our code is available at: https://github.com/SalesforceAIResearch/LiveResearchBench.

1 INTRODUCTION

Deep research requires extensive, up-to-date searching and synthesis, but its open-ended reports and tasks are difficult to benchmark reliably. The paper introduces LiveResearchBench and DeepEval to evaluate dynamic research tasks and the resulting reports.

1 INTRODUCTION

  • The proposed task-design principles are user-centric, dynamic, unambiguous, and multi-faceted and search-intensive, supporting realistic needs, current information, consistent interpretation, and multi-hop research.The principles are grounded in user surveys and comparative analysis.Source: §1 · pp. 1–3
  • LiveResearchBench provides 100 expert-curated, checklist-validated tasks across seven domains and ten categories, designed to be dynamic, multi-faceted, realistic, and unambiguous.Tasks specify scope, audience, and format where appropriate; many require real-time search, and their construction involved over 1,500 hours of human labor.Source: §1 · pp. 2–3
  • DeepEval assesses six dimensions of open-ended reports: coverage, analysis depth, citation association, factual accuracy, presentation and organization, and logical and factual consistency.Tailored checklist-based, pointwise, pairwise, or rubric-tree protocols aim for stable assessments that agree with human evaluations.Source: §1 · pp. 2–3
  • Evaluating 17 single- and multi-agent systems revealed systematic vulnerabilities and error patterns, with most models still unable to write insightful reports.The evaluation covers both open-source and proprietary systems.Source: §1 · p. 3

2 RELATED WORK

The related work identifies gaps in comparing deep-research agent architectures, evaluating dynamic user-centric benchmarks, and assessing long-form reports. LiveResearchBench and DeepEval are presented as addressing these gaps.

Single- and Multi-Agent Deep Research Systems

  • Single-agent and multi-agent research systems have proliferated, but which settings favor either architecture and whether multi-agent systems yield consistent gains remain unclear.The lack of standardized benchmarks and evaluation frameworks also prevents rigorous comparison.Source: §2 · p. 3

Deep Research Benchmark

  • Existing benchmarks are often domain-specific, short-form or closed-ended, and static; LiveResearchBench instead offers unambiguous, time-varying, multi-faceted, search-intensive tasks across diverse domains.The paper also notes that recent benchmarks include short, static tasks or focus narrowly on AI.Source: §2 · p. 3

Evaluation of Long-form Answer

  • Prior long-form evaluations often isolate one dimension; DeepEval combines report-level and fine-grained measures, using structured checklists whose aggregate grades closely track expert judgments.Its measures include presentation, consistency, depth, checklist-based coverage, citation traceability, and rubric-tree accuracy.Source: §2 · p. 3

3 LIVERESEARCHBENCH

LiveResearchBench is a benchmark of user-centered, real-time, multi-faceted deep research tasks, built through expert curation and checked through a staged human verification process.

3.1 TASK DESIGN PRINCIPLES

  • Task design follows four principles: user-centricity, unambiguous requirements, time-varying information needs, and multi-faceted, search-intensive research.Real-time search helps resist pretraining data contamination, while multi-hop searches across diverse sources test more than simple fact retrieval.Source: §3.1 · p. 4

3.2 BENCHMARK OVERVIEW

  • LiveResearchBench comprises 100 expert-curated questions with detailed checklists, spanning seven domains and ten task categories.Tasks specify audience, scope, breadth, depth, and output format, supporting consistent interpretation across models.Source: §3.2 · pp. 4–5

3.3 BENCHMARK CONSTRUCTION

  • Construction combines user interviews and a crowd survey, expert-drafted queries, model-generated clarification questions refined by experts, and GPT-5 checklists that test required coverage.Using two research models for clarifications provides complementary perspectives; checklists decompose queries into verifiable unit questions.Source: §3.3 · p. 5, Figure 3 · p. 5

3.4 DATA VERIFICATION

  • Verification uses independent expert assessments, two quality-control stages, and final expert cross-checking, with sampling to keep overall quality within 5% of true data quality.Experts resolve conflicts and refine queries and checklists before finalizing the dataset.Source: §3.4 · p. 6, Figure 4 · p. 6

4 DEEPEVAL: A COMPREHENSIVE EVALUATION SUITE FOR DEEP RESEARCH

DeepEval evaluates research reports across six complementary dimensions, spanning presentation, consistency, coverage, analysis depth, and citation quality. Its protocols use checklists, issue-based scoring, pairwise comparisons, and web-based citation validation rather than relying on a single overall rating.

4 DEEPEVAL: A COMPREHENSIVE EVALUATION SUITE FOR DEEP RESEARCH

  • DeepEval assesses six dimensions: presentation and organization, factual and logical consistency, coverage and comprehensiveness, analysis depth, citation association, and citation accuracy.The suite targets both report-level and content-level quality.Source: §4 · p. 6
  • DeepEval uses checklists for presentation and coverage, additive pointwise scoring for consistency and citation association, and pairwise comparison for analysis depth.For LLM-as-a-Judge evaluations, Gemini 2.5 Pro and GPT-5 independently assess reports, and their scores are averaged; citation accuracy instead uses a rubric tree.Source: §4 · pp. 6–7
  • Single 0–10 LLM ratings achieved below 60% agreement with humans and showed large run-to-run variability, including analysis-depth differences exceeding 50 points.Judges often gave shallow, conflated overall assessments instead of examining errors sentence by sentence, leading to missed errors and inflated scores.Source: §4 · p. 6

4.1 COARSE-GRAINED: REPORT-LEVEL METRICS

  • Presentation & Organization uses a 10-item binary checklist covering common errors, with the judge ensemble achieving 98.3% agreement with human judgments.The checklist covers structure, grammar, citations, duplicates, and formatting; the reported score is the average success rate across items and judges.Source: §4.1 · p. 7
  • Factual & Logical Consistency uses pointwise evaluation to identify substantive inconsistencies, starting at 100 and deducting for each additional issue.The score is scaled from 0 to 100; a report with no contradictions receives 100.Source: §4.1 · p. 7

4.2 FINE-GRAINED: CONTENT-LEVEL METRICS

  • Coverage & Comprehensiveness scores whether human-curated checklist requirements are met, with the judge ensemble’s preferences achieving 100.0% human agreement.Each item is a binary test, and scores are averaged across items and queries.Source: §4.2 · p. 7
  • Analysis Depth compares reports across five dimensions, using position-swapped pairwise judgments and excluding ties from win-rate denominators; ensemble judges achieve 92.5% human agreement.The dimensions are reasoning granularity, multi-layer insights, critical evaluation, analytical use of evidence, and insight density.Source: §4.2 · p. 7
  • Citation Traceability deducts for missing or misplaced citations, while Citation Accuracy uses web-accessible URLs to check whether sources support their linked claims.Citation Traceability’s full-coverage score is 100, and its judge ensemble achieves 85.9% human agreement; the accuracy procedure checks URL accessibility before assessing support.Source: §4.2 · p. 7

5 MAIN RESULTS AND ANALYSIS

Multi-agent systems lead on average and on some report-quality dimensions, but no system excels across all four. Longer or more polished reports do not necessarily provide deeper, more reliable analysis; context limits constrain coverage and synthesis.

Report length

  • Single-agent deep research systems produce the longest reports, but report length varies more by model than task and can reflect citation formatting rather than substance.Long redirect URLs and duplicate inline and bibliography links inflate token counts; longer reports do not necessarily improve quality.Source: Figure 5 · p. 8, §5 · p. 8

Overall performance

  • Multi-agent deep research averages 69.5, ahead of single-agent web (62.8) and single-agent deep research; Open Deep Research leads systems at 73.6, followed by GPT-5 at 72.7.Deerflow+ is also identified among the leading systems, though no average score is given for it here.Source: Table 1 · p. 9, §5 · p. 9

Citation reliability

  • Citation problems include mismatched references, incomplete URLs, inconsistent formats, uncited bibliography entries, broken tables, and out-of-order numbering; unsupported claims also persist despite web access.In wide information search, most citation errors are unsupported claims rather than invalid or irrelevant URLs.Source: §5 · pp. 9–10

Dimension-specific performance

  • Single-agent web averages 69.7 on Factual & Logical Consistency, while multi-agent systems average 61.9 on Citation Association; single-agent deep research performs worst on association.Gemini 2.5 Pro has the highest consistency score overall (76.5); multi-agent systems’ association advantage is attributed to explicit citation alignment or dedicated citation agents.Source: §5 · p. 9
  • Multi-agent systems average 77.7 on presentation, but polished reports can still contain inconsistent claims because presentation is weakly correlated with association or consistency.Open Deep Research scores 81.0 and Deerflow+ 78.8 on presentation.Source: §5 · p. 9

Coverage and context limits

  • Grok-4 Heavy Deep Research scores 89.3 on Coverage & Comprehensiveness, while Deerflow+ scores 61.6 and Open Deep Research 65.3 despite their strong citation association.Retrieval can scale to thousands of pages (>3,000), straining context; retaining breadth requires tracking processed content and compressing it without losing key evidence.Source: §5 · p. 10

Analysis depth

  • Only Deerflow+ and Gemini Deep Research exceed Open Deep Research on analysis depth; long reports often fail to synthesize sources, leaving many systems as deep searchers rather than deep researchers.Grok-4 Heavy Deep Research, despite leading coverage, barely outperforms Open Deep Research in depth.Source: Figure 6 · p. 10, §5 · p. 10

Trade-offs and open challenges

  • No system achieves coherence, verifiability, breadth, and depth together; improving both coverage and depth requires long-horizon memory, loss-aware hierarchical compression, and explicit synthesis and argumentation.Compression should merge redundant information without losing unique evidence, or prioritize critical details when information is related; representations should adapt to user preferences.Source: §5 · p. 10

6 CONCLUSION … A.3 COVERAGE & COMPREHENSIVENESS

LiveResearchBench and DeepEval provide a benchmark and evaluation suite for citation-grounded deep research, which the paper finds still falls short in citation reliability and analytical depth. The appendix details how selected report-quality criteria are assessed.

6 CONCLUSION

  • The evaluation of 17 leading systems finds that agents can gather and organize information but struggle with citation reliability and analytical depth.The authors identify memory, compression, and synthesis as priorities for enabling more insightful deep research.Source: §6 · p. 10

A.1 PRESENTATION & ORGANIZATION

  • The presentation-and-organization evaluation uses a prompt and checklist, with the same checklist applied to all research questions.The prompt is shown in Figure 7 and the checklist questions in Table 2.Source: §A.1 · p. 16, Figure 7 · p. 16, Table 2 · p. 17

A.2 FACTUAL & LOGICAL CONSISTENCY

  • Factual-and-logical-consistency scores are inversely related to detected inconsistencies, and evaluators can adjust the penalty weight for each error.The grading prompt and scoring rubrics are provided in Figure 8 and Table 3.Source: §A.2 · p. 16, Table 3 · p. 18, Figure 8
  • The consistency criterion targets logical inconsistencies and factual contradictions, not factual accuracy; reusing one source for multiple claims is not itself a contradiction..Source: §A.2 · p. 17

A.3 COVERAGE & COMPREHENSIVENESS

  • Coverage checklists are derived for each research question, with Figure 9 giving the evaluation prompt and Table 4 an example checklist.The checklist for every research question is different.Source: §A.3 · p. 18, Figure 9 · p. 19, Table 4 · p. 19
  • Coverage is scored 1 only when a report fully supplies every component requested by a checklist item; incomplete responses receive 0.Evaluators assign a binary score and justify it by reference to the completeness or incompleteness of the delivered data.Source: §A.3 · p. 18
  • A plan or methodology cannot substitute for requested data, missing-information placeholders earn no credit, and scoring is based on the final report.Any missing, incorrect, or incomplete part of a checklist item requires a score of 0.Source: §A.3 · p. 18

A.4 ANALYSIS DEPTH … B EXPANDED RELATED WORKS

The appendix operationalizes analysis-depth and citation-traceability evaluation, then introduces expanded comparisons of benchmarks and protocols for long-form open-ended information-seeking and deep-research reports.

A.4 ANALYSIS DEPTH

  • Analysis Depth compares reports pairwise and defines depth as moving beyond surface description into reasoning, insight, and layered analysis.Reports receive scores on five dimensions, each rated from 0 to 5; the criteria sum to a total score from 0 to 25.Source: §A.4 · p. 19, §5 · p. 20

1. Granularity of Reasoning (0–5)

  • Granularity of Reasoning scores how specifically a report unpacks ideas, from vague abstractions to causal chains or subcomponents.The rubric anchors scores at 0 for abstract or vague reasoning, 3 for some mechanisms or details, and 5 for consistently specific causal chains or subcomponents.Source: §1 · p. 19

2. Multi-Layered Insight (0–5)

  • Multi-Layered Insight scores whether reports move from surface restatement toward implications, second-order effects, trade-offs, and deeper insights.Its anchors range from surface-level restatement at 0, through some implications or second-order effects at 3, to consistent exploration of trade-offs and deeper implications at 5.Source: §2 · p. 19

3. Critical Evaluation (0–5)

  • Critical Evaluation scores whether reports question assumptions, contrast alternatives, critique claims, weigh scenarios, and probe limitations.The rubric progresses from passive description at 0, through limited questioning or contrast at 3, to strong critique and probing at 5.Source: §3 · p. 19

4. Analytical Use of Evidence (0–5)

  • Analytical Use of Evidence scores whether evidence merely appears or is linked to and consistently advances or sharpens the argument.The anchors range from disconnected facts or examples at 0, through some evidence linked to argument at 3, to evidence that consistently sharpens analysis at 5.Source: §4 · p. 19

5. Insight Density (0–5)

  • Insight Density scores the concentration of substantive analysis, from mostly filler or generic phrasing to highly concentrated insight per token.Scores run from 0 for mostly filler, through 3 for a mix of insight and filler, to 5 for highly concentrated analysis; the five criteria sum to 0–25.Source: §5 · p. 20

A.5 CITATION TRACEABILITY

  • Citation Traceability uses a grading prompt and rubrics to score the number of claims not properly cited, with adjustable error penalties for different use cases.The prompt appears in Figure 11 and the scoring rubrics in Table 5.Source: §A.5 · p. 21, Figure 11 · p. 21

B EXPANDED RELATED WORKS

  • The expanded related-work section compares LiveResearchBench with representative benchmarks and evaluation protocols for long-form open-ended information-seeking and deep-research reports.Source: B EXPANDED RELATED WORKS · p. 21

B.1 BENCHMARK COMPARISON · B.2 EVALUATION PROTOCOLS COMPARISON

No existing benchmark satisfies all the identified criteria, while LiveResearchBench combines expert-curated, human-verified, open-ended multidomain tasks with time-varying scenarios. DeepEval addresses single-judge limitations with multiple evaluation protocols and dimensions for assessing research reports.

B.1 BENCHMARK COMPARISON

  • LiveResearchBench combines human-verified rubrics, expert-curated open-ended multidomain tasks, time-varying scenarios, and ensemble judging; Table 6 compares benchmark dimensions.The comparison dimensions include domain coverage, dynamism, and evaluation methodology.Source: §B.1 · pp. 21–22, Table 6 · p. 22

B.2 EVALUATION PROTOCOLS COMPARISON

  • Independent single-judge runs differed by over 50 points in analysis depth; DeepEval instead combines ensemble judging with checklist-based, additive, pairwise, and rubric-tree protocols.DeepEval enables web search during agent reasoning to retrieve external information; the authors report that its methods reduce variance and provide finer-grained diagnostics.Source: §B.2 · p. 22
  • DeepEval adds Presentation & Organization, Analysis Depth, and Factual & Logical Consistency, with presentation assessed separately to distinguish style from factual or logical errors.The framework also evaluates coverage, citation traceability, and citation accuracy.Source: §B.2 · pp. 22–23
  • ResearcherBench’s coverage, faithfulness, and groundedness roughly correspond to DeepEval’s Coverage & Comprehensiveness, Citation Traceability, and Citation Accuracy, but omit other report-quality aspects.DeepEval additionally evaluates presentation, analysis depth, and factual and logical consistency.Source: §B.2 · pp. 22–23
  • 78% human–LLM agreement for DeepScholar-Bench organization compares with 98.3% agreement for DeepEval’s checklist-based Presentation & Organization metric.DeepScholar-Bench generates related-work sections and uses a single LLM judge; DeepEval’s metric assesses report organization.Source: §B.2 · p. 23
  • 72% agreement for DeepScholar-Bench’s non-verified coverage rubric compares with 100.0% human agreement for DeepEval’s human-validated coverage rubric with Gemini or GPT-5 judgement.DeepEval’s metric is Coverage & Comprehensiveness.Source: §B.2 · p. 23
  • DeepScholar-Bench focuses on academic related-work sections, uses a single LLM judge, lacks human-verified rubrics, and omits Analysis Depth, Factual & Logical Consistency, and Citation Traceability.The paper characterizes its task as narrower than LiveResearchBench’s domain-diverse, open-ended research reports.Source: §B.2 · p. 23

C ALIGNMENT OF LLM JUDGES WITH HUMAN EXPERTS · D DEERFLOW+ · (a) Deerflow (vanilla)

The authors assess whether LLM judges align with human experts and find support for Gemini 2.5 Pro and GPT-5 across evaluation metrics. DEERFLOW+ adds context management and inline citations to stabilize long, search-heavy runs and improve report quality.

C ALIGNMENT OF LLM JUDGES WITH HUMAN EXPERTS

  • 98.3% of Presentation & Organization human preferences favored Gemini 2.5 Pro or GPT-5, leading the authors to average scores from that judge ensemble.Across the three judges, agreement was 67.8%; Gemini 2.5 Pro and GPT-5 agreed in 83.6% of cases.Source: C ALIGNMENT OF LLM JUDGES WITH HUMAN… · p. 23
  • 100.0% of human-aligned Coverage & Comprehensiveness judgments corresponded to Gemini 2.5 Pro or GPT-5, with none uniquely favoring Claude 4 Sonnet.The authors use the Gemini 2.5 Pro and GPT-5 ensemble, averaging their independent scores; agreement across all three judges was 81.5%.Source: C ALIGNMENT OF LLM JUDGES WITH HUMAN… · p. 23
  • 92.5% of disagreement-set cases favored Gemini 2.5 Pro or GPT-5 for Analysis Depth, while humans agreed with LLM judges on 95% of agreement-set samples.Gemini 2.5 Pro and GPT-5 agreed in 89.6% of cases; overall candidate-judge consistency was 78.8%.Source: C ALIGNMENT OF LLM JUDGES WITH HUMAN… · pp. 23–24
  • 82% human agreement for consistency and 85.9% for citation traceability support pointwise additive evaluation, while citation-accuracy judgments aligned with humans in 87.1% of sampled claim-URL pairs.The consistency and traceability rates cover 128 issues identified by GPT-5 and Gemini-2.5 Pro; the authors note final-score alignment may be higher under their weighting scheme.Source: C ALIGNMENT OF LLM JUDGES WITH HUMAN… · p. 24

D DEERFLOW+

  • DEERFLOW+ equips agents with context managers that budget tokens before calls, reserve response capacity, truncate histories progressively, and set role-specific budgets.The authors also add inline citations and validation for one-to-one citation-reference mapping, numbering continuity, and duplicates.Source: D DEERFLOW+ · p. 24
  • DEERFLOW+ completed the full evaluation suite without token-limit failures and improved evidence retention, formatting, factual consistency, and citation-related presentation checks.The cited checks include P4 (Citation Completeness) and P9 (Format Consistency); the authors say the fixes bring DEERFLOW on par with other multi-agent systems in their experiments.Source: D DEERFLOW+ · p. 24

(a) Deerflow (vanilla)

  • The vanilla DEERFLOW example is a broad report on art across historical periods and regions, with period-by-period and regional summaries.Its supplied example includes tables of styles, artists, and techniques or materials, alongside a literature review, methodology, and critical discussion.Source: (a) Deerflow (vanilla) · pp. 25–26

(b) Deerflow+ (ours)

This section combines a dossier tracing how trade, technology, politics, and patronage shaped visual arts and cross-cultural exchange with an assessment of Deerflow+’s evidence retention, structure, and citation practices relative to vanilla Deerflow. It also notes uneven regional coverage and metadata gaps, while reporting that Deerflow+ avoids vanilla Deerflow’s token-limit failures.

Key Points

  • Trade, technical innovation, and patronage reshaped visual production and circulation across Eurasia, while debates over figuration were negotiated locally rather than imposed as uniform bans.The dossier highlights exchanges of artisans, motifs, and technologies, alongside changing markets and courtly and sacred practices.Source: (b) Deerflow+ (ours) · p. 26
  • The normalized dataset and matrices align regional timelines, map transmissions such as Iranian cobalt reaching Chinese porcelain, and flag gaps in Indigenous American and Ancient and Contemporary coverage.They also trace Chinese blue-and-white models reinterpreted in Iranian stonepaste, Venetian adoption of Islamic glass techniques, and Japonisme.Source: (b) Deerflow+ (ours) · pp. 26–27

Overview

  • The dossier combines iconology, social art history, feminist and postcolonial critique, and museum and collecting studies with normalized data and comparative matrices.Its account draws on scholarly resources including the Met’s Heilbrunn Timeline, essays, and collection entries.Source: (b) Deerflow+ (ours) · p. 27

3.1 East Asia -- China

  • In China, advances in printing, porcelain, and underglaze cobalt developed alongside literati and academy aesthetics, while Iranian cobalt and Chinese blue-and-white models crossed regional boundaries.The dossier dates the Diamond Sutra to 868 and notes Yuan-era underglaze cobalt decoration around 1350.Source: (b) Deerflow+ (ours) · p. 28

3.2 East Asia -- Japan

  • In Japan, polychrome ukiyo-e prints emerged around 1765 through registered, multiblock production, serving urban consumers with subjects including actors, courtesans, landscapes, and current events.Designers, carvers, printers, and publishers collaborated; kozo paper supported durable impressions and large editions.Source: (b) Deerflow+ (ours) · p. 28

3.3 Islamic World

  • In the Islamic world, stonepaste enabled underglaze blue and luster painting, muqarnas developed in architecture, and Ottoman workshops codified textiles for court use and export.Courtly figural traditions coexisted with aniconic preferences in sacred contexts, with practices varying locally.Source: (b) Deerflow+ (ours) · p. 29

3.4 Europe

  • Venetian makers adapted Islamic glass techniques and motifs through exchange, while Japanese ukiyo-e influenced European composition, color, and modern subjects amid nineteenth-century print expansion.The dossier notes both industrial print processes and an etching revival.Source: (b) Deerflow+ (ours) · p. 29

Sub-Saharan African Contexts (Interregional Links)

  • From the 9th–16th centuries, East African coastal urban Islamic culture flourished through Indian Ocean trade, exchanging gold, ivory, and slaves for cloth, beads, silks, and porcelain; Portuguese intrusions from 1498 disrupted but did not erase networks [19].In 1000–1600, Guinea Coast and Western/Central Sudan court and ritual arts reflected Islamic and later Portuguese trade; Islam shaped scholarship and monumental adobe architecture in Ghana-Mali-Songhai polities [21][22].Source: (b) Deerflow+ (ours) · p. 29

Cross-Cultural Transmission Pathways

  • Documented vectors include cobalt reaching China, Chinese blue-and-white models reinterpreted in Iran, Islamic glass and textile motifs adopted in Venice, and ukiyo-e influencing Europe.The pathways are presented as examples of bidirectional translation across imperial, mercantile, and artisanal nodes.Source: (b) Deerflow+ (ours) · p. 30

Debates, Institutions, and Markets

  • Palace workshops codified textile designs that diffused into bazaars, ukiyo-e publishers served urban consumers, and 19th-century Europe combined artisanal revival with industrial production and mass advertising [25][26][27][28].The section also frames image practices as negotiated: selective defacement of church mosaics in 8th-century Palestine suggests localized Christian agency rather than a blanket Islamic prohibition [23].Source: (b) Deerflow+ (ours) · p. 31

Validation and Data Quality Flags

  • 6 periods and 2 works are recorded for the Islamic World, versus 0 periods and 0 works for Indigenous Americas, showing sharply uneven regional coverage.The table also records 3 periods and 0 works for Europe, and 3 periods and 2 works for Sub-Saharan Africa.Source: (b) Deerflow+ (ours) · p. 31
  • The dataset flags undated works, absent numeric dates, and missing artist attributions, while all work-technique links and period spans pass the reported checks.It also notes absent Indigenous American information, gaps in ancient and contemporary coverage, and underrepresentation of sculpture and architecture outside documented Islamic and African contexts.Source: (b) Deerflow+ (ours) · pp. 31–32

Critical Discussion

  • Technologies and patronage systems reshaped stylistic possibilities and markets, while materials and models moved bidirectionally and image practices resisted simple figuration/aniconism dichotomies [16][18][23][25][26][27][28].The discussion links these shifts to technologies such as porcelain, stonepaste, muqarnas, silk weaving, and color woodblock registration, alongside imperial workshops, academies, and publishers.Source: (b) Deerflow+ (ours) · p. 32

(b) Deerflow+ (ours)

  • Compared with vanilla Deerflow, Deerflow+ retains more retrieved information, structures evidence more strongly, cites more carefully, and avoids token-limit failures.Source: Figure 12 · p. 33

E EVALUATING CITATION ACCURACY WITH RUBRIC TREE · F LIVERESEARCHBENCH DEMONSTRATION

The section presents a rubric-tree method for evaluating citation accuracy and finds persistent citation errors in leading deep-research systems, especially unsupported claims. It also points to example LiveResearchBench queries across task categories.

E EVALUATING CITATION ACCURACY WITH RUBRIC TREE

  • An agentic judge identifies each claim and linked URL, checks URL accessibility, then assesses whether accessible content supports the claim.This structured validation process is represented as a rubric tree.Source: E EVALUATING CITATION ACCURACY WITH… · p. 33
  • The framework distinguishes inaccessible URLs (E1), task-irrelevant URL content (E2), and content that fails to support a specific claim (E3).These categories separate link-resolution, topical relevance, and claim-support errors.Source: E EVALUATING CITATION ACCURACY WITH… · p. 33, Figure 13 · p. 34
  • In wide information search, most citation errors are unsupported claims rather than invalid or irrelevant URLs, indicating hallucinations persist despite web access.For market analysis, all systems produce large numbers of unsupported claims.Source: E EVALUATING CITATION ACCURACY WITH… · pp. 33–34
  • The rubric tree groups claims sharing a link for one retrieval and checks webpage relevance early to avoid costly full evaluations of irrelevant sources.The coarse check can use a page title or first few hundred tokens; Figure 13 illustrates the grouping and error categories.Source: E EVALUATING CITATION ACCURACY WITH… · p. 33, Figure 13 · p. 34
  • Open Deep Research averages 91.9 unsupported-claim errors per market-analysis report, and all evaluated systems produce non-trivial citation errors.The evaluation covers GPT-5, Grok-4 Deep Research, and Open Deep Research on market analysis and wide information search.Source: E EVALUATING CITATION ACCURACY WITH… · pp. 33–34, Table 7 · p. 34

F LIVERESEARCHBENCH DEMONSTRATION

G ERROR PATTERN EXAMPLES … J.1 ANNOTATION TASKS

The appendices document recurring report-quality errors that informed DeepEval, the annotator and survey inputs behind benchmark design, and human checks for query and checklist quality.

G ERROR PATTERN EXAMPLES

  • Pilot examples span citation mismatches, missing or inconsistent URLs and formats, uncited references, broken tables, out-of-order references, flow-breaking citations, and hallucinated information.The authors say these structural and citation-related issues can undermine report clarity and reliability, motivating DeepEval’s tailored metrics.Source: G ERROR PATTERN EXAMPLES · pp. 34–35
  • E2 errors are counted per problematic URL, while E3 errors are counted per unsupported source attached to a claim.For example, each citation in a multi-source claim that does not support it increments the E3 error count.Source: G ERROR PATTERN EXAMPLES · p. 34

H INFORMATION ON THE ANNOTATION TEAM

  • Salesforce professional annotators held advanced degrees and had 2–6 years of data-annotation and analysis experience.Their expertise included multimodal evaluation of text, images, and audio, as well as quality control for advanced annotation pipelines.Source: H INFORMATION ON THE ANNOTATION TEAM · p. 45

I USER SURVEY

  • The survey found users value real-time, multifaceted research that addresses implicit intent and tailors report content, format, and presentation to its audience.Participants were enterprise professionals (27%), researchers (24%), and general users (49%).Source: I USER SURVEY · p. 45
  • Topic understanding (24.5%), wide information search (22.8%), and top rankings (16.7%) were the most common of ten task categories distilled from survey questions.Other categories were market analysis (8.2%), technical support (6.5%), decision support (5.2%), policy & regulation (4.8%), literature review (4.3%), topic exploration (3.5%), pros & cons (2.0%), and competitive analysis (1.5%). Survey responses also covered seven domains: Science and Technology; Economy and Business; Health and Wellbeing; Law and Governance; Society and Culture; Education and Knowledge; and Media and Entertainment.Source: I USER SURVEY · p. 45

J HUMAN VERIFICATION GUIDELINES

  • Human verification evaluates deep-research query quality and checklist quality to assess whether queries and their checklists support comprehensive report coverage.The stated verification goal is to assess the comprehensiveness and coverage of the report.Source: J HUMAN VERIFICATION GUIDELINES · p. 45

J.1.1 TASK 1: DEEP RESEARCH QUERY QUALITY … J.3 REMARKS AND TIPS

The appendix defines quality checks for research queries and grading checklists, then illustrates them with electric-vehicle and LLM examples. It emphasizes clarity, answerability, verifiability, and avoiding checklist criteria that are redundant, subjective, or impossible to verify.

J.1.1 TASK 1: DEEP RESEARCH QUERY QUALITY

  • Query quality is judged by answerability, clarity, completeness, and—when appropriate—report format and intended audience.Clarifiers may specify temporal bounds, geographic scope, and entities; format and audience criteria should be used judiciously.Source: §J.1.1 · p. 46

J.1.2 TASK 2: GRADING CRITERIA QUALITY

  • Checklist items should be relevant, objectively verifiable, important, and non-redundant; new items should be appended to the corresponding qid.Rate each checklist item individually.Source: §J.1.2 · p. 46

J.2.1 TASK 1: DEEP RESEARCH QUERY

  • The EV-market query is rated appropriate because it specifies a U.S. focus, a concrete time scope, relevant companies, and answerable sub-questions without requiring a particular audience.The example query asks about market size, growth, segmentation, drivers, challenges, and comparisons across major regions; its wording specifies 2025, while the rating describes its temporal scope as 2024.Source: §J.2.1 · p. 46
  • The LLM query needs modification because it leaves the topic broad, omits temporal bounds, and gives vague trend expectations without specifying its intended audience.The example notes that researchers may need technical detail, unlike practitioners or the broader public.Source: §J.2.1 · pp. 46–47

J.2.2 TASK 2: CHECKLIST ITEMS

  • A U.S.-specific EV-market checklist item is appropriate because it is relevant, verifiable, important, and distinct from other criteria.The example checks whether a report includes U.S.-specific market data.Source: §J.2.2 · p. 47
  • A Tesla-only checklist item is valid and verifiable but unnecessary when a broader criterion already covers the queried companies.It is too granular because it omits the query’s wider comparison with BYD, Volkswagen, and Ford.Source: §J.2.2 · p. 47
  • Requiring a report to identify all EV companies is inappropriate because exhaustive coverage lacks authoritative ground truth and is unreasonable to expect.The suggested alternative is to check for specific entities named in the query, such as Tesla, BYD, Volkswagen, and Ford.Source: §J.2.2 · p. 47

J.3 REMARKS AND TIPS

  • Annotation guidance says to rate each query once per qid and checklist items individually, while avoiding unverifiable completeness, redundancy, scope creep, and subjective judgments.Checklist items should assess verifiable facts rather than opinions.Source: §J.3 · p. 47

K RESULTS BREAKDOWN

Presentation results show strong performance on structure, citation placement, and figure/table validity, but persistent difficulty with style, grammar, and citation management. Provider patterns and pipeline design also shape citation-related performance.

K RESULTS BREAKDOWN

  • GPT-5, GPT-4.1, and GPT-5 Mini score 31%, 32%, and 24% on P6, respectively, illustrating correlated strengths and weaknesses among sibling models.The paper suggests shared data and formatting designs may contribute to these provider-level regularities.Source: K RESULTS BREAKDOWN · p. 47
  • Multi-agent wrappers around a common backbone often improve P4 by explicitly mapping in-text citations to references, while P10 numbering accuracy varies by pipeline.Some single-agent web systems show steadier numbering accuracy.Source: K RESULTS BREAKDOWN · p. 47, Figure 22 · p. 48
  • Systems are strong on P1 structure and near-saturated on P7 citation placement and P8 figure/table validity, while grammar, style, and reference alignment vary widely.The hardest cases include style/formatting (P2, P5–P6, P9) and citation management (P3–P4, P10).Source: K RESULTS BREAKDOWN · p. 47, Figure 22 · p. 48

Related papers

Loading 2510.14240v5…