PaperlayerView exact revision on arXiv

Source-linked AI summary

Benchmarking Hybrid Deep Research Across Database Querying and Web Search

Ruofan Wu, Peiran Xu, Xiaolong Li, Fan Shu, Soyoung Yoon, Yite Wang, Xiaodong Yu, Boyi Liu, Feng Yan, Debiao Li, Yuxiong He, Zhewei Yao

arXiv:2609.09410v1cs.CL

TL;DR

Existing deep-research evaluations largely isolate web search from database querying, leaving cross-system constraint handoffs insufficiently measured. HybridDeepResearch introduces a tool-dependent benchmark spanning both modalities and finds that current agents remain unreliable, particularly on directional handoffs. The benchmark’s controlled databases and web corpora also limit how broadly its conclusions generalize.

  • Problem

    Existing evaluations usually assess web search and database reasoning separately, despite real analytical tasks requiring constraints to move between these modalities.

  • Method

    HybridDeepResearch evaluates 380 tool-dependent questions using SQL2S, S2SQL, and Parallel patterns across LiveSQLBench-Base-Lite databases and public web evidence.

  • Results

    Directional handoffs remain substantially harder than parallel intersection, and leading systems reach only about 50–54% Pass@8 on the balanced hard subset.

  • Takeaways & Limitations

    Preserving and reconciling constraints across web search and SQL remains a key challenge for deep-research agents.

  • Takeaways & Limitations

    The benchmark uses nine controlled databases and public corpora, so it does not cover proprietary schemas, specialized scientific databases, or the full diversity of the live web.

Abstract

from arXiv · show

While autonomous agents have made significant strides in "deep research" by iteratively navigating the open web to synthesize information, real-world problem-solving is rarely confined to a single environment. Complex analytical tasks inherently require agents to weave together evidence from both ambiguous unstructured text (e.g., the open web) and highly precise structured data (e.g., relational databases). However, existing benchmarks evaluate these modalities in isolation, failing to capture the critical "handoff" - the ability to preserve constraints when moving evidence between systems. We introduce HybridDeepResearch, to our knowledge the first deep-research benchmark that requires both web search and SQL to form a complete, verifiable answer. The benchmark contains 380 tool-dependent tasks grounded in LiveSQLBench-Base-Lite databases and public web corpora, validated through automated checks and human review, and covering three reasoning patterns: SQL2S, S2SQL, and Parallel. Evaluations across proprietary and open-weight models under various agentic scaffolds reveal that even state-of-the-art models like GLM-5.2, Claude-Sonnet-4.6 and GPT-5 achieve only about 50-54% Pass@8 on the hard subset. Notably, results show that directional reasoning is substantially more difficult than parallel intersection, highlighting that bridging structured and unstructured information spaces without losing constraints remains a major open challenge for agentic systems. Code and datasets are publicly available at GitHub (https://github.com/Snowflake-AI-Research/HybridDeepResearch) and Hugging Face (https://huggingface.co/datasets/Snowflake/HybridDeepResearch).

1 Introduction

HybridDeepResearch addresses the gap between isolated web-search and database evaluation by measuring whether agents can transfer constraints across both modalities. Its benchmark and evaluations show that this handoff remains unreliable, especially for directional reasoning.

  • Motivation: Real-world analytical tasks require integrating structured databases with ambiguous unstructured sources, but existing evaluations largely assess these capabilities separately.The paper frames cross-system constraint preservation as a central requirement for financial auditing, supply-chain risk assessment, and scientific research.
  • Benchmark: HybridDeepResearch is a diagnostic benchmark in which every task requires a verifiable intersection of web search and database querying.The benchmark organizes tasks around SQL2S, S2SQL, and Parallel handoff patterns.
  • Reasoning patterns: The benchmark defines SQL2S, S2SQL, and Parallel to test different directions of evidence transfer between databases and web search.SQL2S transfers a database-recovered entity to search, S2SQL turns a search discovery into a database predicate, and Parallel intersects independently generated candidate sets.
  • Benchmark: 380 tool-dependent questions are grounded in LiveSQLBench-Base-Lite databases and public evidence, with a balanced 120-example hard subset.The questions are automatically validated and human-reviewed, and the hard subset contains 40 examples for each reasoning pattern.
  • Results: About 50–54% Pass@8 is achieved by the strongest proprietary systems on the balanced hard subset, while overall Avg@8 remains below 30%.The largest gaps occur on SQL2S and S2SQL, indicating that directional handoffs are harder than independent or parallel reasoning.
  • Results: The evaluations identify evidence transfer and reconciliation across distinct modalities as a critical bottleneck for current agents.The contribution is framed as revealing a performance bottleneck rather than solving the handoff problem.

2 Related Work

Prior benchmarks separately study web-based deep research, database interaction, and limited hybrid settings. HybridDeepResearch extends this landscape by targeting their intersection.

  • Existing evaluation traditions: Deep-research benchmarks emphasize iterative web search and reasoning over retrieved evidence, while database benchmarks evaluate natural-language-to-SQL and multi-step database interaction.The paper situates HybridDeepResearch among three neighboring evaluation traditions: deep research, database agents, and hybrid benchmarks.

3 Hybrid Reasoning Patterns

HybridDeepResearch organizes cross-modal handoffs into SQL2S, S2SQL, and Parallel, each testing whether agents preserve constraints while moving between databases and web search. Directional patterns require translating an intermediate result into the next tool, whereas Parallel requires exact set intersection.

  • Overview: The three patterns expose the gap between isolated tool proficiency and cross-modal orchestration by varying how evidence and constraints flow.The patterns are designed to stress structurally complex, ambiguous, and distractible constraints passed between tools.
  • SQL2S: SQL2S uses a database result as a rigid anchor for targeted web exploration.A common failure is the salience trap: a generic search retrieves a prominent result about the wrong company instead of the SQL-identified entity.
  • S2SQL: S2SQL requires translating an entity or condition found on the web into a strict SQL predicate for the follow-up query.The motivating example identifies a striking company through news search before calculating database spend with that company.
  • Parallel: Parallel solves the SQL and web components independently and returns only their exact candidate-set intersection.Its failure mode is premature fusion, in which the agent returns all candidates from one source rather than the precise overlap.
  • Shared capability: All three patterns test whether agents can preserve, translate, and reconcile semantic constraints across structurally distinct modalities.The benchmark uses these templates to enforce a verifiable hybrid lock.

4 HybridDeepResearch Construction

HybridDeepResearch constructs hybrid tasks by anchoring independently generated web-search and SQL components to shared entities, composing them into three handoff patterns, and filtering for genuine two-tool dependence.

  • Construction requirements: Each instance links independently verifiable search and SQL components through a shared real-world entity and requires both modalities for the final answer.Violations produce ambiguous, single-source-solvable, or only superficially hybrid questions.
  • Unstructured Track: Search questions use refined multi-hop evidence graphs, controlled clue blurring, and LLM rendering to reduce ambiguity and direct entity leakage.Evidence comes from Wikidata, Wikipedia, and FineWeb-10BT.
  • Dual-Grounded Entity Anchoring: Database-grounded entities are disambiguated with global salience checks and schema-specific SQL exploration before cross-modal generation.The resulting description captures the entity’s precise database meaning and guides matching web evidence.
  • Structured Track: SQL questions use reusable advanced patterns instantiated according to each seed’s role in S2SQL, SQL2S, or Parallel.The catalog contains 35 patterns across 6 categories and three difficulty levels, with execution and human-quality checks.
  • Cross-Modal Composition and the Hybrid Lock: Directional composition passes one modality’s exact output as the other’s constraint, whereas Parallel retains only the intersection of candidate sets.Final items undergo single-tool ablations to remove cases solvable through web search or SQL alone.

5 HybridDeepResearch Statistics

The full benchmark contains 380 questions across nine databases, with uneven hybrid-type coverage and concentrated evidence depth; a balanced 120-question hard subset supports controlled comparisons.

  • Dataset composition: 380 hybrid questions span nine LiveSQLBench-Base-Lite databases and cover hybrid type, source database, answer category, and evidence depth.Final answers include entities as well as numerical or tabular SQL results.
  • Dataset composition: 203 SQL2S questions, 119 Parallel questions, and 58 S2SQL questions make the full set imbalanced across hybrid types.News and cross_db contain most questions, while seven smaller domains contain the remainder.
  • Dataset composition: Evidence depth is concentrated at five and six steps, defined as search reasoning steps plus one SQL step.The dataset covers people, places, countries or regions, organizations, works, and numerical or tabular results.
  • Database knowledge and hints: Most questions use one to four database-knowledge snippets, while 10 questions (2.6%) include explicit hints for rules unavailable from schema alone.Hints may specify thresholds, weighted formulas, or exclusion criteria.
  • Balanced hard subset: The balanced hard subset contains 120 manually selected questions, with 40 examples from each hybrid type, for controlled comparisons under limited API budgets.Selection emphasizes more challenging questions from the full set.

6 Evaluation

Evaluation shows that multi-agent scaffolds can improve the chance of solving hybrid tasks, but gains are less reliable across repeated trials and directional handoffs remain the central bottleneck. Across model comparisons, no system dominates every task type, and even the strongest models solve only about half of hard-subset questions under Pass@8.

  • RQ1: MiroFlow raises overall Pass@8 from 58.95 to 66.32 with Real Web and from 56.58 to 63.42 with Corpus Search, while Avg@8 remains nearly unchanged.The result suggests more first-attempt opportunities without consistent improvement across repeated trials.
  • RQ1: Real Web provides a small Pass@8 advantage over Corpus Search under both scaffolds, while Avg@8 differs only slightly.The fixed-corpus backend remains competitive while offering a more controlled evaluation setting.
  • RQ2: Across matched Qwen sizes, MiroFlow improves Pass@8, especially on directional S2SQL and SQL2S, but Avg@8 is less consistent and Parallel sometimes declines.The scaffold’s gains are therefore not uniform across metrics or reasoning patterns.
  • RQ2: GLM-5.2-FP4 is the strongest open-weight model under both scaffolds, reaching 75.79 Pass@8 and 49.24 Avg@8 overall with MiroFlow.Under MiroFlow, its S2SQL Pass@8 is 62.07 versus 93.28 on Parallel, and the Pass@8–Avg@8 gap indicates inconsistent trial-level success.
  • RQ3: GPT-5 achieves the highest hard-subset Pass@8 at 54.17, while Claude-Sonnet-4.6 achieves the highest overall Avg@8 at 28.12; no model dominates every task type.GLM-5.2 ties for the highest Parallel Pass@8 and has the highest Parallel Avg@8.
  • Failure analysis: Agents fail by returning one-source candidates on Parallel, dropping SQL constraints during SQL2S web search, or applying incorrect SQL operations after S2SQL entity identification.These failure modes describe distinct ways that cross-tool constraints are lost during handoff.
  • Failure analysis: MiroFlow’s extra worker attempts can recover missed constraints and improve Pass@8, but they require more calls, cause more timeouts, and do not consistently improve Avg@8.The scaffold therefore trades broader exploration for higher resource use and uneven reliability.

7 Conclusion

HybridDeepResearch benchmarks agents that must jointly reason over open-web evidence and relational databases across three hybrid task patterns. Results show that current agents remain unreliable at cross-modal handoffs, although multi-agent scaffolds improve Pass@8 while directional tasks remain harder than parallel intersection.

  • Conclusion: HybridDeepResearch contains 380 questions spanning SQL2S, S2SQL, and Parallel, plus a balanced 120-example hard subset.The benchmark is designed around hybrid reasoning rather than isolated web search or SQL use.
  • Conclusion: Across proprietary and open-weight models, agents remain unreliable when information must be passed between web search and SQL.The benchmark identifies cross-tool consistency as a central challenge for deep-research agents.
  • Conclusion: Multi-agent scaffolds improve Pass@8, but directional tasks remain substantially harder than parallel intersection.The conclusion links higher occasional success with a persistent difficulty in preserving information across directional handoffs.

Limitations

The benchmark’s coverage is constrained by its database and web-corpus setting and by an imbalance across hybrid task types. A balanced hard subset supports fairer comparisons, while broader database coverage and more S2SQL examples remain future directions.

  • Scope: The benchmark uses nine LiveSQLBench-Base-Lite databases and public corpora such as Wikipedia and FineWeb-10BT, excluding proprietary schemas, specialized scientific databases, and the full diversity of the live web.An expanded release and public leaderboard are planned using a newly built database.
  • Task distribution: The full set is imbalanced across hybrid types: SQL2S is largest, while S2SQL contains only 58 of 380 questions.The balanced 120-example hard subset provides 40 examples per type, but increasing S2SQL coverage remains a future direction.

Ethical Considerations

The benchmark is intended for research evaluation, not high-stakes deployment, because its public sources and agent outputs may contain errors, bias, objectionable content, or unsupported results.

  • The benchmark is intended for evaluating research agents rather than deployment in high-stakes settings.
  • Public web evidence may inherit factual errors, biases, or objectionable content from its sources.
  • Agent outputs may contain unsupported claims or incorrect database results and should be independently verified before practical use.
  • Repeated multi-agent evaluation incurs non-trivial computational cost.
  • Hybrid tasks require preserving constraints across web evidence and database records, as illustrated by entity-specific SQL and search examples.

D Experimental Details

Experiments compare agent frameworks, retrieval backends, models, and task-specific evaluation procedures under controlled tool and trial settings.

  • Agent frameworks: smolagents uses one tool-calling agent, whereas MiroFlow delegates subtasks to specialized workers through a hierarchical agent graph.
  • Retrieval backends: Real Web searches the live web, while Corpus Search retrieves from a fixed Wikipedia and FineWeb-10BT index.
  • Evaluation scope: Open-weight models are evaluated on all 380 tasks, while Claude-Sonnet-4.6 and GPT-5 are evaluated on the 120-example hard subset.
  • Task-specific evaluation: S2SQL compares executed predicted and gold SQL results, whereas SQL2S and Parallel use an LLM judge for free-text answers.
  • Repeated trials: Each task receives eight independent trials; Pass@8 records solving at least once, while Avg@8 measures average success across trials.
  • Structured tools: SQL tools are read-only and permit SELECT, WITH, PRAGMA, and EXPLAIN statements, with schema discovery left to agents.

G.1 Trajectory Cost

Trajectory analysis shows that multi-agent exploration can improve constraint recovery and Pass@8, but it substantially increases calls, timeouts, and completion cost.

  • Trajectory Cost: MiroFlow uses more LLM and tool calls than smolagents, while maintaining similar or lower cumulative input-token counts through shorter worker contexts.
  • Trajectory Cost: 41.3% of MiroFlow runs time out versus 16.7% for smolagents, indicating substantially higher completion cost.
  • Representative Failure Cases: Parallel failures occur when agents return a candidate supported by only one source instead of the exact intersection of SQL and web candidate sets.
  • Representative Failure Cases: Directional failures include losing a SQL-identified constraint during search and producing incorrect SQL after finding the correct web entity.
  • Trajectory Cost: MiroFlow improves Pass@8 while Avg@8 remains close to smolagents because additional worker attempts recover missed constraints but increase calls and timeouts.

H.3 Evaluating Database Agents

Existing database-agent benchmarks extend text-to-SQL toward contextual, interactive, and knowledge-assisted settings, but generally do not require discovering evidence through open-web search.

  • Early text-to-SQL benchmarks evaluated mapping natural-language questions to executable queries.
  • Later benchmarks added contextual and conversational queries, larger databases, external knowledge, execution efficiency, and agent-oriented SQL diagnosis or repair.
  • Hybrid benchmarks evaluate complementary structured and textual evidence, but generally assume relevant tables and passages were collected in advance.
  • Related hybrid database systems combine execution with language-model reasoning or knowledge without requiring agents to discover supporting evidence through open-web search.
Loading 2609.09410v1…