Source-linked AI summary
Can AI Agents Answer Your Data Questions? A Benchmark for Data Agents
Ruiying Ma, Shreya Shankar, Ruiqi Chen, Yiming Lin, Sepanta Zeighami, Rajoshi Ghosh, Abhinav Gupta, Anushrut Gupta, Tanmai Gopal, Aditya G. Parameswaran
TL;DR
Existing benchmarks do not evaluate the full complexity of enterprise data-agent workloads across heterogeneous data sources. This paper introduces DAB, a realistic multi-database benchmark grounded in enterprise workloads, and finds that even the best frontier model achieves only 38% pass@1 while planning and implementation dominate errors.
Problem
Existing benchmarks evaluate isolated tasks rather than integrating, transforming, and analyzing data across multiple heterogeneous database systems.
Method
The authors construct DAB from enterprise workload patterns, covering 54 queries across 12 datasets, 9 domains, and 4 database systems, and evaluate five frontier models.
Results
Even the best frontier model achieves only 38% pass@1, with incorrect planning and implementation identified as the dominant challenges.
Takeaways & Limitations
Data-agent development should improve planning, data exploration, extraction tools, and semantic support to address the benchmark’s observed failure modes.
Takeaways & Limitations
DAB excludes open-ended reasoning and API integration because deterministic ground-truth evaluation is required, and model selection was constrained by API credit availability.
Abstract
from arXiv · showhide
Users across enterprises increasingly rely on AI agents to query their data through natural language. However, building reliable data agents remains difficult because real-world data is often fragmented across multiple heterogeneous database systems, with inconsistent references and information buried in unstructured text. Existing benchmarks only tackle individual pieces of this problem -- e.g., translating natural-language questions into SQL queries, answering questions over small tables provided in context -- but do not evaluate the full pipeline of integrating, transforming, and analyzing data across multiple database systems. To fill this gap, we present the Data Agent Benchmark (DAB), grounded in a formative study of enterprise data agent workloads across six industries. DAB comprises 54 queries across 12 datasets, 9 domains, and 4 database management systems. On DAB, the best frontier model (Gemini-3-Pro) achieves only 38% pass@1 accuracy. We benchmark five frontier LLMs, analyze their failure modes, and distill takeaways for future data agent development. Our benchmark and experiment code are published at github.com/ucbepic/DataAgentBench.
1 INTRODUCTION
Existing benchmarks do not evaluate the end-to-end challenges of answering natural-language questions across fragmented, heterogeneous enterprise data. DAB addresses this gap with realistic multi-database queries and finds substantial limitations in current agents.
- Motivation: Enterprise data agents must often integrate fragmented data across databases, inconsistent identifiers, and unstructured text.These requirements make production workloads more complex than single-database text-to-SQL or table question answering.
- Motivation: Existing text-to-SQL and Table-QA benchmarks isolate single-query translation or reasoning over tables supplied in context.They do not test multi-step reasoning, direct database querying, or cross-database integration.
- Findings: 85% of wrong answers arise from incorrect planning or faulty implementation, while agents rarely select the wrong data sources.All agents use regular expressions for free-text extraction, and none attempts NLP- or LLM-based extraction.
- Evaluation: 38% pass@1 is achieved by the best model evaluated on DAB.The paper evaluates five frontier LLM agents using a ReAct-style loop with database-querying and Python-execution tools.
- Findings: PromptQL improves pass@1 by 7 percentage points over a same-model ReAct baseline, but both approaches fail entirely on unstructured-text extraction queries.The comparison suggests specialized infrastructure helps overall performance without resolving this failure mode.
2 BENCHMARK CONSTRUCTION
DAB is constructed from enterprise query patterns by systematically perturbing open-source datasets to require multi-database integration, identifier reconciliation, text transformation, and domain knowledge. Each query has a deterministic answer and validation process for reproducible evaluation.
- Formative Study: The formative study interviewed enterprise customers across six industries and analyzed their schemas, database systems, data distribution, and example queries.The study identified recurring challenges that existing benchmarks do not address.
- Benchmark Properties: DAB induces four properties: multi-database integration, ill-formatted join keys, unstructured text transformation, and domain knowledge.The benchmark excludes open-ended reasoning and live API integration because reproducible deterministic ground truths are required.
- Dataset Selection: The benchmark uses 12 open-source datasets spanning domains including news, e-commerce, software engineering, finance, medicine, and patents.The datasets are selected to mirror patterns observed in the formative study.
- Dataset Transformation: Dataset construction removes trivially useful columns, reembeds values into free text, reformats join keys, and distributes tables across at least two DBMSes.These transformations require agents to recover structured values and reconcile schema and query-dialect differences.
- Query and Validation: Each query includes a natural-language question, deterministic ground-truth answer, and validation script, and incorporates at least two benchmark properties.Two authors derive answers from the original data before transformation, while authors and PromptQL independently verify the benchmark.
- Benchmark Scale: All 54 queries require joining data across multiple databases, with some crmarenapro queries spanning six databases and stockmarket queries navigating 2,754 tables.DAB accompanies each dataset with descriptions and hints that can expose schema information and transformation requirements.
3 EXPERIMENTS
The experiments show that frontier LLM agents perform poorly on DAB, with substantial cost, reliability, exploration, planning, implementation, and text-extraction challenges. Specialized infrastructure improves accuracy, but unstructured-text queries remain unsolved.
- Accuracy: 38% pass@1 was the highest accuracy, achieved by Gemini-3-Pro; GPT-5-mini reached 30%, GPT-5.2 25%, Kimi-K2 23%, and Gemini-2.5-Flash 9%.
- Accuracy: No agent solved patents across all trials, while deps_dev_v1 reached only 6% pass@1 at best.
- Accuracy: At k=50, the best agent reached 69% pass@50, and additional attempts remained insufficient for many queries.
- Cost: GPT-5-mini achieved the best cost-accuracy tradeoff: 30% pass@1 at $67 total, versus Gemini-3-Pro’s 38% at $1,355.
- Trajectory patterns: SQL aggregation improved cost-efficiency: GPT-5-mini averaged a 2.6:1 DB-to-Python ratio and cost $67, whereas Kimi-K2 averaged 1.1:1 and cost $1,304.
- Trajectory patterns: Hard stockmarket queries produced 50+ tool calls and over 10 minutes of latency while pass@1 remained near zero, showing that more iterations alone did not solve them.
- Trajectory patterns: The two highest-accuracy agents each allocated roughly 20% of tool calls to exploration, while both insufficient and excessive exploration underperformed.
- Failure analysis: Incorrect implementation and incorrect planning accounted for 85% of wrong answers, while incorrect data selection accounted for 15%.
4 RELATED WORK
DAB addresses limitations in existing data-querying and tool-use benchmarks by evaluating realistic end-to-end data-agent workflows across four challenging properties.
- Text-to-SQL: Text-to-SQL benchmarks evaluate single-query translation over one relational database, unlike DAB’s broader multi-database workflow.They do not cover multi-database integration or reconciliation of ill-formatted join keys.
- Table question answering: Table-QA benchmarks test reasoning over tables supplied in the prompt, whereas production tables must be queried directly from databases.DAB embeds table reasoning within an end-to-end querying workflow over real database systems.
- Benchmark coverage: DAB covers multi-database integration, ill-formatted join keys, unstructured text transformation, and domain knowledge.Table 8 summarizes whether related benchmark categories cover these four properties fully, partially, or not at all.
- Tool-use benchmarks: Existing tool-use benchmarks target external tools such as function calling, code editing, browsing, and command-line tasks, but generally lack database interaction.GAIA combines multi-step reasoning with web browsing, while other data-adjacent benchmarks expose narrower workflows.
5 CONCLUSION
The paper introduces DAB as a realistic benchmark for multi-database data agents and finds that frontier agents remain unreliable, especially in planning and implementation.
- Conclusion: DAB contains 54 queries across 12 datasets, 9 domains, and 4 database systems, requiring multi-database integration and additional data-transformation capabilities.Its required properties include ill-formatted join keys, unstructured text transformation, and domain knowledge.
- Conclusion: 38% pass@1 is achieved by the best frontier model on DAB.The result indicates low single-attempt accuracy on the benchmark.
- Conclusion: Incorrect planning and implementation are the dominant challenges identified by the error analysis.The conclusion positions DAB as a benchmark intended to support development of data agents that can be trusted in production.
A DATASETS AND QUERIES IN DAB
DAB’s appendix lists the benchmark’s query inventory, including operational, software, business, music, biomedical, patent, and financial questions with dataset-specific policies and conditions.
- Query inventory: Table 9 provides all queries in DAB, and the raw data are publicly available in the project repository.The appendix presents the query inventory rather than aggregate benchmark results.
- Operational and business queries: The queries require constrained outputs such as agent IDs, product IDs, knowledge-article IDs, state abbreviations, rankings, counts, proportions, and statistical values.Examples include case closure, handle time, transfers, sales, and policy-breach questions.
- Dataset policies: Several policies define how benchmark quantities are computed, including handle time from case opening to closure and sales amount as Quantity * UnitPrice.Transfer-count rules specify that a transfer from agent A to agent B increments agent A’s count.
- Cross-domain queries: Dataset queries combine filters, aggregations, rankings, temporal conditions, and domain-specific definitions across repositories, businesses, music, genomics, patents, and markets.Examples include GitHub popularity, business ratings, song revenue, gene mutations, patent trends, and stock-index performance.
- Query constraints: Many queries impose explicit eligibility and exclusion conditions, such as date windows, release status, language or license filters, valid annotations, and minimum marginal totals.These conditions appear across software, biomedical, patent, and financial tasks.
C FAILURE DESCRIPTION AND EXAMPLES
The failure-analysis appendix defines the agent tools, trace-based annotation procedure, and five failure modes used to diagnose incorrect answers.
- Annotation setup: The annotation task diagnoses why an agent’s final answer differs from ground truth using failure modes FM1–FM4.The prompt supplies the failed trace, query, and ground-truth answer for analysis.
- Tools and traces: Agents access list_db, query_db, execute_python, and return_answer to inspect databases, query data, process results, and terminate with an answer.The trace records exploratory calls and calls contributing directly to the final answer.
- Annotation rules: Annotators must base failure analysis on contributing tool calls and identify concrete trace evidence for each selected failure mode.The prompt requires at least one failure mode and generally favors one most-specific explanation.
- FM1: FM1 covers failure before planning, including declining to solve or issuing no tool calls.These cases are classified directly from the termination or error type.
- FM2–FM3: FM2 denotes an incorrect logical plan, while FM3 denotes correct planning paired with incorrect data selection.Examples include missing or extra operations, premature termination, wrong columns, and wrong tables.
- FM4: FM4 denotes correct planning and data selection but incorrect implementation, including arithmetic, parsing, regular-expression, or join errors.The mode includes failures to normalize identifiers before joining despite recognizing that alignment is required.
C.2 Examples of Failed Trajectories
This section describes failed book-review trajectories involving two databases, cross-table book matching, and query-specific analytical constraints. The examples cover incorrect aggregation, omitted constraints, and unnecessary result limits.
- Data and queries: The examples use separate SQLite review data and PostgreSQL book data, requiring information to be combined across database systems.The review database contains ratings and review metadata, while the book database contains titles, descriptions, prices, and related information.
- Data and queries: The databases identify the same books through book_id and purchase_id, which can be joined using a fuzzy join.The fields have different names but refer to the same book entities across tables.
- Data and queries: The example queries ask about decade-level ratings, perfect-rating books in Literature & Fiction, and highly rated Children’s Books reviewed since 2020.The listed ground-truth answers include 2020 for Q1 and book-title lists for Q2 and Q3.
- Analysis scope: The section analyzes selected example trajectories rather than presenting full interaction traces.Complete trajectories are omitted for presentation clarity and are available separately.
C.2.1 FM2 : Incorrect Plan for Averaging.
The Q1 trajectory uses an incorrect averaging plan: it averages per-book average ratings instead of directly averaging all ratings within each decade.
- Failure analysis: The trajectory computes decade-level averages by averaging per-book average ratings, producing an incorrect aggregation.The query requires directly averaging all ratings within each decade.
C.2.2 FM2 : Missing Operations.
The Q2 trajectory fails because its operations omit the requirement that selected books have an average rating exactly equal to 5.0.
- Failure analysis: All operations contributing to the Q2 answer ignore the constraint that the average rating must equal 5.0.The final answer depends on multiple earlier Python and database calls that preserve this omission.
C.2.3 FM2 : Adding Operations.
The Q3 trajectory adds unnecessary row limits when retrieving reviews and books, although the query does not require them.
- Failure analysis: The trajectory adds LIMIT 200 for review records and LIMIT 500 for book records without query support.These operations constrain retrieval even though the question specifies no such limits.
C.2.4 FM3 : Incorrect Column Selection.
The trajectory fails because the agent selects an incorrect column when filtering books written in English, despite the database description identifying the correct column.
- The agent searches for “English” across details, description, and other columns instead of relying on the correct column indicated by the database description.This incorrect column selection causes the trajectory to fail.
C.2.5 FM4 : Incorrect Regular Expression.
The trajectory fails because an overly permissive regular expression extracts misleading four-digit values and selects the wrong publication year.
- The agent extracts any four-digit string beginning with 19 or 20, then selects the smallest extracted value as the publication year.This can select 1932 from an ISBN instead of the true publication year, 2004.