Source-linked AI summary
From Data Querying to Data Investigations: Rethinking Natural Language Interfaces for Databases
Fabian Wenz, Zixuan Chen, Carsten Binnig
TL;DR
Text-to-SQL interfaces assume a user’s question can be answered by one query, although practical data problems often require iterative exploration and reasoning across intermediate results. The paper proposes data investigations and presents D^2, a multi-agent prototype with a shared evidence board. On a 50-case benchmark, D^2 is comparable in correctness to baselines but more complete and verifiable.
Problem
Text-to-SQL assumes natural-language questions can be answered with a single SQL query, while data problems may require iterative querying and reasoning over intermediate results.
Source: §1 · p. 1Method
Results
Across 50 benchmark cases, D^2 is comparable in correctness but more complete and verifiable, reaching 76.02% completeness while both baselines remain near 32%.
Source: §4.1 · p. 5Takeaways & Limitations
Data investigation systems can address open-ended data problems by retaining alternative hypotheses and organizing the evidence that supports or contradicts them.
Source: §1 · p. 1, §2 · p. 3Additional material is slated for a technical report that the authors say they will publish on arXiv.
Source: §4 · p. 5
Abstract
from arXiv · showhide
Natural language (NL) interfaces to databases have been optimized for the wrong problem. The dominant Text-to-SQL paradigm assumes that users ask questions that can be answered by single SQL queries. In practice, however, users seek assistance with solving data problems. This requires searching a database by sequences of SQL queries while reasoning over intermediate results instead of just running one SQL query. This paper therefore introduces a new paradigm for NL interfaces to data, which we call data investigations. We present D^2, a first prototype of a data investigation system that embodies this vision by autonomously searching, reasoning over, and collecting data to solve data problems. Using a newly constructed benchmark based on the Murder Mystery dataset, we demonstrate the potential of D^2 for tasks that require data investigations with evidence-backed decisions, extending beyond the capabilities of traditional single-query question answering.
1 Introduction
Text-to-SQL treats database assistance as translating a natural-language question into one SQL query, but many user problems require iterative exploration and reasoning across queries. The paper proposes data investigations as a broader workload for solving data problems with evidence-backed conclusions.
1 Introduction
- Many practical questions, such as comparing treatments, require exploring alternatives, populations, outcomes, and contributing factors rather than issuing one SQL query.The result is a collection of competing hypotheses supported by different evidence.Source: §1 · p. 1
- Figure 1 contrasts Text-to-SQL’s single-query direct result with investigations that use multiple exploratory queries and reasoning steps to verify or falsify solutions.Source: Figure 1 · p. 1, §1 · p. 1
- Data investigations form a distinct workload: systems iteratively explore data, formulate hypotheses, collect evidence, revise understanding, and communicate conclusions.The search proceeds through query sequences whose execution depends on reasoning over intermediate results.Source: §1 · p. 1
- The paper proposes data investigation systems, presents D^2 as a first prototype, introduces a benchmark, and outlines an agenda for future systems.It argues that current pure agentic approaches can stop at one plausible answer and leave reasoning difficult to inspect.Source: Figure 2 · p. 2
2 Our Vision: Data Investigation Systems
A data investigation begins with a high-level objective, then explores available data to develop and test candidate hypotheses. The vision pairs a multi-agent Data Detective with a shared Data Investigation Board that records evolving investigation state.
2 Our Vision: Data Investigation Systems
- A data investigation explores the database before the relevant tables and queries are necessarily known, then gathers evidence to support or contradict candidate hypotheses.An initial query result can contradict a plausible hypothesis, so the system continues examining alternatives.Source: §2 · p. 2
- The workload progressively discovers relevant hypotheses and evidence through exploration, querying, and reasoning about which hypotheses hold.Source: §2 · p. 3
- The Data Detective coordinates specialized agents for generating hypotheses, collecting evidence, and evaluating each hypothesis against that evidence.A multi-agent design is intended to avoid implicit state tracking and premature convergence on a single hypothesis.Source: §2 · p. 3
- The Data Investigation Board externalizes evolving state by linking hypotheses to supporting or contradicting evidence in persistent, shared memory.It supports parallel investigative branches, preserves rejected and supported hypotheses, and helps users inspect the reasoning behind conclusions.Source: §2 · p. 3
3 Our Prototype: The 𝐷2 System
D^2 is a domain-independent prototype that organizes data investigations around specialized agents and a shared board. Its agents generate, select, test, and refine hypotheses, while the board records evidence and links it to hypotheses for collaboration and review.
3.1 The Data Detective
- The Data Detective uses five specialized roles rather than a ReAct-style agent that stops at its first plausible answer; agents share state through the board.Source: §3.1 · p. 3, Figure 3 · p. 4
- The Hypothesizer explores the schema and data to propose multiple candidates, while the Orchestrator selects what to investigate and when to stop without issuing SQL itself.In the example, candidates include treatment benefit, comorbidity-dependent effects, and dosage-dependent effects.Source: §3.1 · p. 3
- Independent Evidence Collectors query for evidence in parallel; the Judge reasons over collected evidence to confirm or contradict hypotheses and can request further evidence.The judge may refine hypotheses, sending new subgroup candidates through another investigation round.Source: §3.1 · p. 3, Figure 3 · p. 4
- After investigations terminate, the Reporter cleans redundant evidence and produces a concise audit graph connecting hypotheses to evidence for user verification.D^2’s workflow is described as domain-independent, with the database replaceable without changing the investigation process.Source: §3 · p. 3, Figure 3 · p. 4
3.2 The Data Investigation Board
- The board stores candidate hypotheses with status and confidence, evidence with queries and provenance, and links recording whether evidence supports or contradicts hypotheses.In the prototype, these concepts are represented in three SQLite tables, and the board preserves investigation history for collaboration and resumption.Source: §3.2 · p. 4
4 Initial Evaluation
The evaluation introduces a 50-case murder-mystery benchmark for evidence-backed investigations and compares D² with general-purpose SQL agents. D² is comparable in correctness but leads in completeness and verification ease; ablations show different roles affect these outcomes differently.
4 Initial Evaluation
- The benchmark contains 50 fictional murder cases of four difficulty levels, requiring exploration, competing hypotheses, and evidence from multiple tables.Each case has manually annotated ground-truth murderers and evidence, supporting evaluation of verdicts and investigation quality, completeness, and provenance.Source: §4 · p. 5
- The comparison uses Claude Sonnet 5 for D² and both baselines, with identical case descriptions, schemas, database access, and instructions.Claude Code and OpenClaw are general-purpose agents that iteratively use SQL tools; all collected evidence remains in system context.Source: §4 · p. 5
4.1 Exp. 1 & 2: Overall Quality & Breakdown
- D² scores 95.26% correctness, 76.02% completeness, and 68.88% verifiability ease, exceeding baselines especially on the latter two.OpenClaw and Claude Code score 94.64% and 93.37% correctness; their completeness is near 32%, and verifiability ease is 2.22% and 2.91%, respectively.Source: Figure 4 · p. 5, §4.1 · p. 5
- Across difficulty levels, correctness declines, but D² remains more complete and verifiable than baselines, including on very hard cases.On very hard cases, D²'s verifiability ease stays above 65%, while both baselines remain below 3%.Source: §4.1 · p. 5
4.2 Exp. 3: Ablation Study
- Ablations associate removing the judge with lower correctness and completeness, removing the reporter with the strongest verification-ease drop, and merging roles with fewer hypotheses.A single detective agent approaches the baselines in completeness, while retaining higher verification ease than those baselines.Source: §4.2 · p. 6
5 Research Roadmap
The roadmap frames data investigations as a new data management workload and identifies research questions for building general-purpose systems. It highlights optimization, provenance and auditing, benchmark scaling, and heterogeneous data and operators.
5 Research Roadmap
- Investigation optimization must choose both what to investigate and how to execute it, balancing accuracy, completeness, verifiability, cost, and resource use as well as runtime.Suggested directions include cost models, adaptive plans, and principled trade-offs among objectives.Source: §5 · p. 6
- The Data Investigation Board can support provenance linking conclusions to evidence and queries, while reverse investigation can expose missing, contradictory, or weak evidence.The paper frames this as extending classical “why” and “why-not” analysis to investigations.Source: §5 · p. 6
- Scaling the prototype requires larger and more complex benchmarks, more diverse questions, and attention to agent coordination as a systems problem.The proposed benchmark scope includes larger schemas and tables, longer evidence chains, and open-ended “why” and “how” questions.Source: §5 · p. 6
- A general investigation system should integrate heterogeneous data and specialized operators, raising questions about tool selection, operator composition, reuse, and provenance across engines.The prototype currently relies primarily on SQL and tables.Source: §5 · p. 6