Source-linked AI summary
Hunt Instead of Wait: Evaluating Deep Data Research on Large Language Models
Wei Liu, Peijie Yu, Michele Orini, Yali Du, Yulan He
TL;DR
Existing evaluations often test agents on predefined questions, leaving autonomous investigation insufficiently assessed. The paper introduces Deep Data Research and DDR-Bench for open-ended database exploration with checklist-based verification, finding that frontier models still struggle with long-horizon exploration and that intrinsic exploration strategies matter beyond scaffolding or parameter scaling.
Problem
Most agentic LLM evaluations measure execution against predefined goals, while autonomous decisions about what to investigate remain rarely assessed.
Method
The paper formalises Deep Data Research and introduces DDR-Bench, where agents explore structured databases without predefined questions and generate insights evaluated against verified checklists.
Results
Frontier models show persistent limitations in long-horizon exploration, including difficulty identifying information gaps, resolving uncertainty, and maintaining a stable exploration policy.
Takeaways & Limitations
Effective investigatory intelligence depends on intrinsic exploration strategies, implicit planning, and autonomous termination criteria rather than external scaffolding or parameter scaling alone.
Takeaways & Limitations
Checklist-based evaluation cannot exhaustively capture all valid or meaningful insights produced in open-ended tasks, creating unavoidable false-positive cases.
Abstract
from arXiv · showhide
The agency expected of Agentic Large Language Models goes beyond answering correctly, requiring autonomy to set goals and decide what to explore. We term this investigatory intelligence, distinguishing it from executional intelligence, which merely completes assigned tasks. Data Science provides a natural testbed, as real-world analysis starts from raw data rather than explicit queries, yet few benchmarks focus on it. To address this, we introduce Deep Data Research (DDR), an open-ended task where LLMs autonomously extract key insights from databases, and DDR-Bench, a large-scale, checklist-based benchmark that enables verifiable evaluation. Results show that while frontier models display emerging agency, long-horizon exploration remains challenging. Our analysis highlights that effective investigatory intelligence depends not only on agent scaffolding or merely scaling, but also on intrinsic strategies of agentic models.
1. Introduction
Existing agentic LLM evaluations largely test execution against predefined goals, leaving autonomous investigation underassessed. The paper formalises Deep Data Research and introduces DDR-Bench to evaluate open-ended exploration and reveal persistent limitations in frontier models.
- Most agentic LLM evaluations assume a specified objective, primarily measuring executional intelligence rather than autonomous decisions about what to investigate.
- Data science offers a natural testbed because human analysts inspect structured data, form hypotheses, and refine them without predefined questions.
- Deep Data Research requires agents to explore structured databases, generate and validate hypotheses, and decide when to terminate without predefined questions or objectives.
- DDR-Bench is a large-scale benchmark enabling open-ended, long-horizon exploration with objective and interpretable checklist-based verification of database-grounded claims.
- Current frontier models show persistent limitations, requiring effective agency to identify information gaps, resolve uncertainty, and maintain a stable exploration policy.
2. Method
DDR formalises database research as unconstrained iterative exploration that produces message-wise and trajectory-wise insights. DDR-Bench operationalises this setting across diverse real-world databases with minimal scaffolding and checklist-based evaluation.
- 2.1. Deep Data Research: DDR models iteratively query a database with tools such as SQL and Python, without a predefined question, objective, interaction limit, or fixed termination point.
- 2.1. Deep Data Research: Agents produce message-wise insights from individual ReAct rounds and trajectory-wise insights by reviewing the complete history before generating a final report.
- 2.2. Implementation Details of Agent: DDR-Bench uses a minimal start prompt, unrestricted exploration, and autonomous termination to evaluate tool use and long-horizon reasoning with limited external scaffolding.
- 2.3. Data Collection and Construction: The benchmark spans distinct real-world scenarios requiring multi-table reasoning, longitudinal time-series analysis, and cross-domain interpretation of behavioural and psychological data.
- 2.3. Data Collection and Construction: Checklist items derived from database-linked facts support objective verification of open-ended insights, while expert screening and manual or LLM-based checking validate their inferability.
3. Benchmark Results
DDR-Bench evaluates proprietary and open-source LLMs on checklist-verifiable insights and novel insight usefulness. Results show strong models lead on accuracy and novelty, while current performance remains far from saturation and checklist evaluation has bounded coverage.
- Overall Performance: Only Claude 4.5 Sonnet achieves average accuracy above 40%, while top open-source models approach proprietary-model performance.Claude surpasses GPT and Gemini on DDR-Bench, and GLM, Kimi, and Minimax approach proprietary models.
- Overall Performance: Accuracy is reported using sample-averaged and item-averaged measures over message-wise and trajectory-wise insights.These four measures combine averaging over task entities or checklist fact items with message-wise or trajectory-wise insights.
- Evaluation Design: DDR-Bench separates execution from post hoc checklist evaluation, reducing exposure to question-answer-format data leakage and training-time overfitting.Models analyse data without evaluation questions, while checklists verify factual support after report generation.
- Evaluation Limitations: Checklist evaluation can produce false positives because predefined checklists cannot exhaustively enumerate valid or meaningful open-ended insights.The authors analyse novel insights pairwise rather than manually annotating every output.
- Novelty Analysis: Across all three scenarios, rankings by novel insight usefulness closely align with rankings by checklist accuracy.Differences are small, especially among top-performing models, and strong models produce more useful novel insights regardless of quantity.
4. Investigatory Dynamics
Investigatory dynamics reveal that effective exploration combines sustained early search with focused late-stage verification, balanced database coverage, and model-dependent stopping behaviour. Stronger models maintain more stable exploration strategies, whereas weaker models often explore narrowly or unfocussedly.
- Scaling Analysis: Delaying commitment improves insight accuracy by supporting longer early exploration followed by a small number of highly valuable late-stage interactions.The most valuable late-stage tokens arise from deep, targeted queries rather than exhaustive search.
- Scaling Analysis: Interaction-scaling curves generally follow a sigmoid trajectory toward saturation, but models differ substantially in improvement timing and performance ceiling.Higher-ceiling models often delay rapid improvement, resembling a plan-then-act strategy.
- Scaling Analysis: Token-based scaling is initially flat and then sharply increasing, because token consumption is dominated by environmental feedback from database tool results.Top-tier models intensively explore and verify specific hypotheses with complex tool parameters and minimal feedback.
- Exploration Patterns: Advanced LLMs use a balanced exploration regime combining adequate database coverage with focused access across scenarios.Coverage measures breadth, while lower normalised exploration entropy indicates more concentrated and targeted access.
- Exploration Patterns: Balanced exploration patterns are consistently associated with higher checklist accuracy, while weaker models often show limited coverage and unfocused access.Claude and GPT display stable balanced behaviour, whereas GLM has higher trajectory variance and Qwen and Gemini explore narrower subsets with high entropy.
- Self-Termination: Qwen3 and Qwen3-Next increasingly emit termination tokens as turns accumulate, whereas Qwen2.5 remains more uncertain and fluctuating.Qwen3-Next maintains higher confidence with lower variance throughout the trajectory.
5. Module Analysis
The module analysis finds that scale and context length alone do not reliably strengthen agency, while training choices and interaction strategies materially shape exploration. Agent modules can alter interaction patterns, but effective agency depends on stable coordination between reasoning and open-ended exploration.
- Training Factors: Parameter scaling yields less than a 3% gain in final accuracy even after a tenfold parameter increase.Smaller models outperform larger counterparts in several settings, and Qwen2.5 scaling alone does not enhance proactive exploration.
- Training Factors: Longer context windows do not consistently improve overall performance or systematically increase interaction rounds.Together with parameter-scaling results, this indicates that neither factor decisively determines agentic capability in deep data research.
- Training Factors: Qwen3 and Qwen3-Next increase exploration rounds on MIMIC and 10-K despite fewer activated parameters, while GLOBEM shows a higher performance ceiling without more rounds.The results attribute these gains primarily to systematic emphasis on reasoning and agentic ability across pre-training and post-training.
- Agent Module Analysis: Agent modules mainly reshape interaction patterns rather than reliably enhancing insight discovery.The paper characterizes effective agency as stable, implicit coordination between reasoning and open-ended exploration.
- Agent Module Analysis: Reasoning depth and interaction frequency trade off, with neither extreme achieving optimal performance.More reasoning can enable comprehensive queries and lower total interactions, but shifting toward either reasoning or interaction can cause information loss.
- Agent Module Analysis: Memory can induce more aggressive tool use, earlier termination, and higher token consumption without consistently improving final accuracy.On 10-K and MIMIC, memory reduced interaction rounds while extending exploration within individual rounds.
6. Failure Modes
The failure analysis identifies insufficient exploration as the dominant error source, alongside interpretation, reasoning, context, and capability-dependent errors. These failures show that models often stop before covering the necessary breadth or depth of the database.
- Failure Analysis: 58% of errors stemmed from insufficient exploration across breadth and depth.Limited breadth omits necessary data sources, while limited depth fails to pursue relevant findings further; premature self-termination mainly causes the former.
- Failure Analysis: Models may stop at descriptive statistics, misinterpret statistics, or impose unsupported assumptions and causal claims.These errors span superficial analysis, insight misinterpretation, and over-reasoning.
- Failure Analysis: Less capable models commonly repeat debugging, struggle with missing data, and disrupt analytical coherence.More powerful models more often exhibit over-reasoning or misinterpret insights, whereas smaller models show more fundamental execution problems.
7. Hallucination Evaluation
Hallucination rates in DDR-Bench are generally low and usually do not affect benchmark scores, because scoring uses only conclusions derivable from the database. Healthcare-related unsupported inferences nevertheless remain a deployment risk.
- Impact and Verification: Unsupported real-world attributions and inferred medications do not change evaluation scores because external associations receive no credit.Medication inference from incomplete MIMIC outputs remains a potential healthcare risk despite having minimal benchmarking impact.
- Dataset Effects: Hallucinations are largely absent in GLOBEM, likely because its wearable and psychological-survey data originate from distinct domains.Overall hallucination rates remain very low and have minimal impact on DDR-Bench results.
8. Trustworthiness on the LLM-as-a-Checker
DDR-Bench evaluates insights against objective database ground truth rather than subjective rubrics, and repeated evaluations are stable. Remaining disagreements arise from strict numerical requirements and occasional incorrect final judgments despite correct intermediate reasoning.
- Evaluation Design: LLM-as-a-Checker matches generated insights against objective ground truth rather than using subjective rubric-based judgment.This design is intended to support objective and fair evaluation.
- Stability: Coefficients of variation remain below 5% across repeated evaluations for both insight types.The study samples about 10% of data across models and scenarios and repeats evaluation five times.
- Discrepancies: Disagreements with human judgment mainly reflect overly strict numerical requirements and occasional incorrect final judgments.The checker may penalize insights that omit explicit values even when those values were correctly retrieved, and it may judge correct intermediate reasoning incorrectly.
9. Related Work
Prior data-intelligence benchmarks largely evaluate predefined questions or subjective report quality, leaving open-ended, directly verifiable insight discovery insufficiently addressed. DDR-Bench instead starts from data alone and evaluates concrete, fact-checked discoveries.
- Existing data-science benchmarks typically assess agents on user-specified questions, emphasizing executional intelligence over autonomous investigation.
- Open-ended data-analysis systems have explored database workflows, but their tasks or evaluation often remain predefined, subjective, or converted into closed-form questions.
- DDR-Bench begins with data alone and measures whether agents can mine concrete, fact-checked insights rather than merely report data statistics.
- Proactive-agent research studies tool-using systems that infer and fulfill immediate user goals, while open-ended investigation seeks novel states or patterns.
10. Conclusion
DDR-Bench frames investigatory intelligence as proactive discovery rather than reactive execution. Its results indicate that effective discovery depends more on intrinsic exploration strategies than on external scaffolding or simple parameter scaling.
- DDR-Bench evaluates the transition from reactive execution to proactive discovery through a benchmark focused on investigatory intelligence.
- Effective agency relies on exploration strategies that balance broad search with targeted reasoning, alongside implicit planning and autonomous termination decisions.
- The conclusion argues that agentic LLMs should move beyond query-response alignment toward autonomy in defining goals and pursuing useful insights from large-scale data.
11. Impact Statement
The paper presents DDR-Bench as an autonomous-data-science benchmark built with privacy, traceability, and controlled agent scaffolding. Its analyses also expose termination, efficiency, and framework-design boundaries relevant to interpretation.
- Data governance and evaluation design: DDR-Bench uses deidentified medical data under credentialed access, processes no personally identifiable information, and follows relevant data-use and ethical requirements.
- Data governance and evaluation design: Every generated insight is linked to an analysis turn, executable SQL or Python query, and raw observation, making agent-derived conclusions verifiable.
- Data governance and evaluation design: The benchmark intentionally simplifies the agent to isolate model capabilities, while supporting future tool extensions through the Model Context Protocol.
- Observed boundaries: Ablations found that complex agent frameworks usually degraded performance relative to ReAct, with only minor planning benefits in some settings.
- Observed boundaries: For open-ended questions, the checker assesses whether collected insights support the ground truth rather than requiring exact surface-form answer matching.
- Observed boundaries: Some models failed to terminate and entered debugging loops, so those runs were forcibly stopped at 100 rounds and excluded from the plotted distributions.
- Observed boundaries: Stronger models generally explored more rounds, while knowledge-intensive 10-K and MIMIC databases induced more interaction than GLOBEM.
E. Valid Insight Ratio
The valid-insight analysis filters exploratory messages, distinguishes informative from descriptive tool calls, and compares insight volume with validity across scenarios and models. Related diagnostics examine execution time, hallucination, prompts, and checklist evaluation.
- Insight construction: Message-wise insight generation labels an interaction “NO INSIGHT” when it cannot support an insight or only invokes descriptive tools such as listing files or describing tables.
- Valid insight ratio: In Figure A8, bar height denotes total generated insights and the solid segment denotes valid insights; stronger models sustain high ratios while producing many insights.
- Efficiency and reliability: Longer tool execution does not necessarily produce more meaningful insights, making efficient and precise database interaction an important evaluation dimension.
- Efficiency and reliability: Hallucination rates were essentially uncorrelated with final accuracy across 10-K, MIMIC, and GLOBEM, with reported correlations of 0.125, 0.056, and -0.046.
- Prompting and evaluation: The benchmark uses a shared ReAct-style system prompt, while message-wise insights are generated independently from the agent trajectory at each turn.
- Prompting and evaluation: Checklist checking asks whether insights substantively support checklist facts, enabling evaluation of open-ended discoveries without direct answer normalization.
I. Trajectory Sample
Trajectory samples illustrate how agents begin exploring different structured-data environments and record reasoning, actions, observations, and insights during analysis.
- Trajectory representation: Trajectory records include reasoning, actions, database observations, message-wise insights, and trajectory-wise insights.These components are shown across the MIMIC, GLOBEM, and 10-K scenarios.
- MIMIC: In MIMIC, the agent begins by inspecting available database information before analyzing an anonymized patient.The initial tool call requests database metadata, including table counts and table information.
- GLOBEM: In GLOBEM, the agent starts by listing files to investigate how behavioral patterns change over time for an anonymized user.The environment exposes activity data files for exploration.
- 10-K: In 10-K, the agent begins exploring structured financial data for Phillips 66 through a database containing company and financial-fact tables.The trajectory identifies a 10-K database with four tables and filtered records.