PaperlayerView exact revision on arXiv

Source-linked AI summary

DiscoveryBench: Towards Data-Driven Discovery with Large Language Models

Bodhisattwa Prasad Majumder, Harshit Surana, Dhruv Agarwal, Bhavana Dalvi Mishra, Abhijeetsingh Meena, Aryan Prakhar, Tirth Vora, Tushar Khot, Ashish Sabharwal, Peter Clark

arXiv:2407.01725v1cs.CLcs.AIcs.LG

TL;DR

DiscoveryBench asks whether LLMs can automate hypothesis search and verification from datasets, a capability whose current extent remains unclear. It formalizes and benchmarks multi-step discovery using real and synthetic tasks, finding that the best system reaches only 25%. The benchmark exposes the difficulty of autonomous data-driven discovery and provides a resource for further progress.

  • Problem

    The paper asks how well current state-of-the-art LLMs can automate data-driven discovery, whose search and verification capabilities remain unclear.

    Source: §1 · p. 1

  • Method

    The authors create DiscoveryBench with 264 manually extracted real-world tasks, 903 synthetic tasks, a formal discovery framework, and facet-based evaluation.

    Source: §1 · p. 2, §4.3 · p. 7

  • Results

    The best evaluated discovery agent reaches only 25% performance, while agents generally struggle with complex statistical, data-preparation, and domain-specific workflows.

    Source: §1 · p. 2, §5.3 · p. 8

  • Takeaways & Limitations

    DiscoveryBench illustrates the challenges of autonomous data-driven discovery and provides a resource for progress toward reliable and reproducible scientific discovery.

    Source: §6 · p. 9

    The study focuses on the data-analysis part of an interleaved experimentation-and-analysis pipeline, rather than the full process.

    Source: §1 · p. 1

Abstract

from arXiv · show

Can the rapid advances in code generation, function calling, and data analysis using large language models (LLMs) help automate the search and verification of hypotheses purely from a set of provided datasets? To evaluate this question, we present DiscoveryBench, the first comprehensive benchmark that formalizes the multi-step process of data-driven discovery. The benchmark is designed to systematically assess current model capabilities in discovery tasks and provide a useful resource for improving them. Our benchmark contains 264 tasks collected across 6 diverse domains, such as sociology and engineering, by manually deriving discovery workflows from published papers to approximate the real-world challenges faced by researchers, where each task is defined by a dataset, its metadata, and a discovery goal in natural language. We additionally provide 903 synthetic tasks to conduct controlled evaluations across task complexity. Furthermore, our structured formalism of data-driven discovery enables a facet-based evaluation that provides useful insights into different failure modes. We evaluate several popular LLM-based reasoning frameworks using both open and closed LLMs as baselines on DiscoveryBench and find that even the best system scores only 25%. Our benchmark, thus, illustrates the challenges in autonomous data-driven discovery and serves as a valuable resource for the community to make progress.

1 Introduction

DiscoveryBench asks how well current LLMs can automate data-driven discovery, addressing the difficulty of evaluating this process across diverse real-world domains. It formalizes discovery tasks and finds that the best evaluated system reaches only 25%.

  • DiscoveryBench formalizes data-driven discovery as searching for relationships among variables whose relevant facets may not match dataset language, enabling systematic evaluation.Source: §1 · p. 2, Figure 1 · p. 2
  • Each task requires statistical analysis and scientific semantic reasoning, including selecting domain-appropriate analyses and mapping goal terms to dataset columns.Source: §1 · p. 2, Figure 1 · p. 2
  • DiscoveryBench targets the broader multi-step discovery pipeline rather than statistical analysis alone, including hypothesis search and verification.Source: §1 · p. 2
  • The benchmark contains 264 manually extracted tasks from over 20 published papers and 903 synthetic tasks spanning 48 domains for controlled difficulty evaluation.Source: §1 · p. 2
  • 25% is the peak performance reported across evaluated LLM-based discovery agents, underscoring the challenge of autonomous data-driven discovery.Source: §1 · p. 2
  • The benchmark is intended as a resource for advancing autonomous, data-driven discovery.Source: §1 · p. 2

2 Related Work

DiscoveryBench differs from AutoML and statistical-analysis datasets by evaluating open-ended scientific discovery, including ideation, semantic reasoning, and pipeline design. Its tasks are broader and less constrained than prior statistical or causal-analysis benchmarks.

  • Earlier data-driven discovery systems often rely on task-specific datasets and pre-built pipelines, whereas DiscoveryBench tests whether systems can design appropriate pipelines across multiple tasks.Source: §2 · p. 2
  • Unlike AutoML datasets, DiscoveryBench evaluates open-ended discovery tasks rather than primarily training models for workflow optimization.Source: §2 · pp. 2–3
  • Unlike statistical-analysis datasets and tools, DiscoveryBench covers ideation, semantic reasoning, and pipeline design in addition to statistical analysis.Source: §2 · p. 3
  • Compared with QRData, DiscoveryBench uses open-ended questions, permits unrestricted statistical techniques, and draws complex tasks from published research rather than mainly textbook questions with numeric answers.Source: §2 · p. 3

3 Formalization

The paper formalizes data-driven hypotheses, their compositional semantic structure, and datasets with varying observability. A hypothesis semantic tree represents hierarchical relationships among target, intermediate, and independent variables.

  • A data-driven hypothesis is a declarative statement whose truth can be inferred from a dataset through a verification procedure such as statistical modeling.Source: §3 · p. 3
  • Hypotheses can be composed from sub-hypotheses with logical connectives, with verification evaluated through the same propositional structure.Source: §3 · p. 3
  • The formalism represents each hypothesis through context, variables, and relationships, corresponding to scope, interacting concepts, and their specified interaction.Source: §3 · p. 3
  • A hypothesis semantic tree is a Markov tree whose root is the target variable, leaves are non-derived independent variables, and internal nodes are intermediate targets.Source: §3 · p. 3
  • Subtrees encode intermediate hypotheses, while selecting different target and descendant levels allows one tree to represent multiple hypotheses.Source: §3 · pp. 3–4
  • The galaxy example shows how a tree can encode a primary visibility hypothesis, an intermediate color hypothesis, and a derived relationship using descendant variables.Source: §3 · p. 4
  • Different datasets may expose different subsets of internal nodes, changing the difficulty of discovering hypotheses represented by the semantic forest.Source: §3 · p. 4

4 DISCOVERYBENCH

DISCOVERYBENCH formalizes data-driven hypothesis discovery as benchmarked, multi-step workflows spanning real-world and synthetic tasks, with structured evaluation of predicted hypotheses. Its components target realistic scientific reasoning, controlled difficulty variation, and reproducible measurement of alignment.

4 DISCOVERYBENCH

  • DISCOVERYBENCH defines discovery tasks as deriving a hypothesis from datasets and a natural-language goal, optionally accompanied by a derivation workflow.The benchmark includes DB-REAL from published scientific work and DB-SYNTH from synthetically generated datasets.Source: §4 · p. 4

4.1 DB-REAL: Collecting data-driven hypotheses in the wild

  • DB-REAL collects and replicates data-driven hypotheses and workflows from published papers using data-first and code-first approaches across six scientific domains.The data-first process can require up to 90 person-hours per dataset, while code-first collection was constrained by missing, untranslated, or proprietary resources.Source: §4.1 · pp. 4–5
  • DB-REAL also includes hypotheses supported by replication workflows but not directly reported in the source papers, combining reported findings with novel dataset-grounded tasks.Each included benchmark item contains a dataset, hypothesis, and implementation workflow after replication or full procedural implementation.Source: §4.1 · p. 5

4.1.1 Features of DB-REAL benchmark

  • DB-REAL covers extensive preprocessing and statistical diversity, including over 500 data-preparation activities and 114 multi-dataset tasks with up to six datasets.Workflows are represented as chained unit actions, and their complexity is measured by workflow length.Source: §4.1.1 · p. 6, Table 1 · p. 6

4.2 DB-SYNTH: Generating data-driven hypotheses using LLMs

  • DB-SYNTH reverses the discovery process to generate semantic forests, noisy datasets, and tasks with controlled workflow difficulty across 903 tasks and 48 domains.Tasks are split 60/20/20 across train, development, and test sets and tagged with difficulty levels from 1–4.Source: §4.2 · pp. 6–7

4.3 Evaluation

  • Evaluation uses GPT-4-based decomposition of predicted and gold hypotheses into contexts, variables, and relationships, then computes a hypothesis match score.Relationship scoring assigns 100 for an exact match, 50 when a broader prediction encompasses the gold relationship, and 0 otherwise.Source: §4.3 · p. 7

5 Experiments

DiscoveryBench evaluates multi-step discovery agents and finds low overall performance, with success shaped by context accuracy, workflow complexity, domain knowledge, goal type, and workflow length.

5.2 Main Results

  • Overall performance is low across agent–LLM pairs on both DB-REAL and DB-SYNTH, while Reflexion with oracle feedback substantially improves over CodeGen.Performance on the real and synthetic benchmarks is similar, suggesting the synthetic tasks capture real-workflow complexity.Source: §5.2 · p. 8, Figure 4 · p. 8

5.3 Analysis

  • Accurate context prediction improves the likelihood of correctly predicting variables and relationships, but does not guarantee success.The relationship is shown using ctxF1 and the combined varF1 × relacc measure.Source: §5.3 · p. 8, Figure 4 · p. 8
  • The best non-oracle model performs best on correlation analysis (55%), data selection (18%), and summary statistics (18%), but reaches 0% on spatial analysis, pollen dating, and ecological modeling.These results indicate a barrier for workflows requiring complex statistical techniques, data preparation, or domain-specific models.Source: §5.3 · p. 8
  • For archaeology-related tasks that needed significant domain knowledge during data collection, adding targeted domain knowledge as hints changes DataVoyager’s GPT-4p performance from 17.5% without it to 9.9% with it.Source: §5.3 · p. 8
  • Performance varies by domain and goal type, with biology at 0% and engineering at 7% versus economics at 25% and sociology at 23%, while discovering relationships from fixed context and variables is easier than finding context or variables.The broader search required for context or variable discovery helps explain the goal-type difference.Source: §5.3 · pp. 8–9
  • Performance decreases as workflow length and complexity increase, with substantial drops already appearing for medium-length workflows.DB-REAL complexity is measured by gold workflow length, while DB-SYNTH uses semantic-tree height.Source: §5.3 · p. 9, Figure 5 · p. 9

6 Conclusion

The conclusion presents DISCOVERYBENCH as a benchmark of real scientific discovery workflows supplemented by synthetic tasks. The accompanying FAQ states that the benchmark and baseline code are publicly released with reproducibility materials.

6 Conclusion

  • DISCOVERYBENCH contains 264 tasks drawn from published scientific workflows and 903 structurally generated synthetic tasks spanning different difficulty levels.The benchmark is intended to evaluate discovery agents and expose the difficulty of autonomous data-driven discovery.Source: §6 · p. 9

A FAQs

  • The benchmark, discovery-agent code, evaluation code, documentation, and standard-format data are publicly released through the project repository and Hugging Face.The FAQ reports ODC-BY licensing for the benchmark and Apache 2.0 licensing for the code.Source: A FAQs · p. 12

B Limitations

The benchmark excludes several demanding scientific settings, including forecasting, massive datasets, and multimodal workflows. It also raises ethical concerns about misuse and false discovery.

  • Forecasting, simulation, and highly specialized modeling tasks were excluded because they were too time-consuming to replicate and discover.This filtering disproportionately removed papers from the natural and physical sciences compared with social sciences.Source: B Limitations · p. 13
  • Massive datasets such as the 8.92-petabyte Cancer Genome Atlas are outside the benchmark’s current scope.The methods also do not address whether discoveries from such datasets are robust or vulnerable to p-hacking.Source: B Limitations · p. 13
  • Multimodal data and complex pipelines, including satellite and geospatial workflows, are not currently handled.Consequently, workflows such as EarthRanger are not incorporated.Source: B Limitations · p. 13
  • Systems tuned on the benchmark could create societal risks including policy misuse, legal ramifications, and false discovery.Source: B Limitations · p. 13

C Data collection for DB-REAL

DB-REAL was assembled by replicating workflows from published studies and documenting the associated repositories and licensing. The process was labor-intensive, especially for specialized datasets.

D Data Generation for DB-SYNTH

DB-SYNTH generates structured datasets from sampled node values and pandas expressions, then introduces realistic execution errors and missing data. Self-refinement helps recover from expression failures.

  • Categorical leaves are sampled with replacement, while numeric leaves use selected distributions and parameters based on specified ranges.Source: D Data Generation for DB-SYNTH · p. 13
  • New columns for each subsequent hierarchy level are generated by executing pandas expressions.The expressions are constructed bottom-up from variables already generated.Source: D Data Generation for DB-SYNTH · p. 13
  • A self-refine procedure regenerates pandas expressions after execution errors, using the error logs as guidance.Source: D Data Generation for DB-SYNTH · p. 13
  • The generator probabilistically adds noise or drops values to create missing data, producing a dataset column for every node in the hierarchy.Each value is noised independently, leaving every row with sufficient true data for discovery.Source: D Data Generation for DB-SYNTH · pp. 13–14

E Datasheets

The datasheets document the benchmark’s purpose, collection, use constraints, distribution, metadata, directory organization, and discovery-agent interface. They describe a reproducible benchmark spanning real and synthetic tasks.

E.1 Motivation

  • DISCOVERYBENCH assesses whether LLMs can automate hypothesis search and verification from provided datasets.Source: §E.1 · p. 14

E.2 Collection Process

  • The authors collected tasks across sociology, biology, humanities, economics, engineering, and meta-science using data-first or code-first procedures.Each instance was manually implemented and verified for solvability.Source: §E.2 · p. 14

E.3 Uses

  • The benchmark is intended to evaluate LLM-based search and verification of hypotheses from datasets, not to train models that aggravate p-hacking.Source: §E.3 · p. 14

E.4 Distribution and Maintainance

  • The benchmark is distributed through the DiscoveryBench GitHub repository and the AllenAI Hugging Face dataset repository.Source: §E.4 · p. 14

F.1 Metadata structure

  • Metadata also stores contextual information, workflow-related insights, and dataset descriptions including maximum hierarchy depth and column details.Source: §F.1 · pp. 14–15
  • Metadata records identifiers, column descriptions, hierarchy depth, hypotheses, workflows, queries, domains, workflow tags, and dataset information.Queries include fields such as true hypotheses, relevant columns, target columns, question types, and discovery goals.Source: §F.1 · pp. 14–15

F.2 Directory structure for DB-REAL

  • DB-REAL contains 14 training metadata files and 144 test metadata files, while full datasets such as nls_raw require substantially more preparation.Source: §F.2 · p. 15

F.3 Directory structure for DB-SYNTH

H Evaluation

The evaluation CLI compares generated and gold hypotheses and workflows using task metadata, with required inputs for the reference and prediction. It supports real or synthetic metadata and saves evaluation results to a specified path.

  • The evaluation command compares discovery-agent outputs against gold hypotheses and workflows using task metadata.Source: H Evaluation · p. 17
  • The required inputs include a gold hypothesis, predicted hypothesis, metadata path, and metadata type.Source: H Evaluation · p. 17
  • The metadata-type option distinguishes real metadata from synthetic metadata during evaluation.Source: H Evaluation · p. 17
  • Evaluation results can be written to a specified output path, while the help option lists available command options.Source: H Evaluation · p. 17

I Experiments

The evaluator decomposes hypotheses into context, variables, and relations, then compares corresponding dimensions between gold and predicted hypotheses. Its prompts enforce structured JSON outputs and constrain extraction to information present in the hypothesis and workflow.

J Evaluator Prompts

  • The decomposition prompt extracts self-sufficient sub-hypotheses covering separate contexts and coherent context-variable-relation dimensions.Source: J Evaluator Prompts · p. 18
  • The evaluator represents each sub-hypothesis through contexts, variables, and relations, defined as scope conditions, interacting concepts, and their interactions.Source: J Evaluator Prompts · p. 18
  • The prompts require structured JSON responses and prohibit adding information absent from the hypothesis and workflow.Source: J Evaluator Prompts · pp. 18–19
  • Context matching returns true only when the predicted and gold boundary conditions semantically match.Source: J Evaluator Prompts · p. 19
  • Variable alignment counts variables in each hypothesis and their fuzzy-matched intersection to accommodate paraphrases.Source: J Evaluator Prompts · p. 20
  • Relationship alignment classifies predictions as very similar, more general, or different using a specificity hierarchy.Source: J Evaluator Prompts · p. 21

Related papers

Loading 2407.01725v1…