Source-linked AI summary
Are LLMs Ready for Scientific Discovery? A Capability-Oriented Benchmark for AI Scientists
Chuhan Shi, Xiaoquan Ren, Sicheng Song, Haobo Li, Rui Sheng, Yushi Sun
TL;DR
Existing benchmarks can conflate successful execution with scientifically valid claims and obscure whether models fail on basic description or complex inference. SDABench evaluates six scientific-analysis capabilities across five domains and finds that models perform well descriptively but struggle with assumptions, latent processes, mechanisms, and later-stage reasoning.
Problem
Existing benchmarks may mistake successful execution for scientific validity and obscure which scientific-analysis capabilities underlie model failures.
Method
SDABench evaluates six scientific discovery capabilities across five domains using real and synthetic datasets in multiple-choice and open-ended formats.
Results
Models perform best on descriptive tasks but degrade on inferential, causal, exploratory, predictive, and mechanistic tasks requiring assumptions, latent structures, or relationship modeling.
Takeaways & Limitations
The five-stage taxonomy shows that improved models reduce scope and variable errors, while function, relationship, and conclusion errors remain the dominant bottleneck.
Takeaways & Limitations
Synthetic semantic graphs may not capture the full complexity of real experimental confounders and measurement artifacts.
Abstract
from arXiv · showhide
Existing benchmarks for scientific data analysis evaluate LLMs primarily on code execution or workflow completion, overlooking that scientific analysis serves to support distinct types of scientific claims: hypothesis exploration, statistical inference, mechanistic explanation, each with different assumptions and validity criteria. We introduce SDABench, a benchmark that reorganizes evaluation around six capabilities (descriptive, exploratory, inferential, predictive, causal, and mechanistic) across five domains (Biology, Chemistry, Environment, Geography, Physics). SDABench comprises 527 real-data instances (SDA-Real) and 6000 synthetic instances (SDA-Synth), each in both multiple-choice and open-ended formats, constructed through an automated pipeline. Evaluating 15 representative LLMs, we find that models handle descriptive analysis well but degrade sharply on tasks requiring assumption selection, latent-process modeling, or mechanistic reasoning. SDABench further provides a five-stage error analysis framework that locates where LLMs fail: more advanced models more reliably identify the relevant scope and variables, but still struggle to select appropriate analytical procedures, model variable relationships, and draw valid conclusions.
1 Introduction
SDABench reframes scientific data-analysis evaluation around the validity of scientific claims rather than computational execution alone. It covers six discovery capabilities across five domains and reveals persistent reasoning failures beyond task identification and variable selection.
- Motivation: Existing benchmarks primarily assess whether models generate and execute analysis code or pipelines, which can conflate successful computation with scientific validity.These evaluations score results against ground-truth answers or task-specific success criteria.
- Benchmark design: SDABench evaluates descriptive, exploratory, inferential, predictive, causal, and mechanistic capabilities across Biology, Chemistry, Environment, Geography, and Physics.Its two complementary components include SDA-Real and SDA-Synth.
- Evaluation findings: Explicit candidate options can degrade performance on certain task types by inducing superficial pattern matching, challenging reliance on multiple-choice evaluation.This finding raises questions about using MCQ-based evaluation as a widespread benchmark format.
- Evaluation findings: More advanced models more often identify task objectives and relevant variables but still struggle to select appropriate analyses, model variable relationships, and support valid conclusions.Aggregate workflow scores can obscure whether failures occur in basic description, analysis selection, or conclusion validity.
- Benchmark design: 527 real-data instances and 6,000 synthetic instances are provided in both multiple-choice and open-ended formats.The benchmark includes an automated construction pipeline using LLM-based template construction, programmatic ground-truth derivation, and option generation.
2 Related Work
Prior data-analysis benchmarks emphasize code execution, workflow completion, or domain-specific tasks, while data-analysis assistants increasingly perform complex planning and verification. These evaluation paradigms can obscure deeper scientific-reasoning failures, because successful workflow completion does not ensure valid assumptions, uncertainty handling, or mechanistic modeling.
- Benchmarks for Data Analysis: Existing benchmarks span execution-oriented code generation, multi-step data-science workflows, and data-grounded scientific reasoning.Examples include DS-1000 and DSCodeBench; DA-Code, DSBench, and DataSciBench; and QRData, KramaBench, BLADE, ScienceAgentBench, and LLM-SRBench.
- Benchmarks for Data Analysis: Most benchmarks prioritize execution success, pipeline completion, or domain-specific tasks, blurring basic data-manipulation failures with deeper scientific-reasoning failures.This organization makes it difficult to distinguish whether an error arises from manipulating data or from reasoning scientifically about it.
- LLMs for Data Analysis: Data-analysis assistants have progressed from basic exploratory tools to autonomous agents capable of complex planning and verification.The cited progression covers early exploratory assistants and later systems designed for autonomous planning and verification.
- LLMs for Data Analysis: Successful workflow completion does not guarantee that an agent can validate statistical assumptions, handle uncertainty, or model scientific mechanisms.The passage identifies these capabilities as limitations not resolved by completing a workflow.
3 Benchmark
SDABench evaluates LLM scientific data analysis across six capability-oriented task types and five scientific domains, using real and synthetic instances in multiple-choice and open-ended formats. Its automated construction and validation pipeline programmatically derives ground-truth answers and filters unreliable instances for controlled evaluation.
- Benchmark scope: SDABench covers five domains and six task types spanning descriptive, exploratory, inferential, predictive, causal, and mechanistic scientific analysis.The task types support characterization, hypothesis-oriented pattern discovery, statistical inference, prediction, causal effects, and theory-based explanations of observed outcomes.
- Benchmark scope: Each instance includes a dataset, metadata, question, and ground-truth answer, released as both multiple-choice and open-ended formats.Multiple-choice questions enable scalable automatic evaluation with distractors, whereas open-ended questions require models to produce answers directly.
- Dataset construction: The benchmark combines SDA-Real, generated from eligible public datasets, with SDA-Synth, whose shared synthetic datasets enable comparisons across all six task types without dataset-specific confounds.SDA-Real retains datasets supporting at least one task type, while SDA-Synth uses common datasets to reduce confounding between task difficulty and dataset artifacts.
- Automated construction: The construction pipeline instantiates typed templates on datasets and computes answers with deterministic, executable derivation procedures.For valid parameters p, each procedure returns g = fτ(D, p), allowing requested analyses, including multi-step reasoning, to be replayed automatically.
- Validation and splits: After validation, the benchmark contains 6,231 SDA-Synth instances and 527 SDA-Real instances, with 6,000 synthetic instances balanced across five domains and six task types.The synthetic evaluation set keeps 200 instances per domain-task pair and splits them into training and test sets while preserving paired formats.
4 Experiments
The experiments evaluate 15 representative LLMs in a zero-shot setting using structured JSON outputs. Only final answers are scored, while reasoning fields support error annotation.
- Base-model evaluations use a zero-shot setting.
- Models produce structured JSON with separate reasoning and answer fields for comparable evaluation across response formats.Responses are parsed automatically; only the answer field is scored against the reference, while the reasoning field supports error annotation.
- 15 representative LLMs spanning closed-source and open-source models are evaluated.
Model OEQ MCQ
On SDA-Synth, OEQ is the primary metric because MCQ has a 25% chance floor, and frontier models achieve the strongest OEQ results. Performance declines from descriptive toward tasks requiring assumption selection, latent-structure reasoning, and mechanistic modeling, with substantial variation across formats and domains.
- Overall performance: OEQ is the primary metric because four-option MCQ has a 25% chance floor, making the two formats not directly comparable.MCQ is treated as a complementary signal rather than a directly comparable accuracy measure.
- Overall performance: GPT-5.4 leads closed-source models at 58.33% OEQ accuracy, followed by Gemini 3.1 Pro at 56.88% and Claude Sonnet 4.6 at 55.79%.The error-type objective raises fine-tuned Llama-3.1-8B-Instruct’s OEQ accuracy to 55.33%.
- Format comparison: Kimi K2.5 reaches near-random MCQ accuracy on Predictive tasks at 25.75% despite solving matched instances under OEQ, while Llama 3.3-70B-Instruct remains near random on causal and exploratory MCQ tasks.The reported near-random baselines are 24.00% for causal and 23.50% for exploratory tasks.
- Performance across task types: Models perform best on Descriptive tasks, decline on Inferential and Causal tasks, and show an especially pronounced format gap on Mechanistic tasks.Descriptive tasks mainly summarize observed variables, whereas Inferential and Causal tasks require valid assumptions and latent-structure reasoning.
- Performance across scientific domains: Environment has the highest overall OEQ accuracy at 46.9%, whereas Biology has the lowest overall accuracy and a Predictive score of 22.5%.The passage attributes this variation to different reasoning burdens imposed by the underlying data structure.
- Error analysis: Function Errors peak in Predictive tasks at 37.7%, while Relationship Errors dominate Mechanistic tasks at 32.1% and Exploratory tasks at 29.9%.These patterns reflect inappropriate method selection or formula implementation in Predictive tasks and failures to model relational structure in Mechanistic and Exploratory tasks.
5 Discussion
SDA-Synth captures core reasoning demands but omits some real-world complexity, while SDA-Real reveals schema and noise effects. Future development should target persistent later-stage errors through improved assumptions, relational reasoning, and verification.
- SDA-Synth versus SDA-Real: Synthetic semantic graphs broadly preserve the task-type hierarchy of real analysis, but real datasets expose canonical-name and noisy-schema effects.Canonical variable names can inflate mechanistic accuracy, whereas heterogeneous schemas and noisy variables affect exploratory analysis.
- Limitations: SDA-Synth mechanisms may miss real experimental confounders and measurement artifacts, while MCQs can induce superficial pattern matching.The error taxonomy may also undercount cascading failures because it annotates only the first execution error.
- Implications for future LLM development: Future work should target assumption selection under uncertainty, relational reasoning over variable interactions, and iterative verification against task constraints.These directions address later-stage errors that persist even as grounding errors resolve with scale.
- Implications for future LLM development: SDABench’s pipeline-ordered error taxonomy can guide adaptive curricula that progressively target capability gaps.The proposed curricula would target gaps identified at each analysis stage.
6 Conclusion
SDABench evaluates LLMs’ scientific data analysis across six discovery capabilities and five domains, revealing strong descriptive performance but sharp degradation on tasks requiring assumption selection, latent-process modeling, or mechanistic reasoning. The results indicate that stronger models often apply scientific knowledge incorrectly rather than lacking it.
- Benchmark and findings: SDABench evaluates scientific data analysis across six discovery capabilities and five domains.The benchmark evaluates 15 representative LLMs.
- Benchmark and findings: Models handle descriptive analysis well but degrade sharply when tasks require assumption selection, latent-process modeling, or mechanistic reasoning.
- Benchmark and findings: Stronger models generally apply scientific knowledge incorrectly rather than lacking that knowledge.