PaperlayerView exact revision on arXiv

Source-linked AI summary

AutoSDT: Scaling Data-Driven Discovery Tasks Toward Open Co-Scientists

Yifei Li, Hanane Nour Moussa, Ziru Chen, Shijie Chen, Botao Yu, Mingyi Xue, Benjamin Burns, Tzu-Yao Chiu, Vishal Dey, Zitong Lu, Chen Wei, Qianheng Zhang, Tianyu Zhang, Song Gao, Xuhui Huang, Xia Ning, Nesreen K. Ahmed, Ali Payani, Huan Sun

arXiv:2506.08140v2cs.LGcs.CL

TL;DR

Data-driven discovery lacks large-scale, high-quality training and evaluation data, while its complete, real-world scientific coding tasks are difficult to mine and costly to annotate manually. AutoSDT automatically searches repositories, selects data-driven discovery programs, and adapts them into standalone code solutions with task instructions. AutoSDT-Coder-32B matches GPT-4o (2024-05-13) on ScienceAgentBench at 7.8% SR and raises DiscoveryBench hypothesis matching from 6.9 to 8.1, a 17.4% relative improvement.

  • Problem

    Data-driven discovery lacks large-scale, high-quality training and evaluation data, while its complete, real-world scientific coding tasks are difficult to mine and costly to annotate manually.

    Source: §1 · p. 2

  • Method

    AutoSDT automatically searches repositories, selects data-driven discovery programs, and adapts them into standalone code solutions with task instructions.

    Source: §1 · p. 2, §2 · p. 2, §2.3 · p. 4

  • Results

    AutoSDT-Coder-32B matches GPT-4o (2024-05-13) on ScienceAgentBench at 7.8% SR and raises DiscoveryBench hypothesis matching from 6.9 to 8.1, a 17.4% relative improvement.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Experts judged 93% of a 256-task sample scientifically authentic and representative of data-driven discovery workflows, and 92.2% of generated programs correct solutions.

    Source: §1 · p. 2

    AutoSDT-5K does not include evaluation scripts for each code solution, limiting its usability in settings such as reinforcement learning.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Despite long-standing efforts in accelerating scientific discovery with AI, building AI co-scientists remains challenging due to limited high-quality data for training and evaluation. To tackle this data scarcity issue, we present AutoSDT, an automatic pipeline that collects high-quality coding tasks in real-world data-driven discovery workflows. AutoSDT leverages the coding capabilities and parametric knowledge of LLMs to search for diverse sources, select ecologically valid tasks, and synthesize accurate task instructions and code solutions. Using our pipeline, we construct AutoSDT-5K, a dataset of 5,404 coding tasks for data-driven discovery that covers four scientific disciplines and 756 unique Python packages. To the best of our knowledge, AutoSDT-5K is the only automatically collected and the largest open dataset for data-driven scientific discovery. Expert feedback on a subset of 256 tasks shows the effectiveness of AutoSDT: 93% of the collected tasks are ecologically valid, and 92.2% of the synthesized programs are functionally correct. Trained on AutoSDT-5K, the Qwen2.5-Coder-Instruct LLM series, dubbed AutoSDT-Coder, show substantial improvement on two challenging data-driven discovery benchmarks, ScienceAgentBench and DiscoveryBench. Most notably, AutoSDT-Coder-32B reaches the same level of performance as GPT-4o on ScienceAgentBench with a success rate of 7.8%, doubling the performance of its base model. On DiscoveryBench, it lifts the hypothesis matching score to 8.1, bringing a 17.4% relative improvement and closing the gap between open-weight models and GPT-4o.

1 Introduction

AutoSDT addresses the shortage of high-quality training and evaluation data for data-driven scientific discovery by automatically collecting and adapting real-world coding tasks. Its dataset receives positive expert assessments, and models trained on it improve on two discovery benchmarks.

1 Introduction

  • AutoSDT-Coder-32B matches GPT-4o (2024-05-13) on ScienceAgentBench at 7.8% SR and improves DiscoveryBench hypothesis matching from 6.9 to 8.1, a 17.4% relative gain.The ScienceAgentBench result doubles the base model’s 3.9% SR.Source: Figure 1 · p. 1, §1 · p. 2
  • Data-driven discovery needs complete code operating on real scientific datasets, but such tasks are difficult to mine automatically and manual annotation takes trained graduate students 2.5–3 hours per task.The paper identifies a shortage of large-scale, high-quality data for training and evaluation as a bottleneck for building AI co-scientists.Source: §1 · p. 2
  • AutoSDT uses LLM-based search, task selection, and program adaptation to diversify sources, identify real-world tasks, and produce standalone code with instructions.The pipeline also uses multiple rounds of adaptation and validation to target functional equivalence to the original code.Source: §1 · p. 2
  • AutoSDT-5K contains 5,404 tasks across four scientific disciplines and 756 unique Python packages, and costs an average of 0.55 USD per task to create.The disciplines are Bioinformatics, Computational Chemistry, Geographical Information Science, and Psychology and Cognitive Neuroscience.Source: §1 · p. 2
  • Experts rated 93% of 256 sampled tasks scientifically authentic and 92.2% of their generated programs correct solutions.These assessments support the reported quality of AutoSDT-5K.Source: §1 · p. 2

2 AutoSDT

AutoSDT automatically collects discovery tasks by searching repositories, selecting relevant programs and their dependencies, then adapting programs into executable task-and-solution pairs.

2 AutoSDT

  • The pipeline has three stages: repository search, task and dependency selection, and adaptation into independently executable programs with task instructions.It is presented as an automatic alternative to labor-intensive manual task annotation.Source: §2 · p. 2, Figure 2 · p. 3

2.1 AutoSDT-Search

  • LLM-expanded discipline keywords broaden repository search; AutoSDT then checks repository relevance, removes duplicates, and excludes repositories used in evaluation benchmarks.The search uses GitHub and PapersWithCode APIs, with an LLM judging repository relevance from README files.Source: §2.1 · p. 3

2.2 AutoSDT-Select

  • AutoSDT-Select filters Python files by size and directory, uses an LLM to identify scientific workflow code with dataset inputs and scientific outputs, and extracts dependencies.Extracted dependencies include datasets, pretrained models, and auxiliary utility code.Source: §2.2 · p. 3

2.3 AutoSDT-Adapt

  • AutoSDT-Adapt revises code imports, input/output routines, and hardcoded paths, then executes and self-debugs it for up to three iterations before generating task instructions.Programs still producing execution errors after the loop are discarded; instructions specify goals, required inputs, and expected outputs.Source: §2.3 · p. 4

3 AutoSDT-5K

AutoSDT-5K contains 5,404 data-driven discovery tasks across four disciplines, and expert review supports the validity and correctness of many sampled tasks. The dataset includes tasks of varied difficulty, while some instructions lack sufficient method guidance.

3.1 Statistics

  • 5,404 coding tasks were synthesized from 2,993 research-related repositories, at an average API cost of $0.55 per task.The pipeline selected files from 1,325 repositories; total API cost was $2,955.Source: §3.1 · p. 4
  • AutoSDT-5K spans multi-step workflows and general-purpose and discipline-specific packages across four scientific disciplines.Examples include preprocessing and advanced analytics, plus packages for atomic simulations, neuroimaging, and geospatial data.Source: §3.1 · p. 4, Figure 3 · p. 5

3.2 Expert Evaluation

  • In a random sample of 256 tasks, experts judged 93% meaningful for day-to-day research, while 73.4% of instructions were clear and complete.Experts also rated 91.4% of instructions as correctly expressed in domain-specific scientific language.Source: §3.2 · p. 5
  • Some instructions lacked detailed guidance about solution methods; one geographic-science example required parameters such as buoyancy beyond the provided coordinates.The authors suggest that limited context for poorly documented code may contribute to this issue and discuss adding relevant repository or publication context.Source: §3.2 · p. 5
  • Experts judged 92.2% of adapted programs to be correct solutions, although adaptations preserved original functionality without alteration 84.4% of the time.Many non-equivalent cases involved locally implementing missing dependencies.Source: §3.2 · p. 5
  • More than 75% of sampled tasks were rated medium or hard, and hard tasks had more code lines and subtasks on average.Difficulty was estimated by expected solution-writing time.Source: §3.2 · p. 6
  • The authors describe AutoSDT-5K as the largest open dataset for data-driven scientific discovery and the only automatically collected one, to their knowledge.They note that it is large enough for training, whereas the compared datasets are used for evaluation only.Source: §3.2 · p. 6

4 Experiments

Experiments evaluate AutoSDT-Coder on ScienceAgentBench and DiscoveryBench, report improved benchmark performance, and examine cross-disciplinary transfer and scaling with training-set size.

4.1 Experimental Setup

  • ScienceAgentBench tests end-to-end Python task completion, while DiscoveryBench tests code-based data analysis and hypothesis generation; metrics are SR and VER for the former and HMS for the latter.The experiments sample three responses and report average scores with standard deviation.Source: §4.1 · p. 6

4.2 Main Results

  • Training improves Qwen2.5-Coder-32B by 3.9% SR and 7.6% VER on ScienceAgentBench, with gains increasing by model size and extending to DiscoveryBench hypothesis generation.The 7B model does not show SR improvements; the paper describes DiscoveryBench as evidence of generalization to open-ended hypothesis-generation questions.Source: §4.2 · p. 7
  • On ScienceAgentBench, AutoSDT-Coder-32B outperforms larger open-weight models and GPT-4o (2024-05-13), while rivaling Claude-3.5-Sonnet-v1 and GPT-4o (2024-11-20).Table 4 compares model performance on ScienceAgentBench.Source: §4.2 · p. 7, Table 4 · p. 7

4.3 Ablation Studies

  • Discipline-only training transfers to some other disciplines, but most disciplines do not benefit from cross-discipline data; Bioinformatics transfers to chemistry, while GIS training broadens general-tool coverage.Table 5 reports 14B ScienceAgentBench success rates by training and target discipline; the paper links Bioinformatics-to-chemistry transfer to shared scientific libraries and tools.Source: §4.3 · pp. 7–8, Table 5 · p. 7
  • The 14B model gains up to 2.5k training examples and then saturates, whereas the 32B model continues gaining success rate as data scales to 5k+.Figure 4 plots training-set-size impact on ScienceAgentBench for the 14B and 32B models, with shaded standard deviations across three runs.Source: §4.3 · p. 8, Figure 4 · p. 8

5 Related Work

Prior scientific coding datasets rely on human curation, while AutoSDT uses automatic collection to scale data-driven scientific discovery tasks. Scientific repositories require filtering and expert evaluation that generic software-engineering datasets may not need.

Scientific Coding Datasets

  • Existing scientific coding benchmarks rely on human curation, which is labor-intensive and leaves datasets with only a few hundred evaluation tasks; AutoSDT adopts automatic collection to scale them.The closest datasets in scope are ScienceAgentBench and DiscoveryBench, which assess data-analysis and code-solution abilities in defined scientific disciplines.Source: §5 · p. 8

Automatically Collected Coding Datasets

  • Unlike software-engineering data pipelines, AutoSDT filters research repositories and files for scientific coding relevance and uses domain experts to evaluate task relevance.Research-repository code also lacks unit tests, creating a verification challenge for generated tasks and solutions.Source: §5 · pp. 8–9

6 Conclusion

The paper introduces AutoSDT for automatic large-scale collection of data-driven scientific coding tasks, builds and expert-evaluates AutoSDT-5K, and reports performance gains for AutoSDT-Coder-32B on two benchmarks.

6 Conclusion

  • AutoSDT is a fully automatic pipeline for collecting data-driven scientific coding tasks at scale; AutoSDT-Coder-32B shows substantial performance gains on two recent challenging data-driven scientific discovery benchmarks.AutoSDT-5K task instances are evaluated with subject matter experts.Source: §6 · p. 9

Limitations

AutoSDT-5K lacks per-solution evaluation scripts, and the work leaves reasoning-model training and agent-framework experimentation for future research. Dataset scale, discipline coverage, and language coverage are also bounded by current resources and design choices.

Verification of task instances

  • AutoSDT does not provide evaluation scripts for each solution, limiting use in settings such as reinforcement learning because reliable outcome-based checks remain difficult to generate automatically.The authors caution that LLM-as-judge or heuristic scripts may fail to measure true solution correctness.Source: Limitations · p. 9

Reasoning Models and Agent Frameworks; Dataset Scale; Discipline and Programming Language Diversity

  • The work focuses on base models and four disciplines using Python; resource constraints limited collection to 2,993 repositories and 5.4k tasks, though the pipeline could extend to other disciplines and languages.The authors leave reasoning-model training and agent-framework experiments for future work, citing challenges in generating effective long chain-of-thought rationales at scale.Source: Limitations · p. 9

Ethical Considerations

The appendix documents repository licensing and details the AutoSDT pipeline’s search, selection, and adaptation stages. It also lists supplementary examples, evaluation, cost, training, and related-dataset materials.

Ethical Considerations

  • Repositories were selected with permissive licenses allowing academic use, with creators’ ownership and intellectual property respected.The paper directs readers to Appendix G for repository-license details.Source: Ethical Considerations · p. 9

Appendix

  • The appendix lists prompts, task examples, cost breakdown, expert evaluation, training details, and related-dataset information.It also includes repository-license details.Source: Appendix · p. 13

A.1 AutoSDT-Search

  • AutoSDT-Search expands discipline seed keywords with GPT-4o, then queries GitHub and PapersWithCode for research-oriented Python repositories with at least 10 stars.GitHub results are filtered by GPT-4o for relevant research code and paper links; PapersWithCode paper links come directly from its API.Source: §A.1 · p. 13

A.2 AutoSDT-Select

  • AutoSDT-Select filters cloned repository files by length and directory, then uses GPT-4o to identify data-driven scientific discovery code.Files over 1,000 lines and files in directories such as “utils,” “config,” and “tests” are removed first; GPT-4o also locates local dependencies.Source: §A.2 · p. 13

A.3 AutoSDT-Adapt

  • AutoSDT-Adapt uses Claude-3.7-Sonnet to minimally adapt programs for execution and GPT-4o to generate task instructions.The adaptation instruction is to preserve original functionality.Source: §A.3 · p. 13

B Example Tasks

The appendix provides dataset comparisons, a geographic-information task example, and a breakdown of API costs by pipeline stage.

B.1 Task Instructions

  • 5,404 instances across 4 subject domains are listed for AutoSDT-5K, marked as scientific, naturally occurring, and automatically collected in Table A.1.The table caption identifies columns for instance count, subject domains, scientific orientation, naturally occurring code, and automatic collection.Source: §B.1 · p. 14

B.2 Full Task Example

  • A full task example pairs a task instruction with its code solution in Geographic Information.The appendix also points to examples of instructions for each covered discipline.Source: §B.2 · p. 14

C Cost Breakdown

  • The cost breakdown reports API costs by AutoSDT stage, using GPT-4o for search, selection, and instruction generation and Claude-3.7-Sonnet for code adaptation.The supplied passage does not state the per-stage cost amounts.Source: C Cost Breakdown · p. 14

E Training Details

The appendix specifies fine-tuning and inference settings for AutoSDT-Coder and notes that DiscoveryBench results come from a reimplemented inference pipeline.

Supervised Fine-tuning

  • AutoSDT-Coder models use full-parameter fine-tuning with a 1e-5 learning rate, at most 1 epoch, and an 8192-token maximum context length.Training used four NVIDIA H100 96G GPUs for 7B/14B models and eight for 32B models.Source: E Training Details · p. 14

Inference

  • Inference uses temperature 0.2 and top_p 0.95, with maximum tokens of 2,000 for ScienceAgentBench and 1,024 for DiscoveryBench.The inference description specifies these settings for the two benchmarks.Source: E Training Details · p. 14

Re-implementation of Inference of DiscoveryBench

  • DiscoveryBench results use a reimplemented inference pipeline that decouples code generation from the original LangChain agent, so they differ slightly from the original paper’s results.The authors based their reproduction on the DiscoveryBench authors’ original codebase.Source: E Training Details · p. 14

F Related Datasets

The related-dataset comparison presents AutoSDT-5K as a large-scale dataset built from naturally occurring scientific code and collected automatically.

Dataset comparison

  • AutoSDT-5K covers considerably more tasks across multiple disciplines than the compared science-oriented data-analysis datasets.The comparison is presented in Table A.1.Source: F Related Datasets · p. 17
  • Unlike datasets drawn from competitions or curated by human annotators, AutoSDT-5K uses naturally occurring code authored by real-world scientists.The passage contrasts it with MLE-Bench, DSBench, RE-Bench, and BixBench.Source: F Related Datasets · p. 17
  • AutoSDT-5K is described as the only automatically generated dataset for scientific coding tasks and the first effort to collect high-quality discovery programs at scale.The authors distinguish this from automatic generation work on software-engineering datasets.Source: F Related Datasets · p. 17

G Repository Licenses

The repositories composing AutoSDT-5K were checked or treated under stated assumptions to support academic use. Custom licenses were manually reviewed, while repositories without license information were assumed permissive for academic purposes.

G Repository Licenses

Loading 2506.08140v2…