Source-linked AI summary
DSGym: A Holistic Framework for Evaluating and Training Data Science Agents
Fan Nie, Junlin Wang, Harper Hua, Federico Bianchi, Yongchan Kwon, Zhenting Qi, Owen Queen, Shang Zhu, James Zou
TL;DR
Existing data science benchmarks provide fragmented coverage and can often be solved without using the actual data, limiting rigorous comparison of data science agents. DSGym introduces a standardized, extensible execution framework and curated task suites, then demonstrates execution-grounded training that enables a 4B model to reach competitive performance with GPT-4o on standardized analysis benchmarks.
Problem
Existing benchmarks differ across interfaces and task coverage, and substantial portions can be solved without actual data interaction.
Method
DSGym unifies tasks, agent interfaces, and execution environments while supporting curated evaluation suites and execution-verified trajectory synthesis.
Results
A 4B model trained with DSGym reaches competitive performance with frontier models such as GPT-4o on standardized analysis benchmarks.
Takeaways & Limitations
DSGym provides a standardized testbed for measuring and developing data science agents across general analysis, scientific workflows, and end-to-end modeling.
Takeaways & Limitations
DSGym currently excludes visualization-centric and other open-ended scientific workflows with multiple valid interpretations or stochastic outcomes.
Abstract
from arXiv · showhide
Data science agents promise to accelerate discovery and insight-generation by turning data into executable analyses and findings. Yet existing data science benchmarks fall short due to fragmented evaluation interfaces that make cross-benchmark comparison difficult, narrow task coverage and a lack of rigorous data grounding. In particular, we show that a substantial portion of tasks in current benchmarks can be solved without using the actual data. To address these limitations, we introduce DSGym, a standardized framework for evaluating and training data science agents in self-contained execution environments. Unlike static benchmarks, DSGym provides a modular architecture that makes it easy to add tasks, agent scaffolds, and tools, positioning it as a live, extensible testbed. We curate DSGym-Tasks, a holistic task suite that standardizes and refines existing benchmarks via quality and shortcut solvability filtering. We further expand coverage with (1) DSBio: expert-derived bioinformatics tasks grounded in literature and (2) DSPredict: challenging prediction tasks spanning domains such as computer vision, molecular prediction, and single-cell perturbation. Beyond evaluation, DSGym enables agent training via execution-verified data synthesis pipeline. As a case study, we build a 2,000-example training set and trained a 4B model in DSGym that outperforms GPT-4o on standardized analysis benchmarks. Overall, DSGym enables rigorous end-to-end measurement of whether agents can plan, implement, and validate data analyses in realistic scientific context.
DSGym
DSGym focuses on the data-driven investigation phase of scientific discovery and provides a unified environment spanning 10+ scientific domains and diverse file types.
- DSGym targets the Data-Driven Investigation phase, where agents connect scientific hypotheses with empirical evidence through complex analysis.
- The framework spans 10+ scientific domains and diverse file types within a unified environment.
- DSGym supports a closed-loop ecosystem for both evaluation and training.
1 Introduction
Existing data science agent benchmarks are fragmented, narrow, and vulnerable to data-independent shortcuts. DSGym addresses these gaps with a unified extensible framework, curated task ecosystem, and support for execution-grounded training.
- Data science workflows require coding, intricate analysis, interactive computation, and decisions grounded in data and validated by execution.
- Existing benchmarks differ in task formats, scoring conventions, and execution environments, hindering fair reproducible comparison.
- DSGym unifies diverse evaluation suites behind a single API and provides modular support for new tasks, agent scaffolds, tools, and evaluation scripts.
- DSGym-Tasks standardizes and audits benchmarks, filters frequently shortcut-solvable tasks, and adds DSBio and DSPredict coverage.
- Over 80% of annotated failures in scientific workflows arise from domain-grounding errors, while DSPredict-hard has near-0% medal rate despite over 60% valid submissions.
- DSGym supports execution-grounded trajectory synthesis for finetuning and releases a state-of-the-art small language model data science agent.
2 DSGym: A Unified Framework for Reproducible Data Science Agents
DSGym standardizes tasks, agent interfaces, and runtime environments for reproducible evaluation across data analysis, scientific workflows, and machine-learning modeling. Its isolated, stateful execution architecture supports scalable evaluation and agent training.
- DSGym provides a unified, reproducible framework for evaluating agents across heterogeneous tasks and domains.
- The framework standardizes task representations, agent interfaces, and runtime environments under one coherent protocol.
- Agents must interact programmatically with real data files in isolated environments with persistent state, resource limits, and filesystem separation.
- DSGym standardizes task prompts, answer formats, evaluation metrics, and environment definitions for cross-benchmark comparison.
- Its modular design supports new datasets, evaluation scripts, metrics, agent scaffolds, trajectory collection, and synthetic data generation.
- Tasks cover data prediction and data analysis, and both categories require code execution over real data files.
- The manager–worker system gives each trajectory an isolated worker with read-only datasets and a writable workspace, while preserving state across interaction steps.
- DSGym executes hundreds of trajectories in parallel while maintaining strict isolation, supporting scalable evaluation and training.
3 Limitations of Existing Data Science Benchmarks
Existing data science benchmarks suffer from weak data grounding, task invalidity and inconsistency, and limited operation and domain coverage. These weaknesses complicate reliable measurement of data-dependent agent ability.
- Current file-grounded benchmarks often allow questions to be answered correctly without reading the associated data.
- Withheld-data performance drops average 40.5% on QRData, 86.8% on DAEval, and 44.4% on DiscoveryBench.
- These results suggest that contamination, superficial pattern matching, or domain priors can inflate performance beyond genuine data interaction.
- Several benchmarks contain annotation errors, mismatched question–answer pairs, vague formatting instructions, or ambiguous multiple-choice options.
- Existing benchmarks overrepresent general statistics while rarely testing specialized terminology, raw scientific modalities, or domain-specific libraries.
4 DSGym-Tasks
DSGym-Tasks standardizes data science evaluation across general analysis, scientific analysis, and prediction workflows while enforcing reproducible, genuinely data-dependent tasks. Its expanded suites include expert-derived bioinformatics tasks and validated Kaggle competitions.
- DSGym-Tasks executes diverse general and domain-specific tasks through a unified, containerized task abstraction for fair, reproducible evaluation.
- Two-stage refinement combines manual quality verification with shortcut filtering based on frontier-model performance without data files.
- Scientific Analysis Tasks Construction: DSBio contains 90 bioinformatics tasks derived from peer-reviewed publications and open-source scientific datasets.
- Scientific Analysis Tasks Construction: DSBio tasks reproduce deterministic reported findings or require expert-derived follow-up analyses involving statistical modeling, dataset integration, and limited reliance on specialized packages.
- Scientific Analysis Tasks Construction: Expert review accepts tasks only when independent solutions match and the task has sufficient analytical depth, discarding or refining ambiguous, simple, or nondeterministic items.
- Modeling Tasks Construction: DSPredict includes 38 intentionally simple competitions in DSPredict-Easy and 54 validated high-complexity competitions in DSPredict-Hard.
5 Evaluation
DSGym evaluation spans general analysis, scientific workflows, and end-to-end modeling, revealing persistent weaknesses in data-dependent reasoning, scientific grounding, and robust execution. Frontier models perform better on easier tasks but struggle with complex scientific and predictive workflows.
- Evaluation covers general data analysis, specialized scientific discovery, and end-to-end modeling across closed-source, open-weight, and small models.
- All models show a performance drop on DABStep-Hard, indicating that multi-step reasoning with heavy data dependencies remains difficult even for frontier models.
- DSBio performance is consistently lower than general-task performance, with Kimi-K2-Instruct achieving the best overall result at 43.33%.
- On DSPredict-Hard, most models fail to exceed 70% Valid Submission Rate, while the Median Rate peaks at only 14.3%.
- Domain-grounding errors account for 85–96% of sampled DSBio failures, contrasting with statistical-knowledge and planning failures on general tasks.
- Agents also face environment restrictions, API incompatibilities, and simplicity bias that widens the gap between valid submissions and above-median performance.
- Enforcing data dependency reduces accuracy across models by up to ∼21% relative, showing that some benchmark tasks admit non-data-dependent solutions.
6 Demonstration: Training Data Science Agents via DSGym
DSGym supports execution-grounded synthesis of training data through validated query and trajectory generation. A 2,000-example synthetic corpus improves a 4B model’s analysis performance, interaction structure, and resistance to shortcut answering.
- DSGym provides infrastructure for supervised fine-tuning, curriculum learning, and reinforcement learning through standardized datasets, distributed environments, and trajectory recording.
- The synthesis pipeline generates executable queries, samples independent solution trajectories, and jointly validates each query–trajectory pair with an execution-aware judge.
- The judge evaluates clarity and feasibility, educational value, exploratory competence, execution robustness, task alignment, and answer plausibility.
- Trajectory sampling and verification can be applied to existing tasks to distill execution-verified reasoning traces for supervised fine-tuning.
- 2,000 high-quality pairs were curated from 3,700 synthetic query candidates to form the DSGym-SFT corpus.
- Fine-tuning on DSGym-SFT improves a 4B model across analysis benchmarks, including out-of-domain DSBio performance and especially DABStep-hard.
- DSGym-SFT increases exploratory depth, finer-grained decomposition, iterative execution, and robustness to shortcut-based answering.
7 Related Works
Prior data science benchmarks and agent studies cover useful but fragmented capabilities, often with limited interactivity or heterogeneous interfaces. DSGym instead emphasizes a unified gym environment for standardized execution across tasks and domains.
- Early benchmarks focused on introductory-to-intermediate code-generation and notebook analysis tasks using restricted libraries and relatively simple problems.
- Later work incorporated iterative reasoning, planning, statistics, domain knowledge, repeated execution, and debugging within interactive environments.
- DSGym focuses on a gym environment tailored to data science tasks, standardizing heterogeneous data and model interfaces.
- Agent-scaffold research explores linear traces, multi-agent workflows, environment modeling, coordination, and task representation.
- Effective agents explicitly conduct multi-step search over hypotheses, candidate solutions, and evaluation feedback.
8 Discussion and Limitations
DSGym supports future research on interactive agent training and deeper scientific grounding, while its deterministic evaluation leaves open challenges for open-ended scientific discovery.
- Extending to RL: DSGym’s distributed, containerized, stateful execution supports RL-style training and evaluation across multiple data science datasets.Open challenges include designing training signals, expanding data and task coverage, and assigning credit under sparse, long-horizon rewards.
- Deepening Scientific Grounding: DSBio reveals that generalist models struggle with domain-specific ontologies, data modalities, and tooling, especially smaller models.Suggested directions include tool-oriented abstractions and domain-adaptive learning.
- Deterministic evaluation and open-ended discovery: Extending evaluation beyond deterministic regimes remains challenging because real-world scientific workflows can involve stochastic outcomes, visualization, and multiple valid interpretations.DSGym currently does not cover visualization-centric or exploratory tasks.
- DSGym as a live testbed: DSGym is envisioned as a live, auditable testbed that evolves with scientific tooling and evaluation needs while supporting reproducible measurement and systematic ablations.This design complements static benchmarks that are prone to memorization and rapid saturation.
9 Conclusion
The paper introduces DSGym as a standardized, extensible environment for evaluating data science agents on real data while addressing shortcut solvability. It also releases curated task suites, identifies persistent weaknesses in frontier models, and extends the environment to training-data synthesis.
- Conclusion: DSGym evaluates whether agents can plan, write, and execute scientific workflows with real data files in stateful, isolated environments.It unifies heterogeneous benchmarks behind a single abstraction and provides tools to mitigate prompt-only shortcut solvability.
- Conclusion: DSGym-Tasks standardizes and audits existing analysis benchmarks, filters shortcut-solvable tasks, and adds DSBio and DSPredict for scientific analysis and end-to-end modeling.The task ecosystem expands evaluation across domain-grounded scientific workflows and realistic modeling challenges.
- Conclusion: A systematic study of frontier proprietary and open-weight LLMs reveals persistent weaknesses in domain-specific workflows, simplicity bias, and insufficient verification.These findings characterize recurring failure patterns across the evaluated agents.
- Conclusion: DSGym’s execution environment can also be used to synthesize training data beyond its evaluation role.The passage presents this as an additional use of the framework.
A.1 Examples of Refinement of Existing Benchmarks
The refinement examples show how DSGym removes invalid or underspecified benchmark items and selects reproducible, data-dependent prediction challenges under explicit competition criteria.
- Existing benchmark refinement: A QRData task is filtered because duplicated answer choices make its multiple-choice question ill-defined.Options A and B are identical, so the provided answer cannot distinguish the alternatives.
- Existing benchmark refinement: A DAEval task is filtered because its provided answer omits the required p_value field.The task explicitly requires both significance_of_difference and p_value, with p_value rounded to four decimal places.
- DSGym-bio: DSGym-bio contains 90 questions grounded in public biomedical datasets, spanning single-cell biology, genetics, and spatial transcriptomics.Single-cell biology accounts for 56/90 questions, genetics for 21/90, and spatial transcriptomics for 13/90.
- DSPredict-Hard: DSPredict-Hard competitions are selected using requirements for CSV submissions, valid machine-learning challenges, datasets under 15 GB, active leaderboards, and meaningful engineering.These criteria target reproducible quantitative benchmarking on feasible modeling tasks.
- DSPredict example: The web-traffic forecasting example evaluates submissions with SMAPE between forecasts and actual values.Participants predict future traffic for article-day combinations using a submission file with the required format.
B.1 Case Studies of Data Analysis Tasks
Case studies show that data-analysis agents fail through domain misinterpretation, incorrect task execution, computational breakdowns, and shortcuts that bypass the data. Prediction cases similarly reveal that simple heuristics can produce valid submissions without achieving strong leaderboard performance.
- Scientific analysis failures: Domain-grounding errors arose when agents substituted an unsupported tissue proxy or used subtype labels instead of the dataset’s primary cell-type annotations.One agent replaced missing blood cells with lymph-node cells, while another grouped cells using LHS_sub and LASP_sub rather than author_cell_type.
- Scientific analysis failures: A donor-specificity task failed after an OLS-per-gene computation timed out, leading the agent to return the first variable from the ranked list rather than calculate the score.The attempted model was also numerically problematic because the stated design used full one-hot encodings without an intercept, making X’X singular.
- General analysis failures: Instruction-following errors occurred when agents violated the required preprocessing order or excluded a numerical variable explicitly required by the task.The errors involved computing the age median before cabin deletion and omitting Happiness Rank from the correlation candidates.
- Prediction case studies: A computer-vision fallback predicted 11 identical cars for every test image, yielding a valid submission but only the 4.57 percentile.The submission received no medal and ranked below the median.
- Prediction case studies: A forecasting agent selected a 30-day median after validation, achieving SMAPE 0.404 versus 0.416 for a weekly seasonal baseline.The run generated 8,993,906 forecast rows and produced a valid submission, but ranked above the median without receiving a medal.
- Shortcut-solvable tasks: Some tasks were solved through memorized answers or domain priors without data interaction, including a treatment-group answer and classifying “smoke” as categorical.The shortcut examples explicitly describe answers produced without code execution or access to the supplied files.
C.2 DSPredict failure mode analysis
DSPredict failures were categorized into environment restrictions, API incompatibilities, low-effort heuristics, and benign or other outcomes. The analysis links task difficulty and solution complexity to distinct failure patterns across models.
- Failure taxonomy: Four primary failure categories were defined: Environment Access Restrictions, API Incompatibilities, Low-Effort Heuristics, and Benign / Other.Examples include timeouts or unauthorized-library attempts, deprecated API arguments, simplistic baselines, and successful runs or outliers.
- Model-specific patterns: In DSPredict-Hard, Claude Sonnet 4.5 had the highest frequency of Environment Access Restrictions (N=37), while showing near-zero API Incompatibilities.The passage attributes the pattern to sophisticated solutions that misjudge runtime constraints alongside stronger library knowledge.
- Difficulty effects: Qwen3 showed more Low-Effort Heuristics on the hard benchmark, indicating greater use of path-of-least-resistance baselines for complex modeling challenges.DSPredict-Easy instead had a flatter distribution with higher Benign completion rates.
- Evaluation setting: DSPredict evaluates end-to-end prediction tasks using training and testing data, with performance assessed by a task-specific target metric.The benchmark instructions require data exploration, preprocessing, model training, validation, and executable code within persistent execution contexts.