PaperlayerView exact revision on arXiv

Source-linked AI summary

DSAgentBench: Can Agents Automate End-to-End Data-Science Workflows in Real Computer Environments?

Mizanur Rahman, Mohammed Saidul Islam, Ridwan Mahbub, Md Tahmid Rahman Laskar, Shafiq Joty, Enamul Hoque Prince

arXiv:2608.10366v1cs.AIcs.CL

TL;DR

Existing benchmarks do not adequately evaluate whether agents can complete end-to-end data-science workflows through real computer interaction and coordinated tools. DSAgentBench addresses this gap with 275 realistic tasks and deterministic evaluation, finding that current agents achieve low success on these workflows.

  • Problem

    Existing benchmarks assess analytical code or general computer interaction separately, rather than autonomous end-to-end data-science workflows in real operating systems.

  • Method

    DSAgentBench evaluates 275 human-authored lifecycle-spanning tasks in real operating environments using coordinated tools and deterministic evaluators of final analytical outcomes.

  • Results

    Claude-4.6-Sonnet achieves 56.70% overall task success, while all open-source agents remain below 1%.

  • Takeaways & Limitations

    The benchmark identifies substantial limitations in grounding, tool orchestration, and iterative analytical reasoning for current agents.

  • Takeaways & Limitations

    Open-source agents do not support A11y Tree observations, and detailed error analysis covers manually inspected trajectories from only selected models.

Abstract

from arXiv · show

Real-world data science involves long-horizon workflows that span data wrangling, exploration, modeling, visualization, and validation, and require coordinated use of tools such as notebooks, IDEs, terminals, browsers, and databases within real operating environments. Yet existing benchmarks lack real-computer interaction and do not evaluate whether agents can execute complete end-to-end data-science workflows in realistic computing environments, failing to capture the multi-stage, multi-tool nature of data-science practice. We introduce DSAgentBench, the first benchmark to evaluate whether agents can automate full data-science workflows inside real computer environments. DSAgentBench contains 275 diverse tasks covering the entire data-science life-cycle, reflecting the complexity and tool coordination required in practice. Each task requires grounding decisions in intermediate outputs and coordinated tool use, and includes a deterministic evaluator that verifies analytical correctness, visual outputs, and model performance rather than code-only execution. Our extensive experiments with 15 closed- and open-source models show that even the strongest agent, Claude-4.6-Sonnet, achieves only 56.70% task success, while all open-source agents remain below 1%, frequently failing at tool orchestration, OS grounding, and multi-step reasoning. These results reveal a substantial capability gap between current agentic systems and real data-science workflows, positioning DSAgentBench as a foundation for developing grounded, verifiable, autonomous data-science agents. We release DSAgentBench at https://github.com/vis-nlp/DSAgentBench.

1 Introduction

Existing benchmarks and agent capabilities do not adequately test autonomous, end-to-end data-science work in real computing environments. DSAGENTBENCH addresses this gap with realistic workflows, multi-tool interaction, deterministic outcome evaluation, and broad agent experiments.

  • Real-world data science requires inspecting heterogeneous data, managing dependencies, executing and debugging scripts, visualizing results, and iteratively refining models.
  • Existing data-science benchmarks mainly test code correctness in isolation, while general computer benchmarks evaluate interaction without data-science reasoning.
  • DSAGENTBENCH evaluates autonomous data-science workflows inside real operating systems across the full data-science lifecycle.
  • 275 long-horizon tasks use coordinated tools and deterministic evaluators that verify analytical correctness rather than surface-level code execution.
  • 56.70% task success is achieved by Claude-4.6-Sonnet, while all open-source agents remain below 1%.The results expose limitations in grounding, tool orchestration, and long-horizon reasoning.

2 Related Work

Prior work addresses either analytical code capabilities or general computer control, but rarely combines realistic system interaction with end-to-end data-science reasoning. DSAgentBench is positioned at this intersection.

  • Early program-synthesis benchmarks evaluate functional code correctness without data-analysis tasks.
  • Data-science benchmarks cover subsets such as data manipulation, modeling, reasoning, or visualization rather than complete workflows.
  • LLM data-science agents are often evaluated through case studies or controlled settings, limiting assessment of robustness, long-horizon reasoning, and tool coordination.
  • GUI benchmarks support multi-step computer control across web, desktop, mobile, and office environments.
  • General computer benchmarks assess application use and navigation rather than loading data, analyzing results, training models, visualizing findings, and debugging workflows.

3 DSAGENTBENCH Benchmark

DSAGENTBENCH is a curated benchmark of 275 human-authored, reproducible tasks spanning data-science capabilities, modalities, sources, and difficulty levels. Human-defined deterministic evaluators verify final analytical outcomes.

  • 3.1 Problem Formulation: Each task combines an initial operating-system configuration, a natural-language analytical instruction, and a deterministic Python evaluator.
  • 3.1 Problem Formulation: The benchmark was constructed through dataset sourcing, collaborative task and evaluator design, and independent dual-annotator verification.
  • 3.2.2 Task and Evaluator Design: Tasks span data acquisition, exploratory analysis, feature engineering, modeling, evaluation, and deployment capabilities.
  • 3.2.2 Task and Evaluator Design: 275 tasks were developed by expert annotators, with human-defined task logic, expected outputs, and evaluators refined for wording and edge cases using vision-language models.
  • 3.3 Dataset Diversity & Statistics: Exploratory data analysis comprises 43.3% of tasks, while 47.6% are hard, 46.9% medium, and 5.5% easy.
  • 3.3 Dataset Diversity & Statistics: Tabular data accounts for 95.3% of tasks, supplemented by image and text data from platforms including GitHub, Kaggle, OpenML, and SQLite.

4 Methodology

The benchmark extends OSWorld into a realistic data-science environment where agents perceive desktop state, act through standard GUI controls, and receive updated observations in an iterative loop. Evaluation is based on final workflow outcomes without privileged task-specific APIs.

  • 4.1 Environment Architecture: DSAGENTBENCH extends OSWorld with an execution environment for end-to-end analytical workflows.
  • 4.1 Environment Architecture: The environment provides Ubuntu, Python libraries, VS Code, Jupyter Notebook, Chrome, and automated access to Kaggle, OpenML, direct URLs, and SQLite.
  • 4.1 Environment Architecture: Agents can receive either desktop screenshots or screenshots augmented with accessibility-tree metadata such as roles, names, bounding boxes, and interaction states.
  • 4.1 Environment Architecture: The action space includes mouse interactions, keyboard input, and WAIT, DONE, and FAIL meta-actions.
  • 4.1 Environment Architecture: Agents operate through a standard GUI action space rather than privileged task-specific APIs, while still supporting terminal, CLI, and script workflows.
  • 4.1 Environment Architecture: Each task follows initialization, perception, action, state transition, and updated-observation steps until completion.

5 Evaluation

DSAgentBench evaluates diverse agents on lifecycle-spanning data-science tasks using real tools, deterministic artifact-based scoring, and two observation settings. Results show low success on long-horizon workflows, with modest benefits from accessibility information and interaction budgets.

  • 5.1 Models: 15 agents spanning closed-source, hybrid, and open-source architectures are evaluated across Screenshot-only and Screenshot + A11y Tree settings.The benchmark reports lifecycle accuracy across Data Acquisition, Exploratory Data Analysis, Feature Engineering, Modeling, Visualization, and Evaluation.
  • 5.2 Evaluation: Deterministic Python evaluators collect scripts, files, visualizations, and trained models, then score them against task-specific criteria.Evaluators produce continuous scores in [0, 1], while task success requires an overall score of at least 0.95.
  • 5.3 Main Results: 56.70% overall accuracy makes Claude-4.6-Sonnet the strongest agent under Screenshot + A11y Tree, while GPT-5 reaches 29.81%.Human reference participants achieved 85.09% under the same environment and evaluation protocol.
  • 5.3 Main Results: At most 1% accuracy is achieved by open-source agents under Screenshot-only, and none support A11y Tree input.These agents often generate code and interface actions but fail to ground instructions in the UI state and complete end-to-end workflows.
  • 5.3 Main Results: Data acquisition, model validation, and evaluation are the most challenging task types, while adding accessibility information generally improves performance.The benefit varies widely across models, and overall success remains low even in the hybrid setting.
  • 5.4 Ablation Studies: Jupyter tasks outperform VS Code tasks, single-stage workflows outperform multi-stage workflows, and performance declines monotonically as task difficulty increases.The reported differences are associated with terminal and environment failures, state maintenance, error recovery, tool coordination, and iterative refinement.
  • 5.4 Ablation Studies: 24.54% →25.81% task success and 0.55 → 0.57 average score result when the interaction budget increases from 15 to 50 steps.Terminal-first execution also changes performance only marginally, from 19.34% to 20.73%.

6 Error Analysis

Error analysis shows that failures arise from both environment grounding and data-science execution, with timing and efficiency patterns differing substantially across agents. Open-source systems are dominated by grounding failures, while stronger closed-source systems exhibit more varied failure modes.

  • Root Causes of Failure: 97–98% of open-source-agent failures are grounding errors, revealing poor alignment with desktop state.Stronger closed-source agents show mixed grounding, terminal, code, and reasoning failures.
  • Temporal Failure Structure and Recovery: CUA and GUI-OWL-7B have over 93% of failures after prolonged interaction, whereas open-source models often fail while opening or controlling the terminal.GPT-4o and Gemini-2.5-Pro show higher proportions of early and midtrajectory failures.
  • Temporal Failure Structure and Recovery: CUA and open-source agents almost always exhaust their step budgets, while Gemini 2.5 Pro and GPT-5-Mini terminate earlier on many tasks.The analysis uses first failure step to characterize when errors occur during interaction.
  • Efficiency of Successful Runs: Gemini-2.5-Pro completes successful tasks in 6.76 steps on average, followed by GPT5-Mini at 7.33, GPT-4o at 10.03, and CUA at 15.00.Earlier termination is associated with faster successes but higher early-failure rates.

7 Conclusion

DSAgentBench is introduced to assess autonomous, end-to-end data-science workflows inside real operating systems rather than isolated code generation or generic GUI interaction. Its results expose substantial limitations in current agents and identify grounding, tool orchestration, and iterative analysis as key challenges.

  • 7 Conclusion: DSAgentBench is presented as the first benchmark for assessing end-to-end data-science automation inside real operating systems.It covers long-horizon, multi-tool workflows spanning data acquisition, analysis, and modeling.
  • 7 Conclusion: Unlike prior benchmarks focused on isolated code generation or generic GUI interaction, DSAgentBench requires planning and executing long-horizon workflows.The benchmark targets workflows across data acquisition, analysis, and modeling.
  • 7 Conclusion: Experiments reveal substantial limitations in current agents, with even the strongest systems achieving low success rates on these tasks.The analysis identifies grounding, tool orchestration, and iterative analysis as key challenges.
  • 7 Conclusion: The benchmark is positioned as a foundation for developing more robust and autonomous agents capable of performing real-world data science.This conclusion follows the benchmark’s systematic evaluation and analysis of current-agent limitations.

Limitations

The evaluation has three stated limitations: incomplete observation support for open-source agents, partial trajectory coverage in error analysis, and emphasis on final visualization artifacts.

  • Open-source agents were evaluated screenshot-only because they do not support A11y Tree observations.Closed-source agents were evaluated with both screenshot-only and screenshot+A11y Tree settings.
  • 604 closed-source and 150 open-source trajectories were manually inspected for detailed error analysis.These trajectories represent only a subset of the full benchmark, so rare failure patterns may not be fully captured.
  • Visualization-heavy tasks were evaluated primarily through final artifact quality.Checks cover required outputs, data mappings, labels, task-specific correctness, visual clarity, and semantic alignment.

Ethical Considerations

DSAgentBench applies licensing constraints to its datasets and excludes sources with ambiguous, restrictive, or missing license information.

  • All tabular and structured datasets were collected from openly licensed platforms, including GitHub, Kaggle, and OpenML.Included datasets had MIT or clearly permissive public-use licenses suitable for research, redistribution, and modification.
  • Datasets with ambiguous, restrictive, or missing license information were systematically excluded.This filtering supports safe sharing and extension of the benchmark by the research community.
  • AI-based writing assistants were used only to improve the paper’s presentation.

A.1 Task Categories

DSAgentBench organizes realistic data-science work into lifecycle categories and evaluates agents across coordinated desktop tools, environments, and interaction settings.

  • Task Categories: The benchmark groups tasks across data acquisition, exploratory analysis, feature engineering, modeling, evaluation, and deployment.Tasks cover heterogeneous data retrieval, statistical inspection, feature transformation, predictive modeling, validation, interpretation, visualization, and reporting.
  • Task Categories: Visualization tasks assess semantic correctness through labeled axes, titles, legends, and data mappings.The benchmark includes charts such as bar charts, scatter plots, heatmaps, pairplots, and model comparison graphs.
  • Task Categories: Tasks run in configured environments containing datasets, applications, libraries, and execution entry points for notebook- or editor-based analysis.The environment supports Jupyter Notebook, external data sources, SQLite databases, file systems, IDEs, visualization, execution, debugging, and web retrieval.
  • Task Categories: Agents interact through a unified desktop action space using screenshot-only or screenshot+A11y Tree observations.Actions are coordinate-based at 1920×1080 resolution and may return executable actions or WAIT, FAIL, or DONE control tokens.
  • Task Categories: The benchmark uses fixed experimental settings including PyAutoGUI actions, temperature 0.1, top-p 0.9, and a 1800-second task timeout.The setup includes separate provider-API and self-hosted vLLM configurations for closed- and open-source models.

B.6 Reproducibility and Environment Portability

DSAgentBench is designed for reproducible execution and portability through task-level configuration, fixed run parameters, and deterministic evaluation across supported desktop environments.

  • Reproducibility and Environment Portability: DSAgentBench extends OSWorld and is designed to support reproducible execution across Ubuntu/Linux, Windows, and macOS.The reported experiments use an Ubuntu-based environment, while the benchmark is built for broader desktop extensibility.
  • Reproducibility and Environment Portability: Each task specifies datasets, file-system state, applications, libraries, setup procedures, and deterministic evaluation metrics.Public datasets are pre-downloaded and released with the benchmark to avoid changing external-data availability.
  • Reproducibility and Environment Portability: Visualization evaluation combines deterministic checks with grounded semantic validation after required outputs, mappings, labels, and legends pass validation gates.A constrained LLM-based visual judge is used only for a small subset requiring qualitative assessment.
  • Reproducibility and Environment Portability: Core execution parameters are fixed and logged for every run, including observation type, action space, resolution, delays, timeouts, step budgets, and decoding settings.The setup uses PyAutoGUI, a 2.0-second postaction delay, 1800-second timeout, temperature 0.1, and top-p 0.9.
  • Reproducibility and Environment Portability: Numerical evaluators verify script execution and numerical accuracy within predefined tolerances without relying on language models.This makes numerical scores stable and repeatable across runs.

D.1 Error Analysis Examples

The error analysis identifies recurring failures in interactive development environments, especially around system notifications and code formatting. The examples and prompts illustrate how these failures disrupt execution and task completion.

  • Models repeatedly struggle with environmental notifications and system-level UI elements, disrupting task progression across error cases.Generic code-editor pop-ups are frequently misinterpreted or left undismissed, leading to incorrect or incomplete actions.
  • Models often generate the literal string \n instead of an actual line break, producing invalid or incorrect code.The mismatch between formatting intentions and execution semantics propagates downstream errors and task failure.
  • These recurring failures expose limitations in robustness to interactive development environments and in translating formatting intentions into executable code.
  • The illustrated agents use different prompting strategies, including direct execution, chain-of-thought triggers, thought requirements, tool schemas, and structured planning.The figures contrast standard and reasoning-oriented prompts for code-generation, UITARS, and JEDI agents.
Loading 2608.10366v1…