PaperlayerView exact revision on arXiv

Source-linked AI summary

DSAEval: Evaluating Data Science Agents on a Wide Range of Real-World Data Science Problems

Maojun Sun, Yifei Xie, Yue Wu, Ruijian Han, Binyan Jiang, Defeng Sun, Yancheng Yuan, Jian Huang

arXiv:2601.13591v2cs.AIcs.CL

TL;DR

Real-world data science is open-ended and difficult to evaluate with standardized answers, especially across diverse domains and modalities. DSAEval addresses this with a benchmark and interactive evaluation framework spanning multimodal perception, multi-query workflows, and reasoning, code, and results. Across 13 agents, Claude-Sonnet-4.5 performs best overall, multimodal perception improves vision-related tasks by 2.04%–11.30%, and unstructured domains remain challenging.

  • Problem

    Open-ended real-world data science problems lack standard answers, while existing benchmarks provide limited coverage of domains, workflow stages, and multimodal observations.

  • Method

    DSAEval combines a real-world benchmark, sandbox-based multimodal and multi-query interactions, and LLM-based multidimensional evaluation against soft ground truth.

  • Results

    Claude-Sonnet-4.5 achieves the strongest overall performance, while multimodal perception consistently improves vision-related tasks by 2.04%–11.30%.

  • Takeaways & Limitations

    Current agents handle structured data and routine workflows well but still struggle with unstructured domains and complex modeling tasks.

  • Takeaways & Limitations

    Current models still struggle in unstructured domains such as Computer Vision and NLP, motivating greater emphasis on execution-grounded workflow completion.

Abstract

from arXiv · show

Recent LLM-based data agents aim to automate data science tasks ranging from data analysis to deep learning. However, the open-ended nature of real-world data science problems, which often span multiple taxonomies and lack standard answers, poses a significant challenge for evaluation. To address this, we introduce DSAEval, a benchmark comprising 641 real-world data science problems grounded in 285 diverse datasets, covering both structured and unstructured data (e.g., image and text). DSAEval incorporates three distinctive features: (1) Multimodal Environment Perception, which enables agents to interpret observations from multiple modalities, including text and vision; (2) Multi-Query Interactions, which mirror the iterative and cumulative nature of real-world data science projects; and (3) Multi-Dimensional Evaluation, which provides a holistic assessment across reasoning, code, and results. We systematically evaluate 13 recent advanced agentic LLMs using DSAEval. Our results show that Claude-Sonnet-4.5 achieves the strongest overall performance, MiMo-V2-Pro and GPT-5.2 lead in duration and step efficiency, respectively, and MiMo-V2-Flash is the most cost-effective. We further demonstrate that multimodal perception consistently improves performance on vision-related tasks, with gains ranging from 2.04\% to 11.30\%. Overall, while current data science agents perform well on structured data and routine data analysis workflows, substantial challenges remain in unstructured domains. Finally, we offer critical insights and outline future research directions.

1 Introduction

DSAEval addresses the difficulty of evaluating open-ended, multifaceted real-world data science by combining broad coverage, multimodal perception, iterative interactions, and multidimensional assessment. Across 13 evaluated models, Claude-Sonnet-4.5 leads overall performance, while multimodal perception improves vision-related tasks and unstructured domains remain challenging.

  • Motivation: Open-ended real-world data science problems lack standardized solutions, making exact-match evaluation insufficient.Prior benchmarks also cover only partial domains or isolated workflow stages.
  • Contributions: DSAEval combines broad domain coverage, multimodal environment perception, multi-query interactions, and multidimensional evaluation across reasoning, code, and results.Its interactions reflect iterative workflows, while tailored judges assess multiple dimensions.
  • Findings: Claude-Sonnet-4.5 achieves the best overall performance among 13 evaluated LLMs and VLMs.MiMo-V2-Pro, GPT-5.2, and MiMo-V2-Flash lead token efficiency, step efficiency, and cost-effectiveness, respectively.
  • Findings: 2.04%–11.30% gains show consistent improvement from Multimodal Environment Perception over a single-text-observation baseline on vision-related tasks.The results also indicate strong structured-data performance but substantial challenges in unstructured domains and clustering.

2 Background and Related Works

Existing data-agent benchmarks largely emphasize tabular data, isolated workflow stages, and easily verifiable outputs. DSAEval broadens evaluation to heterogeneous real-world data, deep learning, multimodal observations, and report-style deliverables assessed across multiple dimensions.

  • Existing limitations: Most existing benchmarks focus on tabular data or isolated workflow stages and provide limited coverage of complex deep learning tasks.They also offer limited support for multimodal environment perception.
  • Existing limitations: Conventional protocols commonly evaluate code unit tests, metric comparisons, or exact matches against objective ground truth.These approaches are easy to verify but do not fully represent open-ended data-science outputs.
  • Evaluation gap: Real-world deliverables integrate executed code, quantitative results, visual evidence, methodological choices, and interpretive conclusions.This report-style structure makes them difficult to assess with conventional evaluation methods.
  • DSAEval: DSAEval addresses these gaps with heterogeneous datasets, complex deep learning workflows, multimodal observations, and LLM-based evaluation of reasoning, code, and final results.Its design is explicitly broader than benchmarks limited to partial domains or pipeline stages.

3 Methodology

DSAEval constructs realistic benchmark cases, lets agents solve interdependent tasks in a multimodal sandbox, and evaluates their notebooks and reports against soft ground truth across reasoning, code, and results.

  • Framework: DSAEval comprises a curated benchmark suite, a sandbox-based interactive environment, and an LLM-judge protocol for reasoning, code, and results.The framework is designed to evaluate agents in real-world scenarios.
  • Benchmark Suite Construction: The benchmark pipeline draws from over 2,000 open-source datasets and competitions plus 50 authoritative textbooks, then filters and annotates cases.Advanced LLMs synthesize questions, reasoning, and answers as reference soft ground truth.
  • Data Agent Framework: The sandbox executes agent-generated Python code and returns text, tabular, and image observations, including logs, dataframe previews, and plots.GPU acceleration and a Jupyter kernel support workflows requiring visual feedback.
  • Multi-Query Interactions: A session contains one dataset and a sequence of logically connected subtasks, with persistent sandbox state across interactions.The completed session yields a full executed-code notebook and a textual report.
  • Multi-Dimensional Evaluation: A specialized judge compares the notebook and report with soft ground truth using separate reasoning, code, and result dimensions.The reported configuration sets α = 0.3 and β = 0.3, placing slightly greater emphasis on final-findings accuracy.

4 Experiments

The experiments evaluate 13 LLM/VLM agents across performance, efficiency, cost, and multimodal perception. They compare model capabilities across domains and workflow stages, and test visual observations against single-text baselines.

  • Experimental Questions: The experiments address agent performance, efficiency and cost-effectiveness, and the impact of multimodal observations through three research questions.RQ1 covers domain and workflow variation; RQ2 covers operational trade-offs; RQ3 compares multimodal and text-only settings.
  • Experimental Setup: 13 recent LLMs and VLMs from proprietary and open-source families are evaluated across varying model sizes.The selection includes models such as GPT-5.2, Gemini-3-Pro, Claude-4.5-Sonnet, MiniMax-M2.7, DeepSeek-V3.2, and Qwen3-VL-30b.
  • Efficiency and Cost: Efficiency analysis measures task duration, agent steps, token consumption, and total cost in USD.Cost is computed using model pricing information from OpenRouter.
  • Multimodal Evaluation: Multimodal perception is tested by comparing three representative models with visual observations enabled against their single-text variants.The comparison spans four vision-related domains and tasks.

5 Results

DSAEval reveals a hierarchy in agentic data science capabilities: models perform best on structured tasks, while unstructured domains and complex workflow stages remain challenging. Performance, efficiency, and cost trade off substantially across models, and multimodal perception improves vision-related tasks.

  • Claude-Sonnet-4.5 achieves the highest overall score at 8.164, ahead of MiMo-V2-Pro at 7.912 and GPT-5.2 at 7.713.
  • Domain Analysis: Computer Vision and Natural Language Processing are the most challenging domains, with average scores of 6.31 and 6.25, respectively.Models perform substantially better on structured Data Analysis and Business Analytics tasks.
  • Task Analysis: Prediction & Forecasting and Model Training & Optimization have the lowest workflow-stage averages, at 6.03 and 6.49, respectively.These stages require iterative experimentation and deeper analytical intuition than routine data preparation.
  • Efficiency & Cost-Effectiveness: MiMo-V2-Pro offers the best balance between duration and performance, while GPT-5.2 uses relatively few agent steps and MiMo-V2-Flash costs approximately $0.007 per task.Claude-Sonnet-4.5 has the highest cost, approximately $1.08 per task, despite achieving the strongest overall performance.
  • Impact of Multimodal Perception: Multimodal perception improves all four evaluated tasks, with gains from 2.04% to 11.30%; Qwen3-VL-30b gains 11.30% on Computer Vision.On EDA tasks, models gain an average of 4.54% when visual outputs such as plots are available.

6 Future Directions

The paper identifies unstructured, execution-intensive workflows as a major frontier for future data science agents. It recommends stronger verification and highlights DSAEval’s experimental traces as resources for training, debugging, and evaluation.

  • Current agents still struggle in Computer Vision and NLP because these tasks require complex data loading, preprocessing, architecture design, and iterative debugging.
  • Unsupported APIs, incompatible configurations, data-loading errors, and result hallucinations account for many observed failures.Result hallucination occurs when final reports state metrics or conclusions unsupported by executed code.
  • Future systems should verify intermediate states and enforce consistency among execution traces, generated artifacts, and final conclusions.
  • DSAEval’s real experimental traces include multi-step interactions, execution outputs, failure cases, and final reports that can support training, debugging, and evaluation.These traces are positioned as resources for reinforcement learning and self-improvement settings.

7 Conclusion

DSAEval benchmarks autonomous data science agents on real-world workflows using multimodal perception, multi-query interactions, and multidimensional evaluation. Results show strong structured-data performance but persistent difficulty with unstructured and complex modeling tasks.

  • DSAEval evaluates autonomous data science agents across reasoning, code, and final results using multimodal environment perception and multi-query interactions.
  • Claude-Sonnet-4.5 leads overall performance, while MiMo-V2-Pro, GPT-5.2, and MiMo-V2-Flash lead duration efficiency, token/step efficiency, and cost-effectiveness, respectively.
  • Multimodal perception improves vision-related tasks by up to 11.30%.
  • Current agents handle structured data and routine workflows well but continue to struggle with unstructured domains and complex modeling tasks.

Limitations

DSAEval’s evaluation remains subject to run-to-run variability, evaluator subjectivity, and possible contamination from publicly available source materials.

  • Randomness and Human Evaluation: LLM and judge outputs may vary across repeated runs because of inherent model randomness.The authors use a fixed protocol, consistent scoring dimensions, and aggregated human consensus to reduce this effect.
  • Randomness and Human Evaluation: Human evaluation can be subjective because experts may interpret open-ended reasoning, code correctness, and report quality differently.The authors state that slight variations remain inevitable for open-ended data science workflows.
  • Data Leakage and Contamination: Public datasets and notebooks may expose evaluated models to related material during pre-training, creating data leakage and contamination risks.Dynamic execution and newly synthesized question–reasoning–answer formulations mitigate but cannot completely rule out implicit memorization.
  • Data Leakage and Contamination: Future mitigation includes pairing established task templates with newly collected or unseen datasets and continuously updating DSAEval.These measures are intended to better distinguish genuine problem-solving ability from memorized dataset-specific strategies.

Ethical Considerations

The study uses public, consented data and sandboxed code execution, while benchmark construction relies on expert validation and strict quality filtering.

  • Data Safety and Privacy: All agent-generated code was executed in an isolated sandbox to prevent damage to external programs, data, or infrastructure.The benchmark uses publicly available datasets and reports checks for personally identifiable information and privacy or copyright compliance.
  • Consent and Licensing: Human-evaluation participants volunteered, received study briefings, and provided informed consent.The authors also state that public-dataset use followed the respective licenses and terms of use.
  • Benchmark Validation: Two expert annotators reviewed synthesized question–reasoning–answer pairs for data accessibility, question clarity, and answer validity.The reference reasoning and answer function as soft ground truth and had to be logically sound, relevant, and reliable.
  • Benchmark Validation: 73.5% of 2,421 synthesized candidates were rejected, leaving 641 problems in the final benchmark.The retained subset was described as the top 26.5% high-quality portion after strict multi-stage filtering.
  • QRA Construction: The generation procedure requires questions, reasoning, and answers to be returned in a valid JSON structure.The QRA format includes data type, domain, task type, language, question, reasoning, answer, and confidence fields.

F Data Science Agent Final Report

The final report cleans and analyzes social-media text with a VADER-based sentiment pipeline, then summarizes sample outputs and the resulting sentiment distribution.

  • Sentiment Analysis: VADER compound scores are computed from the original text and mapped to positive, neutral, or negative sentiment labels.The thresholds are positive ≥0.05, neutral from −0.05 to 0.05, and negative ≤−0.05.
  • Text Cleaning: The task applies lowercasing, punctuation, URL, non-ASCII, stopword, and stemming transformations to create Clean_Text.The reported pipeline uses NLTK and processes 732 posts.
  • Reported Outputs: The report presents the first five cleaned-text samples with their VADER scores and labels.One example assigns “enjoy beauti day park” a VADER score of 0.8221 and a Positive label.
  • Reported Outputs: Positive sentiment is the dominant category with 466 posts (63.66%), followed by negative sentiment with 184 posts (25.14%) and neutral sentiment with 82 posts (11.20%).The distribution is shown in the reported VADER sentiment summary and visualization dashboard.
  • Model Evaluation: Overall agreement with standardized labels is 42.62%.The report attributes the discrepancy to original labels containing nuances that VADER consolidated into broader categories.

G Evaluation Process

The evaluation examines agent performance across reasoning, code execution, and final results, using both aggregate component scores and qualitative inspection of completed solutions. Results illustrate strong successes alongside cases where execution succeeded but reported outcomes were inconsistent or incorrect.

  • Model evaluations: Claude-Sonnet-4.5 receives reasoning, code-step, and final-result scores of 8, 8, and 7 on a sentiment-analysis task.The solution implements text cleaning, VADER sentiment scoring, and visualization, but its sentiment distribution differs substantially from the standard answer.
  • Evaluation discrepancies: Execution success does not guarantee reliable evaluation outcomes, because solutions can complete code execution while producing inconsistent sentiment results or incorrect forecasting interpretations.The examples combine successful final-cell execution with discrepancies between generated code, reasoning, and standard answers.
  • Model evaluations: GPT-5.2 receives scores of 9 for reasoning, code steps, and final results on a computer-vision classification data-loading task.The run is marked consistent, indicating agreement between the evaluated solution components for this task.
  • Successful solutions: A Boot/Sandal/Shoe image-loading solution matches the standard answer while creating normalized 128×128 iterators, an 80/10/10 split, and 12,000/1,500/1,500 train, validation, and test images.The solution also builds PyTorch DataLoaders and maps Boot, Sandal, and Shoe to classes 0, 1, and 2.
  • Evaluation discrepancies: An ARIMA forecasting solution executes successfully but reports metrics that differ fundamentally from the standard answer.The reported MAPE is 77.28% versus the standard 2.62%, while the code uses the specified 15-observation test slice and calculates different values.

H Detailed Judge Results

The paper compares judge-model and human assessments to determine which automated judges best align with expert consensus. Final reported scores are based on the judges with closer human alignment, with model-coverage differences noted for later-added systems.

  • Judge selection: Claude-Haiku-4.5 and GPT-5.1 align more closely with human expert annotations than Doubao-Seed-1.8.Doubao-Seed-1.8 shows substantially larger deviation and is excluded from the final evaluation metric.
  • Judge selection: Final results use the average scores from Claude-Haiku-4.5 and GPT-5.1.The averaging decision follows the Human-LLM Alignment Study’s comparison with human expert annotations.
  • Human evaluation: Table 9 reports averaged human scores across reasoning, code, final results, and the overall score.These scores provide the human-consensus reference for the evaluation dimensions.
  • Model coverage: MiMo-V2-Pro and MiniMax-M2.7 appear in the Claude-Haiku-4.5 and GPT-5.1 results but not in the Doubao-Seed-1.8 results.The Human-LLM Alignment Study preceded their addition to the final evaluation set and followed Doubao-Seed-1.8’s exclusion from final judging.

I Qualitative Error Analysis

Qualitative inspection identifies recurring failure modes in complex computer-vision and NLP workflows, especially where agents must execute multi-step pipelines or handle unstructured data. These failures motivate stronger execution-grounded reasoning and validation of intermediate outputs, architectures, preprocessing, and reports.

  • Multi-step Code Execution Failures: Multi-step execution failures arise from hallucinated APIs, outdated libraries, and unsupported method calls that agents often fail to diagnose.Local edits may leave the underlying error unresolved, preventing modeling or evaluation, especially in deep-learning and non-standard-format tasks.
  • Incorrect Model Architecture Selection: Incorrect architecture selection causes shape mismatches when agents modify pretrained CNN components or classifier heads without grounding choices in input data and task requirements.The failure mode is especially apparent in computer-vision tasks.
  • Data Loading and Preprocessing Issues: Data-loading and preprocessing failures include incorrect file identification, directory parsing, dataset construction, field assumptions, tokenization, class mappings, and image transformations.These errors occur early and affect both NLP and computer-vision workflows.
  • Result Hallucination and Failure Masking: Result hallucination occurs when agents report metrics, hyperparameters, or configurations that executed code never successfully produced.Such unsupported reports can mask underlying execution failures and make responses appear more complete than they are.
  • Implications: The paper recommends verifying intermediate outputs, checking architectures against data shapes, prioritizing unstructured-data preprocessing, and grounding final reports in execution.These recommendations directly target the four observed failure modes.
Loading 2601.13591v2…