PaperlayerView exact revision on arXiv

Source-linked AI summary

DataSciBench: An LLM Agent Benchmark for Data Science

Dan Zhang, Sining Zhoubian, Min Cai, Fengzu Li, Lekang Yang, Wei Wang, Tianjiao Dong, Ziniu Hu, Jie Tang, Yisong Yue

arXiv:2502.13897v1cs.CLcs.AIcs.LG

TL;DR

Existing data-science benchmarks often emphasize narrow tasks with easily obtained ground truth and simple metrics, leaving complex prompts and uncertain evaluation less covered. DataSciBench addresses this gap with curated challenging prompts, semi-automated ground-truth generation, and TFC-based programmatic evaluation. API-based models outperform open-source models overall, with GPT-4o leading all metrics and Deepseek-Coder-33B-Instruct leading open-source models.

  • Problem

    Existing benchmarks typically focus on narrower tasks with easy-to-obtain ground truth and straightforward metrics, whereas complex data-science prompts require broader evaluation design.

  • Method

    DataSciBench curates challenging prompts, generates ground truth through self-consistency and human verification, and evaluates outputs with TFC-based programmatic rules.

  • Results

    API-based models greatly outperform open-source models on average; GPT-4o surpasses all other models on all metrics, while Deepseek-Coder-33B-Instruct leads open-source models.

  • Takeaways & Limitations

    DataSciBench provides a broader evaluation of data-science LLM capabilities and reveals strengths, weaknesses, and directions for improvement.

  • Takeaways & Limitations

    Initial visualization metrics and VLM-as-a-judge methods may lack precision, requiring further refinement for fine-grained visualization evaluation.

Abstract

from arXiv · show

This paper presents DataSciBench, a comprehensive benchmark for evaluating Large Language Model (LLM) capabilities in data science. Recent related benchmarks have primarily focused on single tasks, easily obtainable ground truth, and straightforward evaluation metrics, which limits the scope of tasks that can be evaluated. In contrast, DataSciBench is constructed based on a more comprehensive and curated collection of natural and challenging prompts for uncertain ground truth and evaluation metrics. We develop a semi-automated pipeline for generating ground truth (GT) and validating evaluation metrics. This pipeline utilizes and implements an LLM-based self-consistency and human verification strategy to produce accurate GT by leveraging collected prompts, predefined task types, and aggregate functions (metrics). Furthermore, we propose an innovative Task - Function - Code (TFC) framework to assess each code execution outcome based on precisely defined metrics and programmatic rules. Our experimental framework involves testing 6 API-based models, 8 open-source general models, and 9 open-source code generation models using the diverse set of prompts we have gathered. This approach aims to provide a more comprehensive and rigorous evaluation of LLMs in data science, revealing their strengths and weaknesses. Experimental results demonstrate that API-based models outperform open-sourced models on all metrics and Deepseek-Coder-33B-Instruct achieves the highest score among open-sourced models. We release all code and data at https://github.com/THUDM/DataSciBench.

1 Introduction

DataSciBench targets the gap between simple single-task benchmarks and complex data-science prompts requiring multiple subtasks, uncertain ground truth, and tailored evaluation. It combines curated prompts, semi-automated ground-truth generation, and the Task-Function-Code framework to evaluate LLMs comprehensively.

  • Existing benchmarks typically evaluate narrower tasks with easy-to-obtain ground truth and straightforward metrics, while real-world data analysis involves more complex multi-subtask scenarios.
  • API-based models greatly outperform open-source models on average; GPT-4o leads all metrics, while Deepseek-Coder-33B-Instruct leads open-source models.
  • DataSciBench introduces a comprehensive benchmark and semi-automated pipeline for generating ground truth and evaluating aggregated metrics on carefully crafted data-science questions.
  • The Task-Function-Code framework links task selection, evaluation functions, and programmatic code to assess data-science responses.
  • DataSciBench evaluates 23 models and analyzes their results across coarse-grained and fine-grained perspectives.

2 Background on using LLMs for Data Science

Prior data-science LLM benchmarks often rely on simplistic metrics and readily available ground truth. DataSciBench seeks a more comprehensive and nuanced assessment of model strengths and weaknesses.

  • Existing evaluations may use simplistic metrics and readily available ground truth, limiting thorough assessment of LLMs in data science.
  • DataSciBench addresses these shortcomings through a rigorous evaluation framework intended to provide a more comprehensive and nuanced assessment.

3 DataSciBench

DataSciBench combines curated multi-task prompts, validated ground truth, and TFC-based programmatic evaluation. Its dataset contains 222 prompts, 519 test cases, and 25 aggregated functions for evaluating diverse LLMs.

  • Prompt Definition and Collection: The benchmark defines six data-science task types, including preprocessing, statistics, visualization, predictive modeling, mining, and interpretability.
  • Prompt Definition and Collection: Prompts are collected from online and public code benchmarks, filtered for quality, and reviewed by experts before standardized formatting.
  • Response Integration and Validation: Ground truth is generated with repeated LLM sampling, self-consistency, and author verification, including cross-validation when outputs are uncertain.
  • Response Integration and Validation: GPT-4o-mini selects valuable task types, evaluation functions, and evaluation code, while generated task structures are represented as TFC tuples and organized in DAGs.
  • Response Integration and Validation: The framework aggregates functions into 25 categories and applies unified boolean or decimal scoring rules, with thresholds for selected decimal outputs.
  • Summary: The resulting benchmark contains 222 effective prompts, 519 test cases, and 25 aggregated functions for evaluating 6 API-based and 17 open-source models.

4 Experiments

DataSciBench evaluates 23 models with coarse- and fine-grained metrics across curated prompts, using programmatic and model-based assessments of task completion, success, accuracy, and visualization quality.

  • Experimental Setup: 23 models are evaluated, comprising 6 API-based models, 8 open-sourced general models, and 9 open-sourced code-generation models.The benchmark includes both general-purpose and code-specialized open models alongside API-based systems.
  • Coarse-Grained Metrics: Completion Rate scores each TFC step from 0 to 2 and normalizes total step scores by the maximum possible score, 2 × T.Missing or failed steps receive 0, while successful compliant steps receive 2.
  • Coarse-Grained Metrics: Success Rate measures the proportion of 10 runs in which every TFC for a prompt passes.For BigCodeBench prompts, pass status is determined by comparing completion outputs with the ground-truth function outputs.
  • Fine-Grained Metrics: Fine-grained evaluation includes VLM-based visual judgments, data-quality matching, plot validity, data accuracy, visualization completeness, and model accuracy.These metrics assess both whether outputs exist and whether they are accurate or complete.
  • Final Score: The final score combines five metrics, a VLM score, Success Rate, and Completion Rate using weighted factors.Each factor receives a specified weight, including weights of 0.05 or 0.65.

5 Results and Analysis

DataSciBench shows that API-based models generally outperform open-sourced models, while performance varies with task difficulty and complex data-science requirements. The results also reveal mismatches between benchmark performance and practical abilities such as instruction following, tool use, and planning.

  • Overall Performance: 64.51%: GPT-4o achieves the highest total score, outperforming GPT-4-Turbo’s 54.65% total score by 9.86 percentage points.GPT-4o also surpasses all other models on every reported metric.
  • Overall Performance: 56.76%: Deepseek-Coder-33B-Instruct achieves the highest score among open-sourced models, while CodeLlama-34B-Instruct reaches only 1.33%.Qwen2.5-Coder-7B-Instruct and Qwen2.5-7B-Instruct score 47.67% and 45.99%, respectively.
  • Difficulty Analysis: GPT-4o, GPT-4o-mini, GPT-4-Turbo, and Deepseek-Coder-33B-Instruct maintain consistent completion rates across difficulty levels, indicating robustness.GPT-series and Deepseek-Coder-Instruct models perform particularly well on hard, data-driven tasks.
  • Benchmark Comparison: Most LLMs occupy the upper triangular region when comparing Pass@1 on DataSciBench and HumanEval, but some models show substantial cross-benchmark discrepancies.The comparison separates models that perform well on both benchmarks from those whose performance differs between datasets.
  • Deeper Insights: Reasoning proficiency does not ensure success on complex data-science tasks requiring precise instruction following, tool use, and strategic planning.The results indicate that even models strong in reasoning can fail on these integrated requirements.
  • Deeper Insights: 29.77%: o1-mini’s overall success rate trails GPT-4o and GPT-4-Turbo because of instruction non-compliance, incorrect calls, forgetfulness, and execution errors.Observed failures include not exporting required outputs, calling nonexistent functions, and hallucinating CSV column names.
  • Deeper Insights: Larger models can underperform smaller versions when they struggle to generate the output formats required by simple instructions.StarCoder2-15B and CodeLlama-34B-Instruct are cited as examples of this pattern.

6 Conclusion

The paper introduces DataSciBench to evaluate LLMs on challenging data-science tasks using curated prompts, generated ground truth, aggregated metrics, and programmatic TFC evaluation. It applies this framework to 23 models and reports research and engineering insights alongside error analyses.

  • Conclusion: DataSciBench is a benchmark tailored to assessing LLM capabilities in data-science tasks.Its prompts are curated to make evaluation more challenging and comprehensive.
  • Conclusion: The framework generates ground truth with robust LLMs and self-consistency, then evaluates model performance through aggregated metrics and programmatically synthesized TFC lists.This combines benchmark construction with structured execution-outcome evaluation.
  • Conclusion: The study evaluates 23 API-based and open-source models and provides research and engineering insights together with model error analyses.The evaluation covers both performance assessment and diagnosis of failure patterns.

7 Limitations

The paper identifies imprecision in some visualization-task metrics and describes six broad data-science task categories covered by the benchmark. These categories span data preparation, analysis, modeling, pattern discovery, visualization, and reporting.

  • Limitation: Visualization metrics such as VLM-as-a-judge may lack precision for certain tasks, requiring further refinement.The paper suggests training critic models with VLMs as one possible direction for finer-grained visual evaluation.
  • Task Scope: Data cleaning and preprocessing covers missing values, outliers, duplicates, and standardized data formats.Dates and times are given as an example of formatting standardization.
  • Task Scope: Data exploration and statistics understanding includes descriptive statistics, distribution charts, correlations, and correlation maps.Examples include means, medians, standard deviations, histograms, and box plots.
  • Task Scope: Data visualization focuses on analyzing data and creating interactive charts for user exploration.The task is distinct from broader statistical analysis because it emphasizes interactive visual presentation.
  • Task Scope: Predictive modeling includes algorithm selection, feature engineering, train-test evaluation, and task-appropriate metrics.The stated algorithms include linear regression, decision trees, and random forests.
  • Task Scope: Data mining and pattern recognition use mining, clustering, classification, text mining, image recognition, and time-series detection to identify patterns.The category spans both structured and unstructured data applications.
  • Task Scope: Interpretability and report generation explain model results and produce accessible, shareable summaries.Feature importance and model parameters are cited as examples of explanations.

A.2 Comparison with Existing Benchmarks

DataSciBench correlates positively with established coding benchmarks while adding domain-specific coverage of data-science tasks. Its differing model rankings indicate that it captures capabilities that general programming benchmarks may miss.

  • Correlation Analysis: Pearson’s r and Spearman’s p are used to assess alignment between DataSciBench and the BCB and LCB coding evaluations.The analysis is intended to examine relationships across evaluation dimensions.
  • Correlation Analysis: DataSciBench shows strong positive correlations with established coding evaluation metrics.This indicates alignment between the benchmark and those coding evaluations.
  • Distinctive Contributions: DataSciBench targets data-science and analytics tasks, whereas existing benchmarks primarily emphasize general programming problems.The domain-specific focus is intended to assess real-world data-analysis scenarios.
  • Distinctive Contributions: Its task diversity includes preprocessing, visualization, and statistical analysis, which are underrepresented in current benchmarks.This provides additional insight into data-science-specific capabilities.
  • Distinctive Contributions: Overall correlations coexist with meaningful differences in model rankings, including distinct patterns for Meta-Llama-3-8B-Instruct and CodeLlama-34B-Instruct.These differences indicate capabilities specific to data-science tasks that other benchmarks may not capture.
  • Conclusion: The benchmark’s correlation with existing evaluations supports its methodology, while its domain-specific focus contributes new information for assessing AI models in data-science applications.The two roles are presented as complementary rather than interchangeable.

A.3 Motivation and Example of Task-Function-Code (TFC)

The TFC framework structures data-science evaluation by linking task categories, evaluation functions, executable code, and ground-truth verification. It aims to make assessment comprehensive, standardized, reproducible, and scalable.

  • Framework components: TFC systematically categorizes key data-science tasks across six established types.This organization supports comprehensive coverage and consistent evaluation.
  • Framework components: TFC defines task-specific evaluation functions to standardize assessment across operations such as preprocessing and visualization.Different data-science tasks receive appropriate metrics rather than a single criterion.
  • Framework components: Executable task and metric code improves evaluation efficiency, reproducibility, and testing scalability.
  • Framework components: TFC supports ground-truth construction for complex tasks where reliable ground truth is not readily available.It also enables systematic verification and validation of model outputs.
  • Framework contribution: The framework bridges task definition, evaluation criteria, and automated assessment in data-science contexts.The paper presents this integrated TFC structure as a novel contribution.

A.5 Caveats when Using LLMs for Data Science

The paper identifies several caveats affecting LLM use in data science, including dataset memorization, preprocessing hallucinations, and multilingual encoding errors. These issues can disrupt otherwise intended workflows.

  • General caveat: Some testcase-generation issues may be disruptive, while others are subtle but important.Most identified issues were addressed by modifying prompts.
  • Dataset and preprocessing risks: LLMs may memorize well-known open-source datasets and process columns absent from a customized data split.The paper gives GPT-4o processing excluded penguin-dataset columns as an example.
  • Dataset and preprocessing risks: During preprocessing, models may hallucinate a common column when merging CSV files instead of checking all available columns.
  • Multilingual risks: On multilingual tasks, LLMs may select an incorrect file encoding, including latin instead of gbk for Chinese CSV content.

A.6.1 LLMs for Data Science

The paper positions DataSciBench as a broader evaluation of data-science agents than benchmarks centered on isolated, simple operations. Its materials include programmatic and VLM-based assessment examples for diverse outputs.

  • Evaluation materials: Table 4 contains the programmatic rules used to evaluate generated outputs.
  • Benchmark scope: Data-science agents require comprehensive evaluation because analysis correctness can substantially affect decision-making.
  • Benchmark scope: Existing data-science benchmarks often use single-task prompts and relatively simple data-analysis operations.The paper contrasts these settings with practical questions involving multiple subtasks and complex operations.
  • Evaluation materials: The VLM-as-a-judge prompt scores chart similarity using data representation, axes and scales, graphical elements, legends and annotations, and layout.The stated scoring procedure assigns one point to each criterion.
  • Evaluation examples: The examples illustrate both matching charts receiving positive criterion scores and mismatched charts receiving zero scores.The comparisons cover data, axes, graphical elements, legends, and dimensions.
  • Prompt examples: A collected prompt combines missing-value analysis, row selection, and weight imputation using height and region information.The qualified version also specifies CSV output files for the requested results.

B Error Analysis and Case Study

The case study examines recurring execution and formatting errors and follows a multi-step correlation task from preprocessing through output-file verification. The example reports successful completion of the requested workflow.

  • Error analysis: Common errors include coding or execution failures caused by issues such as hallucinated CSV column names.
  • Error analysis: Agent frameworks can also produce JSON-format errors when wrapping actions such as WriteAnalysis.
  • Case study workflow: The correlation case loads data.csv, sets the first column as an index, computes daily percentage changes, and drops the resulting first-row NaN values.
  • Case study workflow: The most_corr function computes a correlation matrix, removes diagonal and duplicate entries, and selects the maximum remaining correlation.
  • Case study outcome: The reported example successfully completes the function and writes the result to most_corr_output.csv.
  • Case study workflow: The output task verifies that most_corr_output.csv has columns Ticker 1 and Ticker 2 and exactly one row.

B.2 Failure Case

The failure cases show that generated code often omitted required outputs or libraries, applied the wrong algorithm, or produced execution errors. These failures occurred across plotting, clustering, preprocessing, and data-construction tasks.

  • B.2 Failure Case: Required-output violations caused failures in normal-distribution plotting and data-cleaning tasks.One response returned processed arrays instead of the required figure, while another failed to output the required elements.
  • B.2 Failure Case: The grouped-bar-plot task included preprocessing and plotting steps, but the response still failed to output the required elements.The code handled missing values, duplicates, outliers, date formats, and grouped means before plotting.
  • B.2 Failure Case: The K-Means task failed because the model removed outliers with z-scores instead of calling sklearn.cluster functions.The implementation checked numeric inputs, dropped missing rows, and filtered outliers, but did not perform the requested clustering.
  • B.2 Failure Case: Missing library imports caused an otherwise specified grouped-bar-plot solution to fail.The failure was explicitly attributed to forgetting relevant libraries.
  • B.2 Failure Case: Execution failures also included a KeyError during data processing.The recorded error was KeyError: "[’target’] not found in axis".
Loading 2502.13897v1…