Source-linked AI summary
AgenticDataBench: A Comprehensive Benchmark for Data Agents
Zhaoyan Sun, Shan Zhong, Daizhou Wen, Jiaxing Han, Guoliang Li, Ying Yan, Peng Zhang, Yu Su, Xiang Qi, Baolin Sun, Chengyuan Yang, Tao Fang, Huaiyu Ruan
TL;DR
Data-agent research lacks benchmarks that capture diverse workflows and fine-grained performance, while existing skill sets suffer from scalability, redundancy, and entanglement. AgenticDataBench addresses this with skill-annotated real and generated tasks, finding that no harness dominates across domains and that Claude Code scores 46.6 versus 43.3 for Kimi-K2.5 at 1.5× the cost.
Problem
Existing skill sets are unsuitable for benchmark creation because they are difficult to scale, redundant, and entangled across abstraction levels.
Method
AgenticDataBench combines real and generated tasks with ground-truth solutions, annotated skills, task-specific evaluation, and skill-coverage-driven LLM generation.
Results
46.6 vs 43.3: Claude Code with Claude 4.6 scores higher than with Kimi-K2.5 while costing only 1.5× more, and no harness leads across all domains.
Takeaways & Limitations
Skill annotations expose agent-specific strengths, weaknesses, and failure patterns, including consistent difficulties with heterogeneous and non-relational data.
Abstract
from arXiv · showhide
Data science aims to derive actionable insights from heterogeneous raw data, unlocking the value of the massive amounts of data generated in modern society. Automating this process is essential to reducing labor-intensive efforts for data scientists and enabling scalable data-driven applications. Recently, large language model (LLM)-based data agents have emerged as a promising solution to automate data science workflows. However, the field lacks comprehensive benchmarks to rigorously evaluate these agents across diverse scenarios with fine-grained granularity. To address this gap, we propose AgenticDataBench, a comprehensive benchmark featuring realistic tasks spanning diverse domains with fine-grained ground-truth labels. This enables evaluations to capture the diversity and complexity of data science workflows and the detailed performance of agents. First, to cover diverse domains, we collect real datasets and tasks from 15 vertical domains, including 5 real-world B2B use cases from a leading fintech company. Second, to remove redundancy in real-world tasks and generate high-quality tasks for domains lacking real data, we introduce data science skills, recurring data-centric operational patterns, and quantify benchmark coverage by the number of skills included. Representative skills are extracted from large-scale task solutions on Stack Overflow using skill-aligned hierarchical clustering. Third, for real-world business tasks, we select task-solution pairs that maximize diversity in skill composition, ensuring broad coverage of practical scenarios. Fourth, to generate realistic tasks for devise domains without real tasks, we propose a systematic LLM-based task generation approach to create workflows and tasks based on these skills. Finally, we evaluate state-of-the-art data agents using our annotated benchmark and open-sourced testbed, providing detailed skill-level insights.
1 INTRODUCTION
AgenticDataBench addresses the lack of comprehensive, fine-grained benchmarks for LLM-based data agents by covering diverse data-science workflows and operational skills. It combines real-world task collection, skill discovery, redundancy reduction, and systematic task generation.
- Motivation: LLM advances in planning, reasoning, database operations, and code generation have enabled data agents to automate end-to-end insight extraction with minimal human intervention.These capabilities extend data science automation from individual subtasks to complete workflows over raw data.
- Data-agent challenges: Data agents must resolve ambiguous instructions, heterogeneous schemas, large datasets, and iterative execution across environments such as Python and databases.The workflow involves planning, generating executable code, interacting with execution environments, and repeatedly revising actions.
- Benchmark construction: The benchmark collects real datasets and tasks from 15 vertical domains, including 5 real-world B2B practices from a leading fintech company.These tasks feature large-scale noisy data and long code implementations but require processing because raw collections contain redundancy and domain gaps.
- Skill framework: AgenticDataBench extracts representative skills from Stack Overflow solutions using LLM decomposition and skill-aligned hierarchical clustering to reduce repeated task patterns.The method converts solutions into stepwise skill descriptions, then clusters semantically similar skills using pretrained text embeddings.
- Task generation: The benchmark generates tasks by sampling frequency-aware skill compositions, using structured dataset profiles, and producing corresponding workflows through a systematic LLM-based approach.This generation strategy targets domains lacking high-quality real tasks while preserving broad coverage of discovered data-science skills.
2 PRELIMINARIES
The preliminaries define data science skills as hierarchical, recurring operational patterns and formalize benchmark instances that associate tasks with datasets, solutions, required skills, and evaluation scores. They also distinguish this data-driven skill taxonomy from reusable or theoretical Agent Skills.
- Data Science Skills: Data science skills are hierarchical operational patterns, with broad parent nodes representing abstract capabilities and leaf nodes representing specific actionable skills.Each skill node includes a textual description and links to finer-grained child skills.
- Benchmark Instances: A benchmark instance is a quintuple comprising a task description, dataset, executable solution, required-skill set, and evaluation function.The evaluation function maps an agent’s output to a scalar score in [0, 1].
- Benchmark Instances: Each benchmark task requires specific skills and is paired with a ground-truth solution and evaluation function to assess data agents comprehensively.This design supports broad coverage of data science skills across task-solving workflows.
- Data Science Skill Category: Data science skills are categorized into seven exclusive workflow-stage categories, including Data Format Handling and Data Preprocessing.The listed categories cover operations such as data parsing, file handling, cleaning, transformation, validation, and feature engineering.
- Discussion: Unlike Agent Skills implemented as reusable modules or studied as atomic units of LLM performance, this work proposes a representative, data-driven skill set for data science scenarios.The proposed skill set is intended to provide a quantifiable foundation for evaluating data agents.
3 BENCHMARK OVERVIEW
AgenticDataBench is designed for broad, skill-measured coverage of data science workflows, combining real datasets and business tasks with systematically generated tasks. Its instances and evaluation pipeline support reproducible, fine-grained testing of data agents.
- Benchmark Design Criteria: 97 real datasets span 15 domains, covering diverse datasets and recurring data science task patterns.The collection includes Kaggle, UCI ML, Mendeley, academic, and government sources.
- Systematic Creation Methodology: The benchmark introduces a data science skill framework to measure task-pattern diversity and skill coverage.Representative skills are extracted from large-scale task solutions and guide benchmark development.
- Benchmark Pipeline: Each benchmark instance contains a task description, dataset, ground-truth solution, expert-annotated skills, and a task-specific evaluation method.The benchmark tests agents through Benchmark Instances, Data Agent, Executor, and Leaderboard components.
- Systematic Creation Methodology: Real-world fintech task-solution pairs are selected with a greedy approximation to maximize skill diversity under a fixed task budget.The underlying subset-selection problem is NP-hard, so tasks are iteratively chosen for broad skill coverage.
- Systematic Creation Methodology: An LLM-based pipeline generates practical tasks by sampling skill compositions and few-shot task–solution examples from a weighted skill graph.The graph merges skill application traces from Stack Overflow solutions and real practices to improve coverage of representative skills.
- Benchmark Pipeline: The Docker-based Executor supports Bash, Python, and database operations while loading benchmark datasets for agent exploration.An evaluation function scores task performance against the benchmark instance’s ground-truth answer.
4 HIERARCHICAL SKILL EXTRACTION
The benchmark derives a representative hierarchy of recurring data-science skills from large-scale task solutions, addressing scalability, redundancy, and entanglement through embedding-based clustering and expert refinement. This process produces 433 top-level skills for controlled benchmark construction.
- Skill Extraction: 6,510 Stack Overflow data-science tasks and solutions are filtered for relevant tags and quality indicators, then decomposed by an LLM into stepwise skill rationales.Selection requires accepted answers or scores above 3; complex solutions are decomposed because they often combine multiple skills.
- Motivation: The initial skills are unsuitable for benchmarking because tens of thousands of descriptions create scalability problems, while synonymous descriptions cause redundancy and overlapping abstractions cause entanglement.These issues reduce manageable benchmark size, skill diversity, and hierarchy clarity.
- Hierarchical Clustering: Qwen3-Embedding, UMAP, GMM, and LLM-based refinement organize skills into a hierarchy while allowing skills to associate with multiple higher-level abstractions.UMAP preserves local manifold structure, GMM supports soft clustering, and LLM refinement addresses abstractions missed by embedding similarity.
- Redundancy Reduction: DBSCAN merges synonymous skills using a strict distance threshold and selects the shortest description as the representative.The procedure separates semantically divergent skills before consolidating equivalent ones.
- Hierarchy Refinement: Recursive clustering summarizes skills with representative solution steps and repeatedly refines clusters until the number of top-level skills is manageable.Summaries augment generated descriptions with steps having the largest average cosine similarity to other steps using the skill.
- Expert Validation: 433 top-level skills remain after data-science experts review their scope, diversity, practical representativeness, and alignment with realistic evaluation scenarios.The resulting skills support benchmark tasks with controlled coverage and divergence.
5 SKILL-BASED BENCHMARK CREATION
AgenticDataBench combines real-world tasks with skill-based selection and LLM-generated workflows to provide broad, fine-grained coverage of data science capabilities. Each instance includes a task, ground-truth solution, required skills, and task-specific evaluation.
- Benchmark Construction: Each benchmark instance contains a real-world task description, ground-truth solution, annotated required skills, and a task-specific evaluation method.The construction is explicitly grounded in real-world datasets and business domains.
- Real-World Tasks: 30 domain experts from 5 business units curated 600 representative tasks using over 600 person-hours of anonymized Ant Group B2B production data.The tasks span industries including banking, finance, insurance, automotive, aviation, manufacturing, and retail, and scenarios such as exploration, modeling, and operations.
- Skill-Diverse Selection: The skill-diverse selection method annotates task skills with LLMs and greedily maximizes skill coverage while reducing redundancy under a budget.The resulting optimization is an NP-hard submodular maximization problem with a 1 −1/𝑒 approximation guarantee via greedy selection, followed by expert refinement.
- Cross-Domain Coverage: 58 datasets from open repositories extend coverage to 10 previously uncovered domains, including 46 Kaggle, 2 UCI ML, 2 Mendeley, and 8 academic or government datasets.Repositories are selected for real-world relevance, inherent complexity, and additional criteria described in the benchmark construction process.
- Skill-Based Generation: The task-generation pipeline samples realistic skill paths from a frequency- and dependency-weighted skill graph, retrieves representative task-solution examples, and uses dataset profiles to guide generation.The graph merges skill application traces from Stack Overflow solutions and real practices; sampling follows weighted nodes and edges.
- Quality Control and Diversity: Generated workflows are synthesized from sampled skills before LLMs produce task descriptions, with verification enforcing skill necessity, solvability, clarity, actionability, concision, and verifiable answers.Failed skill integrations are retried until a threshold and discarded when unsuccessful; repeated skills, graph edges, and examples receive dynamic sampling penalties to improve diversity.
6 EXPERIMENTS
Experiments evaluate multiple LLMs and agent harnesses on AgenticDataBench, revealing trade-offs among accuracy, cost, adaptability, and fine-grained skill performance. Skill- and failure-level analyses expose weaknesses that aggregate scores miss, especially for heterogeneous data and domain-specific workflows.
- Agent Harness Comparison: Production-grade harnesses outperform DA-Agent overall, but no harness achieves the best score across every domain.DA-Agent’s lightweight design lacks engineering optimizations and specialized components such as data profiling and data-science-specific skills.
- LLM Comparison: The best LLM varies by harness, demonstrating that agent accuracy depends on LLM–harness compatibility.Kimi-K2.5 performs best within CodeX, while Qwen3.5 and Claude 4.6 exhibit harness-specific execution or completion problems.
- Cost and Efficiency: 46.6 vs 43.3: Claude Code with Claude 4.6 scores higher than with Kimi-K2.5 while costing only 1.5× more.Claude Code appears particularly optimized for Claude models through higher token efficiency and cache utilization.
- Skill Analysis: Skill-level analysis identifies Data Alignment & Merging, Histogram Creation and Manipulation, and Text Processing and Cleaning as the lowest-scoring skills.These annotations reveal root causes and show that weaknesses vary across agents, domains, and skill categories, with persistent difficulty processing heterogeneous and non-relational data.
- Failure Analysis: Failure distributions vary substantially by domain: Marketing has the largest shares of Global Limit Exceeded and Single-Step Timeout, Healthcare the highest Self-Repair Failure, and Loan Model the most Data Modeling failures.Differences reflect data scale, file formats, file structure, and task complexity.
7 CONCLUSION
AgenticDataBench is presented as a comprehensive benchmark for evaluating data agents on realistic, diverse tasks with fine-grained labels. Its hierarchical skill extraction, task selection, and task generation modules enable controlled and skill-diverse benchmark coverage.
- AgenticDataBench evaluates data agents using realistic tasks spanning diverse domains with fine-grained labels.
- A hierarchical skill extraction algorithm uses LLM-based semantic refinement and agglomerative clustering aligned with skill boundaries.
- Task selection and generation modules provide controlled skill coverage, incorporating skill-diverse real-world tasks and realistic generated tasks.