Source-linked AI summary
AgentDS Technical Report: Benchmarking the Future of Human-AI Collaboration in Domain-Specific Data Science
An Luo, Jin Du, Xun Xian, Robert Specht, Fangqiao Tian, Ganghua Wang, Xuan Bi, Charles Fleming, Ashish Kundu, Jayanth Srinivasa, Mingyi Hong, Rui Zhang, Tianxi Li, Galin Jones, Jie Ding
TL;DR
AgentDS asks whether autonomous AI agents can match human experts on domain-specific data science tasks and where human advantages remain. It introduces a 17-challenge benchmark and open competition comparing AI-only and human–AI approaches. The results show that agents struggle with domain-specific reasoning, while the strongest solutions emerge from human–AI collaboration.
Problem
Existing research and benchmarks often emphasize generic code and tabular pipelines, leaving evidence limited about AI agents’ performance on domain-specific, multimodal data science tasks.
Method
AgentDS combines 17 realistic challenges across six domains with an open competition and autonomous agent baselines to compare AI-only and human–AI collaborative performance.
Results
Current agentic AI systems struggle with domain-specific reasoning, whereas the most successful solutions combine human strategic guidance with AI-assisted coding, experimentation, and iteration.
Takeaways & Limitations
The findings support designing AI systems for effective human–AI collaboration and augmentation of human expertise rather than fully autonomous data science.
Takeaways & Limitations
AI capabilities evolve rapidly, so the performance gaps documented by AgentDS may narrow as agentic systems advance.
Abstract
from arXiv · showhide
Data science plays a critical role in transforming complex data into actionable insights across numerous domains. Recent developments in large language models (LLMs) and artificial intelligence (AI) agents have significantly automated data science workflow. However, it remains unclear to what extent AI agents can match the performance of human experts on domain-specific data science tasks, and in which aspects human expertise continues to provide advantages. We introduce AgentDS, a benchmark and competition designed to evaluate both AI agents and human-AI collaboration performance in domain-specific data science. AgentDS consists of 17 challenges across six industries: commerce, food production, healthcare, insurance, manufacturing, and retail banking. We conducted an open competition involving 29 teams and 80 participants, enabling systematic comparison between human-AI collaborative approaches and AI-only baselines. Our results show that current AI agents struggle with domain-specific reasoning. AI-only baselines perform below the top quartile of competition participants, while the strongest solutions arise from human-AI collaboration. These findings challenge the narrative of complete automation by AI and underscore the enduring importance of human expertise in data science, while illuminating directions for the next generation of AI. Visit the AgentDS website here: https://agentds.org/ and open source datasets here: https://huggingface.co/datasets/lainmn/AgentDS .
1 Introduction
AgentDS addresses whether human experts outperform autonomous AI agents on domain-specific data science tasks and where human advantages arise. Its benchmark and competition show that domain reasoning remains difficult for agents, while the strongest performance comes from human–AI collaboration.
- Motivation: AgentDS targets a gap in benchmarks that often emphasize generic code and tabular pipelines rather than domain-specific knowledge and multimodal insight.The benchmark includes 17 challenges across six domains and realistic industry problems designed to reward domain-informed processing and feature engineering.
- Findings: Current autonomous agents struggle with domain-specific reasoning, especially when multimodal signals must be incorporated.Some teams abandoned autonomous frameworks for interactive human-guided workflows.
- Findings: Human experts contribute failure diagnosis, domain-informed feature design and rules, and strategic decisions about model selection and generalization.
- Findings: Human–AI collaboration produces the most successful approaches by combining human strategic reasoning with AI-assisted coding, experimentation, and iteration.
- Contribution: AgentDS provides a replicable framework for evaluating human, AI, and collaborative performance in domain-specific data science.Its challenge construction, evaluation protocol, and competition infrastructure can be adapted by other institutions and communities.
2 The AgentDS Benchmark and Competition
AgentDS is designed to test domain-specific reasoning under realistic, heterogeneous data conditions rather than generic machine learning performance. Its benchmark combines six industry domains, multimodal inputs, synthetic but plausible datasets, and a systematic curation pipeline.
- Design principles: Strong performance requires domain-specific insights because generic methods yield baseline results while competitive performance depends on context-specific features and processing.
- Design principles: The benchmark integrates tabular data with images, text, and structured files to reflect heterogeneous real-world data science challenges.
- Data construction: Although the datasets are synthesized, their generation process mirrors genuine industry relationships, constraints, and correlations informed by domain literature.
- Benchmark scope: AgentDS covers six domains with challenges spanning classification, regression, and ranking tasks shaped by domain knowledge and business-specific evaluation criteria.The domains were selected for real-world importance, technical challenge, and diversity of required skills.
- Data construction: The curation pipeline researches each domain, generates constrained data, calibrates difficulty using performance bounds, and validates realism and documentation with domain experts.Latent variables can be transformed into additional modalities so that effective feature extraction requires domain-specific insight.
2.4 Evaluation Framework
AgentDS converts heterogeneous challenge metrics into comparable scores through quantile normalization, then aggregates them hierarchically across challenges and domains. Tie-breaking uses submission efficiency indicators.
- Quantile scoring: Quantile scoring normalizes challenge-specific metrics such as Macro-F1, RMSE, and normalized Gini coefficient to a common [0, 1] scale.
- Quantile scoring: The quantile score assigns the top performer 1, the worst performer 1/n, and intermediate ranks by linear interpolation.Participants who do not submit successfully receive 0 for that challenge.
- Score aggregation: Domain scores average quantile scores across each domain’s challenges, while the overall score averages the six domain scores.This challenge-to-domain-to-overall hierarchy gives every challenge equal weight in the final ranking.
- Tie breaking: Ties are broken first by fewer submissions and then by the earlier final submission.
2.5 The AgentDS Competition
The AgentDS competition evaluates human–AI collaboration in realistic data science workflows by allowing participants to use AI tools freely. It attracted broad participation and produced reproducible submissions for analysis.
- Competition setup: Participants could freely use AI tools, allowing the competition to capture human–AI interaction in realistic domain-specific data science workflows.
- Competition setup: 29 teams comprising 80 participants made successful submissions during the 10-day competition.The competition received more than 400 registrations and allowed teams of up to four people.
- Competition setup: Teams could submit up to 100 solutions per challenge, after which code and reports were collected for reproducibility checks and further analysis.
2.6 AI-Only Baselines
AI-only baselines are evaluated through direct prompting and autonomous agentic coding, but their performance remains below the strongest human competition teams. Agentic workflows generally outperform direct prompting, yet still struggle to exploit domain-specific signals consistently.
- Baseline configurations: Two AI-only baseline families are compared: one-shot frontier-LLM prompting and autonomous coding or data-analysis tools.Direct prompting produces a complete Python solution once, whereas agentic tools can execute code, inspect results, and revise submissions within a 10-minute budget.
- Overall performance: 0.510 overall quantile ranks MorphMind highest among the agentic baselines, followed by Claude Code at 0.458 and Julius AI at 0.382.These correspond to ranks 9th, 10th, and 11th among 29 teams.
- Overall performance: 0.415 overall quantile makes GPT-5.5 the strongest direct-prompting baseline, while GPT-4o trails at 0.183.Claude Opus 4.7 scores 0.391, DeepSeek V4 Pro 0.325, and Gemini 3.1 Pro 0.216; all AI-only baselines remain below the strongest human teams.
- Domain-level performance: Agentic baselines outperform direct-prompting LLMs across most domains, with MorphMind strongest in Healthcare at 0.699 and Manufacturing at 0.647.Claude Code performs best in Retail Banking at 0.553, while direct-prompting models show competitive scores in selected domains such as Food Production.
- Challenge-level performance: Challenge-level performance is highly variable: GPT-5.5 reaches 0.933 on Food Production Ch. 1, while Julius AI scores zero on several challenges.MorphMind also records strong results on Healthcare Ch. 3 at 0.783 and Manufacturing Ch. 1 at 0.786, but no baseline consistently matches the strongest human solutions.
- Interpretation: Agentic tool use improves performance over direct prompting, but AI-only baselines still rely on standard modeling strategies and underuse domain-specific signals in additional modalities.Task-specificity alone is insufficient; workflows must support strategy-level iteration for domain-specific data science.
3 Empirical Findings from AgentDS
AgentDS findings show that current AI agents struggle with domain-specific reasoning, while the strongest performance comes from human-guided collaboration that combines human judgment with AI implementation and iteration.
- AI Agents Struggle with Domain-Specific Reasoning: AI agents struggle to leverage multimodal signals and often default to generic pipelines that perform poorly when domain-specific insight is essential.The weakness spans images, CSV, PDF, text, and JSON modalities.
- AI Agents Struggle with Domain-Specific Reasoning: Fully autonomous agentic approaches remain ineffective for complex domain-specific tasks and were abandoned by several teams in favor of interactive human-AI workflows.One team reported that multiturn orchestration required extensive prompt engineering and significant API costs before switching to interactive coding agents.
- Human Expertise Provides Irreplaceable Value: Human expertise contributed through strategic diagnosis, domain-informed feature design, filtering AI suggestions, and judgment beyond validation scores.Participants identified modeling failures, encoded clinical knowledge, rejected harmful pipelines, and assessed generalization risks.
- Human-AI Collaboration: Successful teams used AI for routine implementation and rapid experimentation while humans retained control over strategic decisions and feedback loops.Effective workflows were human-initiated: people proposed approaches, evaluated results, diagnosed problems, and refined instructions.
- Human-AI Collaboration: Human-AI teams excel through complementarity, combining human domain grounding, causal reasoning, and error correction with AI computational power and rapid prototyping.The reported collaboration pattern outperformed either humans or AI acting alone.
4 Limitations and Future Work
AgentDS identifies limitations in its synthetic data, participant pool, domain coverage, evolving AI landscape, and observational analysis of collaboration, while outlining reproducible benchmark components and future studies.
- Limitations: Synthetic datasets cannot capture the full messiness, ambiguity, and noise of genuine industry data.Future iterations may incorporate real anonymized datasets where feasible.
- Limitations: The competition’s participation pool was valuable but limited, and larger, more diverse engagement would strengthen the findings.The authors aim to expand outreach in future editions.
- Limitations: Six domains do not exhaust applied data science, so additional domains are needed to test generalization.Suggested extensions include energy and other areas of finance.
- Limitations: Rapidly improving AI systems may narrow the documented performance gaps, making repeated evaluation over time important.The open datasets and protocol support direct comparison of future systems on the same challenges.
- Limitations: Participant reports, code, and workflow inspection provide qualitative insight but do not permit controlled comparisons of collaboration strategies.Future work could vary autonomy, prompting, and human oversight systematically.
- Future Work: AgentDS can be reproduced through challenge construction, evaluation protocol, competition infrastructure, and AI baseline evaluation.The proposed framework includes domain-grounded data generation, metric-agnostic scoring, submission validation, and documented direct-prompting or agentic baselines.
5 Conclusion
AgentDS introduces a controlled benchmark and open competition for evaluating autonomous agents and human-AI collaboration on domain-specific data science. Its results show that human expertise remains essential and that the strongest solutions combine human guidance with AI acceleration, motivating collaboration-oriented systems.
- Contribution: AgentDS comprises 17 challenges across six domains designed to require domain knowledge, multimodal reasoning, and thoughtful modeling beyond generic pipelines.Its controlled data generation framework and open competition evaluate both autonomous AI agents and human-AI collaboration.
- Findings: Current agentic AI struggles with domain-specific reasoning, while humans diagnose failures, inject domain knowledge, and make strategic generalization decisions.These findings are presented as three consistent results of the benchmark.
- Implications: The strongest solutions emerge from human-AI collaboration, with humans guiding problem solving while AI accelerates coding, experimentation, and iteration.The conclusion suggests designing systems that augment rather than replace human expertise.
Disclosure Statement
The benchmark and competition were developed and organized at the University of Minnesota with financial support from University of Minnesota institutes and cloud computing provided through Microsoft Azure.
- Support: AgentDS received financial support from the Data Science and AI Hub and the Institute for Research in Statistics and its Applications at the University of Minnesota.
- Organization: The AgentDS benchmark and competition were developed and organized at the University of Minnesota, with cloud computing services provided through Microsoft Azure.
- Affiliations: Xun Xian and Jie Ding are affiliated with AIScientists Inc., which developed the MorphMind platform used in the numerical study.
A Implementation details for direct prompting LLM baselines
The direct-prompting baseline gives one LLM call per challenge a documented task, data preview, and strict execution environment, then evaluates the resulting submission through a standardized pipeline.
- A Implementation details for direct prompting LLM baselines: Each run uses one direct LLM call for one of 17 challenges spanning six domains.The model receives the challenge specification and produces a solution for only the selected challenge index.
- A.1 Chat messages: The prompt contains a fixed system message and a user message assembled from challenge metadata, the full description.md, a data preview, and closing instructions.The preview summarizes on-disk layouts, tabular rows and columns, JSON structures, and image or PDF filenames.
- A.1.1 System message: The harness supplies dataset and writable output paths through environment variables rather than machine-specific absolute paths.The script is saved and executed later in a fresh process against the real files under BENCHMARK_DATA_DIR.
- A.1.1 System message: The generated output must be a single Python program that reads BENCHMARK_DATA_DIR and writes submission.csv to BENCHMARK_WORK_DIR with the required columns and row order.The system message also restricts the response to code wrapped in specified markers and disallows prose outside them.
- A.1.2 User message: The user prompt identifies the challenge, dataset root, official specification, and data preview before instructing the model to produce the required program.Input files may be joined or stacked for feature construction, but the deliverable must remain a single submission for the selected task.
- A.1.2 User message: Input files may be combined for the selected task, while structured measurement lists require missing numeric entries to be removed before aggregation.The prompt specifically recommends guarding empty lists when computing aggregate values.
- A.2 Evaluation protocol: Each challenge–model pair proceeds through prompt construction, code extraction, fresh-process execution, and platform submission.Execution outcomes distinguish successful output, process failure, missing submission, and absent extractable code; runs use deterministic provider settings and may receive minimal repairs that do not alter modeling strategy.
- A.2 Evaluation protocol: Minimal manual repairs can make failed generated code runnable, but they cannot add features, change the intended modeling strategy, or replace the solution with a stronger baseline.The protocol records raw model output and submits the resulting file through the AgentDS evaluation platform.
A.3 Scoring for reported results
AgentDS converts successful raw challenge scores into human-relative quantiles and averages those quantiles within domains and across all six domains.
- A.3 Scoring for reported results: Each raw score is mapped to a human-participant quantile on the same challenge, excluding zero scores and respecting whether higher or lower values are better.Domain scores average challenge quantiles within a domain, while overall scores average across the six domains.
B Implementation details for agentic coding baselines
The agentic coding baselines use autonomous coding environments with a shared base model, challenge-specific prompts, and a ten-minute run budget to generate and submit predictions.
- B Implementation details for agentic coding baselines: Three autonomous coding tools—Claude Code, MorphMind, and Julius AI—are evaluated with Claude Opus 4.7 as their shared base model.Using one base model makes comparisons across the tools more consistent.
- B Implementation details for agentic coding baselines: Each agent reads the official specification, builds a predictive model, generates test-set predictions, writes submission.csv, and submits it for one challenge number.The best submitted score before voluntary stopping or the ten-minute limit is recorded.
- B Implementation details for agentic coding baselines: The prompt template supplies data and working directories, the description path, and the selected challenge number before requesting a complete model and submission.The agent is told to stop when satisfied with its submission.
- B Implementation details for agentic coding baselines: Submission uses the AgentDS BenchmarkClient with the configured API key, team name, domain, challenge number, and submission-file path.The command authenticates and uploads the generated prediction file through the evaluation API.
- B Implementation details for agentic coding baselines: Claude Code runs locally in non-interactive autonomous mode, whereas MorphMind and Julius AI use cloud environments that download the relevant domain data before execution.All three tools are limited to a ten-minute time budget per challenge.