PaperlayerView exact revision on arXiv

Source-linked AI summary

The Silent Failures in Agentic Data Science

Xue Li, Hongqian Xia, Daniel Gomm, Aécio Santos, Cornelius Wolff, Madelon Hulsebos

arXiv:2610.14115v1cs.AI

TL;DR

Existing agentic data science evaluations emphasize predictive performance and code correctness, leaving methodological practice less examined. This paper analyzes failures across tasks, tests selected pitfalls under controlled conditions, and compares agent pipelines with expert solutions; most contain failures, and leakage failures correlate with higher predictive performance.

  • Problem

    Agentic data science evaluations emphasize downstream performance and code correctness, while task- and data-dependent methodological validity is less directly assessed.

    Source: §1 · p. 2

  • Method

    The paper combines failure analysis across 220 tasks, a controlled suite targeting six failures, and qualitative comparisons with expert Kaggle solutions.

    Source: §1 · p. 2

  • Results

    Most agent-generated pipelines contain at least one failure mode, and data leakage failures correlate with inflated predictive performance.

    Source: §1 · p. 2, §4.2 · p. 8

  • Takeaways & Limitations

    The findings support evaluating agentic data science beyond accuracy and considering methodological validity throughout the pipeline.

    Source: §5 · p. 10

    The large-scale analysis uses a predefined set of common failure modes and therefore captures only a subset of potential pipeline problems.

    Source: §3.1 · p. 4

Abstract

from arXiv · show

Large language models have significantly lowered the barrier to writing code. Naturally, this has also led to the development of data science agents that promise automating the design and execution of data science from task formulation to model development. Existing evaluations of agentic data science (agentic DS) assess downstream predictive performance and code correctness, providing preliminary insight in agentic DS capabilities. We argue that these evaluations overlook the fundamental principles of data science: methodological data science- and domain knowledge. As such, only assessing downstream performance masks methodological failures such as data leakage and inappropriate validation, limiting the development of robust agentic DS systems. In this paper, we evaluate agentic DS beyond accuracy from three different angles: 1) a large-scale analysis of failure modes in common agentic DS frameworks and models across 220 data science tasks, 2) a reusable and extensible test suite with curated data science tasks that test for six specific failures in a controlled manner, and 3) a qualitative assessment of agentic DS pipelines against expert solutions on Kaggle competitions. We find that the majority of agentic solutions contain at least one failure mode despite the complexity of agentic harness or model capacity. Moreover, we identify a positive correlation between the occurrence of data leakage failures and predictive performance, confirming the necessity to evaluate agents beyond accuracy. Our qualitative analysis shows that DS agents generally lack problem solving creativity and domain knowledge, compared to competition-winning data scientists. We believe that principled failure mode analysis, as presented here, should be integral part of agentic DS evaluations, and that the insights we surface will be key in developing trustworthy agentic DS pipelines.

1 INTRODUCTION

Data science agents need evaluation that goes beyond code execution and predictive scores because sound practice depends on the task, data, and domain knowledge. The paper examines methodological failures that conventional evaluations may miss and argues for assessing agents’ practices as well as outcomes.

1 INTRODUCTION

  • Data science pipelines are task- and data-dependent, and assessing their validity and utility requires input from domain experts, engineers, and data scientists.For example, temporal use cases may require chronological splitting, while metrics should suit the target distribution.Source: §1 · pp. 1–2
  • Existing evaluations emphasize downstream performance, but silent failures can distort assessments of agents’ generalizability and capability.The paper argues that reliable evaluation requires finer-grained analysis of failure modes.Source: §1 · p. 2
  • Silent failures are task-relevant methodological violations that compromise solution validity or soundness but may escape output-based evaluation.They can coexist with executable code and strong predictive performance.Source: §1 · p. 2
  • The paper evaluates data science agents through a 220-task failure analysis, a controlled test suite, and qualitative comparisons with human experts.Across these analyses, most agent-generated pipelines contain at least one data science failure mode.Source: §1 · p. 2

2 RELATED WORK

Prior work on automated and agentic data science spans optimization, foundation models, and agents that automate broader workflow stages. Existing agent benchmarks largely emphasize predictive performance, while this paper analyzes the practices and methodological failures behind agent-produced models.

2 RELATED WORK

  • AutoML systems search predefined model and hyperparameter spaces, while tabular foundation models aim to predict in one forward pass and agents automate more of the data science workflow.Agent approaches include single-loop construction, iterative refinement, and systematic design-space exploration.Source: §2 · p. 2
  • Agentic data science benchmarks mainly use Kaggle competitions and assess predictive performance, with failure analyses often limited to execution errors.This paper instead characterizes agents’ modeling practices and where those practices fail.Source: §2 · p. 3
  • Existing agent failure taxonomies focus largely on coordination, grounding, and execution, leaving data science methodological violations incompletely represented.Prior analyses also identify planning, reasoning, data understanding, and broader-context errors as important areas of agent failure.Source: §2 · p. 3

3 EVALUATION METHODOLOGY

The evaluation combines large-scale analysis, controlled tests, and comparisons with high-performing real-world solutions. Its methods account for task-specific applicability, deliberately test selected pitfalls, and compare agent practices and performance with expert work.

3 EVALUATION METHODOLOGY

  • The methodology combines LLM-as-judge analysis of large-scale failures, curated tests targeting specific failures, and qualitative comparison with high-performing solutions.These are presented as three complementary approaches to understanding why data science agents fail.Source: §3 · p. 3

3.1 LARGE-SCALE FAILURE MODE ANALYSIS

  • The large-scale analysis uses a two-stage judge: it first identifies pipeline structure and applicable failures, then checks each applicable failure and extracts relevant code lines.The first stage identifies components such as feature engineering, validation, tuning, model comparison, and ensembling; the second gives a binary decision for each failure mode.Source: §3.1 · p. 3
  • The analysis uses predefined data science failure modes grouped into leakage and non-leakage categories, with some modes evaluated conditionally on pipeline characteristics.The taxonomy draws on commonly identified bad practices, but does not exhaust all possible pipeline problems.Source: §3.1 · pp. 3–4, Table 1 · p. 4

3.2 CURATED TEST SUITE WITH DATA SCIENCE FALLACIES

  • The controlled test suite embeds a targeted pitfall in each task, varies hint explicitness, and checks for predefined acceptable treatments across five runs per agent-task-hint combination.Its six tests are selected for controllable instantiation, clearly specifiable responses, and limited confounding.Source: §3.2 · p. 4, Table 2 · p. 5

3.3 SKILL GAP ANALYSIS WITH EXPERT DATA SCIENCE SOLUTIONS

  • The expert comparison submits agent solutions to late-submission Kaggle leaderboards and compares both predictive performance and modeling practices with high-ranking solutions.AutoGluon provides an additional reference on selected tasks.Source: §3.3 · p. 4

4 SURFACING SILENT FAILURES IN DATA SCIENCE AGENTS

Across large-scale, controlled, and Kaggle evaluations, data science agents exhibit methodological failures and do not consistently match expert solutions. The findings show that predictive performance alone can misrepresent pipeline quality.

4.1 EXPERIMENTAL SETUP

  • The study evaluates 220 tasks drawn from five sources, comparing three data science agents and three LLM capacities; a separate curated test suite contains 17 tasks across three underlying datasets.Datasets are split 80:20 into train and test sets; agents receive both and are expected not to use the test set during training.Source: §4.1 · pp. 5–6

4.2 LARGE-SCALE ANALYSIS SURFACES RECURRENT SILENT FAILURES

  • At least 43% of pipelines in every framework–model configuration had a failure; greater workflow coverage shifted failures from skipped steps to errors within steps.Raw failure counts are not a direct measure of pipeline quality because omitted workflow stages remove opportunities for within-stage errors.Source: §4.2 · pp. 6–7
  • Failures outside the fixed taxonomy appeared in 37% of pipelines, and 50% of MLEvolve pipelines with gemini-3.7-flash showed validation-to-train leakage.That MLEvolve configuration had the fewest failures within the taxonomy, despite its dedicated leakage-checking agent.Source: §4.2 · p. 7
  • All tested leakage failure modes improved test-set ranks, while non-leakage failures generally worsened them, showing that scores can reward methodological errors.Not using a validation split had the largest non-leakage effect, worsening rank by more than one position on average.Source: Figure 2 · p. 7, §4.2 · pp. 7–8

4.3 CONTROLLED EVALUATION CONFIRMS CONSISTENT FAILURE PATTERNS

  • In controlled tests, correlation–causation reasoning and feature-selection leakage failed in every configuration; normalization also failed every run in five configurations.Class imbalance, domain feature engineering, and metric selection varied substantially across agents and models.Source: §4.3 · p. 8, Figure 3 · p. 8
  • More explicit hints sharply reduced ReAct and AIDE failures, but stronger hints did not further improve MLEvolve, which remained less reliable under explicit guidance.The authors suggest many failures reflect not recognizing a methodological constraint, while MLEvolve may fail to maintain constraints during pipeline generation.Source: §4.3 · p. 8, Figure 4 · p. 8

4.4 DATA SCIENCE AGENTS RARELY DISCOVER EXPERT-LEVEL STRATEGIES

  • Across five Kaggle challenges, expert solutions remained ahead; gaps were smaller on two accessible Playground tasks and larger on three specialized challenges.AutoGluon results apply only to the two CSV-upload Playground tasks.Source: Table 4 · p. 9, §4.4 · p. 9
  • No agent led across all Kaggle tasks, and the Problematic Internet Use case study found leaked target-derived features gave ReAct and AIDE near-perfect local QWK scores despite competition-incompatible pipelines.The winning solution instead predicted continuous PCIAT Total, applied optimized thresholds at the end, and used a Tweedie objective, multi-seed 10-fold cross-validation, and diverse model families.Source: §4.4 · p. 9

5 CONCLUSION

The paper evaluates data-science-agent pipelines beyond accuracy through analyses across 220 tasks, a controlled test suite, and comparisons with expert solutions. It finds that leakage can accompany stronger test performance and that more sophisticated frameworks do not necessarily eliminate failures.

5 CONCLUSION

  • Data leakage failures can lead to stronger predictive performance on a test set, while sophisticated frameworks do not necessarily eliminate failures.The authors argue that these failures are masked by purely downstream performance evaluation.Source: §5 · pp. 9–10
  • The evaluation combines a large-scale analysis across 220 tasks, an extensible controlled test suite, and comparisons with expert real-world solutions.Source: §5 · p. 9
  • Dedicated leakage checks in some frameworks do not ensure that leakage failures are absent.The authors suggest that preventing failures requires reasoning about methodological validity throughout the pipeline.Source: §5 · p. 10

AI USE STATEMENT

The authors report using generative AI tools for implementation, analysis support, software and figure work, and editing. They report not using the tools for research-method design or several specified forms of theoretical and hypothesis work, and say they reviewed the AI-assisted work.

AI USE STATEMENT

  • Generative AI tools supported method implementation, dataset cleaning and reformatting, qualitative and thematic analysis, and result interpretation.Source: AI USE STATEMENT · p. 10
  • The authors state that generative AI tools were not used to design or provide feedback on methodology or experiments, generate synthetic datasets, develop theoretical models, or propose or refine hypotheses.They also report no AI assistance with proofs or mathematical claims; translation was not applicable.Source: AI USE STATEMENT · p. 10
  • The authors report using AI tools for figures, experimental parameters, code, project artifacts, and readability edits, and say they reviewed AI-assisted work and validated generated artifacts.One author thoroughly reviewed and tested generated code.Source: AI USE STATEMENT · p. 10

A.1 DETAILS ON MEMORIZATION CHECK SETUP

The appendix reports paired Wilcoxon tests comparing performance across high- and low-memorization tasks. None of the six tests was significant at α = 0.05.

A.1 DETAILS ON MEMORIZATION CHECK SETUP

  • The authors report no significant performance changes regardless of memorizing the tables.Source: §A.1 · p. 14
  • None of the six paired Wilcoxon tests of Δalt versus rerun randomized Δplc reached significance at α = 0.05.Source: Table 5 · p. 14
  • The tests were computed separately for high- and low-memorization-rate tasks across agentic frameworks.Source: Table 5 · p. 14

A.2 FAILURE MODES DEFINITIONS

Table 6 contains definitions for each failure mode evaluated in the large-scale analysis.

A.2 FAILURE MODES DEFINITIONS

A.3 DATASET DETAILS

The curated benchmark covers 220 tasks, with dataset characteristics summarized by source, task type, median rows, and median features; a separate table reports domain distribution.

A.3 DATASET DETAILS

  • The benchmark’s 220 tasks are characterized by source, task type, median rows, and median features.Table 7 presents these dataset characteristics.Source: §A.3 · p. 15, Table 7 · p. 15
  • Task-type columns distinguish regression, binary classification, and multi-class classification.The listed fields also include median rows and median features.Source: §A.3 · p. 15
  • A separate table summarizes the distribution of tasks across domains.Table 8 reports domain distribution across the 220 benchmark tasks.Source: Table 8 · p. 15

A.4 HUMAN ANNOTATOR AGREEMENT

A human annotator independently checked 15 failure modes on 19 recoverable, randomly sampled pipelines; agreement with the LLM judge was substantial, with disagreements concentrated in two modes.

A.4 HUMAN ANNOTATOR AGREEMENT

  • The judge agreed with the human annotator on 89.5% of 285 pipeline-by-failure-mode ratings (Cohen’s κ = 0.62, 95% CI [0.50, 0.73]).The 19 pipelines were randomly sampled, and 11 of the original 30 were excluded because their generated code was unrecoverable.Source: §A.4 · p. 15
  • Present-class precision was 0.71 and recall was 0.65, with no significant directional over- or under-flagging bias (p = 0.29).Agreement was scored as present versus not-present; absent/N/A distinctions were not scored as disagreement.Source: §A.4 · p. 15
  • Agreement was lowest for naive ensembling and reliance on a single evaluation metric (0.68 each), while the other 13 modes reached at least 0.79.Nine of those remaining modes reached at least 0.89 agreement.Source: §A.4 · p. 15

A.5 FAILURE ANALYSIS

The failure analysis adjusts predictive-rank comparisons for agent and model identity, while the additional-failure analysis identifies frequent leakage and training-protocol issues beyond the taxonomy and compares agent procedures with expert solutions.

A.5 FAILURE ANALYSIS

  • The failure analysis reports applicable-pipeline counts by failure mode, agent, and model.Table 10 covers modes requiring a particular pipeline stage or data type.Source: §A.5 · p. 15, Table 10 · p. 16

A.6 EFFECTS OF FAILURES ON PREDICTIVE PERFORMANCE

  • Predictive performance is measured by solution rank across nine configurations, with each failure-mode effect estimated after adjusting for model and agent identity.Scores are calculated on pre-processed test sets; the adjustment addresses confounding from differences in model and agent capability.Source: §A.6 · pp. 16–17
  • The plotted partial-residual group-mean difference represents the adjusted effect of a failure mode on predictive rank.Figure 7 presents effects in residual rank; its caption does not provide effect values.Source: §A.6 · pp. 17–18, Figure 7 · p. 18

A.7 FAILURE MODES BEYOND THE TAXONOMY

  • The judge flagged 2,294 additional failures, of which 1,343 were retained as data-science-principle violations after filtering other issues.Categories were derived through manual screening and assigned using independent keyword matching and an LLM judge, with disagreements manually resolved; labels agreed on 95% of 59 blindly reviewed reports.Source: §A.7 · p. 18
  • Beyond-taxonomy categories include validation→train leakage, test→train leakage, and flawed training protocols; each of several less frequent categories occurs in at most 6% of pipelines.Examples include preprocessing fitted before cross-validation, test-data transforms fitted jointly with training data, and failure to retrain on all training data.Source: §A.7 · p. 18, Table 11 · p. 19
  • Validation→train leakage rises from 8% of pipelines for qwen-3.5-4b to 16% for qwen-3.8-27b and 26% for gemini-3.7-flash.It reaches 33% for MLEvolve with qwen-3.8-27b and 50% with gemini-3.7-flash; rates are lower bounds because the judge was not specifically prompted to find these failures.Source: §A.7 · pp. 18–19, Table 11 · p. 19
  • MLEvolve with gemini-3.7-flash has the highest share of pipelines with an outside-taxonomy failure (57%), despite having the fewest failures within the taxonomy.Validation→train leakage is almost absent when there is no validation step, as in ReAct with qwen-3.5-4b.Source: §A.7 · p. 19

A.8 PROCEDURAL COMPARISONS BETWEEN AGENTIC SOLUTIONS AND EXPERT SOLUTIONS

  • Figure 8 compares key steps and procedures in solutions from three agents with expert solutions from Kaggle competitions.The supplied passage describes the comparison setup but does not state its outcome.Source: §A.8 · p. 19, Figure 8 · p. 19
Loading 2610.14115v1…