Towards Reliable AI Data Scientists: Data Agents with Workflow Harnesses

1 Introduction

• Five recurring challenges are incomplete semantic context, underdetermined solutions, dynamic tool execution, sparse failure signals, and repairs entangled with local data context.A task can produce a plausible result without an exception, while explicit errors may reveal only a downstream symptom.Source: §1 · pp. 1–2
• A workflow harness governs an iterative process of perception, planning, execution, verification, and repair rather than relying on general reasoning or tools alone.The harness is intended to guide behavior where common data-environment failures may produce no explicit error.Source: §1 · pp. 2–3
• The harness goals are semantic grounding, solution-space navigation, dynamic tool orchestration, routine verification, and generalized repair that can transfer across data contexts.Routine checks can catch errors before they cascade, while generalized repair abstracts local corrections into reusable knowledge.Source: §1 · p. 3
• Prior work is fragmented across modalities, tasks, and workflow stages, limiting integrated support for cross-modal or long-horizon analysis [50] [70] [109].The literature is also shifting from isolated query answering and static prompting toward agentic analysis and feedback-driven refinement.Source: §1 · p. 3
• The survey organizes Data Agents around five workflow stages and 15 technical routes, identifies four open reliability problems, and reviews task families, applications, and benchmarks.Its contribution also includes a formal foundation for heterogeneous data environments and agent trajectories.Source: §1 · pp. 3–4

2.1 Evolution of the AI Data Scientist Paradigm

• Data Agents extend earlier automation, from static AutoEDA reports and rigid AutoML assumptions to code agents, by combining reasoning with a disciplined harness for end-to-end data tasks.AutoResearch systems add iterative training-code modification but generally lack comprehensive workflow design.Source: §2.1 · p. 4

2.2 Formal Definition of the Data Environment

• The environment is a finite collection of heterogeneous sources, each represented by payload, type, structure, and properties that together define its data context and state.Structures include schemas or layouts, while properties include constraints, dependencies, and statistical distributions.Source: §2.2 · p. 4
• Structured sources have explicit structure and well-defined properties, semi-structured sources require structure inference, and unstructured sources require active extraction from payloads.The respective examples include relational data, spreadsheets or JSON, and PDFs, images, or raw text.Source: §2.2 · pp. 4–5

2.3 Data Agents as AI Data Scientists

• A Data Agent is an LLM-driven system that interacts with a persistent data environment and computational tools, iteratively refining actions into verifiable analytical artifacts.Artifacts include cleaned datasets, visualizations, reports, and trained models.Source: §2.3 · p. 6

2.3.1 Operational Formalization of a Data Agent.

• The operational model maps a task, initial data state, execution environment, and LLM to a trajectory of actions, observations, evolving states, and a final artifact.The execution history conditions the agent’s next decision.Source: §2.3.1 · p. 6
• The survey covers analytics and ML-oriented tasks, distinguishing multi-step agents whose later actions depend on intermediate observations from one-shot prediction or generation.Examples include profiling, visualization, data manipulation, feature engineering, model selection, and training.Source: §2.3.1 · p. 6

2.3.2 Handling Silent Failures with a Workflow Harness.

• A silent failure can yield incorrect or incomplete output without an explicit error; the harness governs, monitors, and revises the task trajectory as data states evolve.The harness is not a fixed sequence: it guides interpretation of state, action selection, outcome evaluation, and response to deviations.Source: §2.3.2 · p. 6
• Unlike developer-specified task graphs in traditional orchestrators, Data Agents select, evaluate, and revise analytical actions as the data state evolves.This allows them to detect and respond to silent failures during execution.Source: §2.3.2 · p. 6

3 The Workflow Harness of AI Data Scientists

• Perception, planning, execution, verification, and repair are invoked as needed rather than run once in a fixed sequence.The harness allows stages to be revisited and iterated throughout task execution.Source: §3 · p. 7
• Methods within each stage are grouped into complementary routes that may be combined in one Data Agent.Source: §3 · p. 7
• The harness frames questions about understanding data state, choosing plans, executing tools, validating progress, and repairing failures for future tasks.Source: §3 · p. 7

3.1 Perception

• Perception combines data structure probing, cross-source alignment, and confidence-calibrated state scoring to construct a useful perceived state.Probing inspects organization and sample values; alignment connects heterogeneous sources; scoring flags fragile assumptions for downstream stages.Source: §3.1 · p. 7

3.1.1 Data Structure Probing.

• Data structure probing iteratively updates its understanding through exploratory queries and inspection of structures and sample values.Across structured, semi-structured, and unstructured data, probing examines properties and context needed to form the perceived state.Source: §3.1.1 · p. 8

3.1.2 Cross-Source Data Alignment.

• Cross-source alignment connects semantically related information across heterogeneous sources while preserving provenance to the original data.The route addresses differences in schemas, formats, granularities, and representations.Source: §3.1.2 · pp. 8–9

3.1.3 Confidence-Calibrated State Scoring.

• Confidence-calibrated scoring attaches reliability signals to perceived information so later stages can distinguish supported content from uncertainty.Signals can come from model confidence, sampling, or disagreement, and should remain attached as the state moves downstream.Source: §3.1.3 · pp. 9–10

3.2 Planning

• Planning turns the perceived data state into an executable blueprint using tree search, heuristic search, or strategy routing.Search routes explore candidate paths; routing selects a predefined strategy based on the current state.Source: §3.2 · p. 10, Figure 4 · p. 11

3.2.1 Tree Search.

• Tree search explores branching decision paths and uses estimated values and downstream outcomes to guide which branches receive further exploration.In unstructured settings, approaches can discretize retrieval and reasoning decisions into scoreable branches.Source: §3.2.1 · pp. 11–12

3.2.2 Heuristic Search.

• Heuristic search iteratively generates candidate trajectories and retains promising ones using task-specific scoring rather than a global search tree.Local heuristics help adapt to large or irregular action spaces, but exploration depends on their quality.Source: §3.2.2 · p. 12

3.2.3 Strategy Routing.

• Strategy routing maps the current state and task requirements to a predefined strategy instead of constructing new candidate paths.The selected strategy can change as new information arrives, while the strategy space remains predefined.Source: §3.2.3 · pp. 12–13

3.3 Execution

• Execution organizes tool-mediated work into capability boundary learning, compositional tool reasoning, and feedback-adaptive tool calling.The routes draw on analytical requirements, tool specifications, and observed execution feedback.Source: §3.3 · pp. 13–14, Figure 5 · p. 14

3.3.1 Tool Capability Boundary Learning.

• Capability boundary learning matches a tool’s supported operations and inputs to the current analytical requirement, refining estimates when observed behavior differs from specifications.An allocation study finds that models can make near-optimal decisions abstractly yet fail to preserve them when instantiating them through executable code.Source: §3.3.1 · pp. 14–15

3.3.2 Compositional Tool Reasoning.

• Compositional tool reasoning assembles multiple operations into a sequence in which intermediate results inform subsequent requirements and calls.Tool dependencies and specifications can be represented explicitly, while outputs update the analytical state used to determine the next step.Source: §3.3.2 · pp. 15–16

3.3.3 Feedback-Adaptive Tool Calling.

• Feedback-adaptive calling evaluates whether tool use advanced the analytical objective and revises calls or future policies when execution evidence indicates a problem.Adaptation is anchored to task requirements, constrained by available tool specifications, and informed by execution traces.Source: §3.3.3 · pp. 16–17

3.4.1 Process Score Estimation.

• Process score estimation evaluates intermediate states or actions before the final outcome, using learned scores, reviewer critique, execution traces, or statistical signals.Structured-data methods include reward models; semi-structured methods use multi-agent review or scored traces; unstructured-data work uses statistical estimation and benchmarks to expose process failures.Source: §3.4.1 · pp. 17–18

3.4.2 Rule-based Constraint Checking.

• Rule-based constraint checking tests intermediate observations against explicit validity conditions, from database equivalence and performance bounds to evidence grounding and semantic criteria.The conditions may be structural in databases, tied to task and table evidence for semi-structured data, or based on analytical invariants and domain assumptions in unstructured settings.Source: §3.4.2 · p. 18

3.4.3 Trace-Level Explainable Attribution.

• Trace-level explainable attribution locates which earlier action contributed to a detected failure, using interventions, trajectory patterns, or backward tracing as data properties become less explicit.Structured-data methods intervene on trace steps; semi-structured methods analyze broader failure trajectories; unstructured methods trace failures backward through execution history.Source: §3.4.3 · pp. 18–19

3.5 Repair

• The three repair routes address where recovery begins, how a failure is corrected, and what validated experience should be retained.They are data state reconstruction, search-guided intervention-based repair, and reusable memory skills.Source: §3.5 · pp. 19–20

3.5.1 Data State Reconstruction.

• Data state reconstruction resumes from a recoverable intermediate state and recomputes affected work rather than restarting the workflow.Examples include rollback with compensable steps, restoring constraint-valid states, branching spreadsheet work after an edit, and preserving validated retrieval-and-reasoning prefixes.Source: §3.5.1 · p. 20

3.5.2 Search-Guided Intervention-Based Repair.

• Search-guided intervention-based repair tests alternative corrections by re-executing affected steps and using downstream outcomes to refine or retain interventions.The search ranges from narrowing repair variants to iterative replanning and global workflow search when local changes interact unpredictably.Source: §3.5.2 · p. 21

3.5.3 Reusable Memory Skills.

• Reusable memory skills convert validated repairs into retrievable experience or learned strategies that can guide later recovery.Approaches store trajectories or contrastive repairs, learn transferable patterns, or retrieve success and failure guidance under similar conditions.Source: §3.5.3 · pp. 21–22

The Inactive Semantic Calibration Problem

• Semantic calibration remains inactive when uncertain perceptions of data semantics are collapsed into deterministic assumptions that downstream analysis cannot condition on.Because those assumptions can be syntactically valid, the workflow may silently produce flawed charts, reports, or models.Source: §4 · p. 23

The Missing Clarification Problem

• Missing clarification arises when agents guess among valid interpretations of ambiguous requests instead of deciding when user alignment is needed.Without standards for useful questions or stopping, agents may ask too little or too much.Source: §4 · p. 23

The Missing Experience Transfer Problem

• Experience transfer remains sensitive to task similarity, so repairs based on superficial similarities can mismatch new data structures or distributions and cause invalid results.Stored trajectories may mix broadly useful corrections with steps incidental to the original dataset.Source: §4 · p. 23

The Missing Verification-Repair Repository Problem

• The field lacks large-scale standardized repositories of faulty trajectories, failure signals, and repairs, limiting shared evidence for training and evaluation.Heterogeneous failures range from formatting anomalies and distribution shifts to statistical assumptions and algorithm misconfigurations.Source: §4 · pp. 23–24

5 Horizontal Analytical Task Taxonomy

• The horizontal taxonomy classifies what analytical objectives agents pursue, distinct from the workflow stages describing how they operate.A single objective may involve several workflow stages, and the same operations may serve different objectives.Source: §5 · p. 24

5.1 Data Collection and Acquisition

• Data collection and acquisition obtains information needed for downstream analysis by selecting sources and iteratively gathering relevant evidence.Tasks may involve dataset discovery, web gathering, database access, or external-resource retrieval.Source: §5.1 · p. 24

5.2 Data Preparation and Transformation

• Data preparation and transformation converts data into a representation suitable for analysis through context-sensitive, often iterative operations.Agents inspect data, select transformations, and assess whether the resulting representation preserves semantics and suits the analytical objective.Source: §5.2 · pp. 24–25

5.3 Data Querying and Information Seeking

• Data querying and information seeking answers defined questions by grounding requests in data structures and producing executable queries or programs.An executable query may still select the wrong column, value, or relationship, leaving a gap between executable and semantic correctness.Source: §5.3 · p. 25

5.4 Data Analysis and Prediction

• Data analysis and prediction identifies patterns and produces predictions, classifications, or diagnostic results using methods suited to the data and objective.Agents commonly use specialized statistical or machine-learning tools and interpret their outputs.Source: §5.4 · pp. 25–26

5.5 Data Interpretation and Multimodal Reasoning

• Data interpretation and multimodal reasoning explores patterns and synthesizes evidence across sources, requiring findings to remain faithful to underlying data.This family includes exploratory analysis, visualization and insight generation, and document or multimodal reasoning.Source: §5.5 · p. 26

6.1 Interactive Data Assistance

• Interactive Data Assistance translates users’ analytical intentions into operations and interpretable results through natural-language interaction and iterative dialogue.Conversational business intelligence is a representative setting for querying enterprise data and refining questions.Source: §6.1 · p. 27

6.2 Collaborative Analytical Work

• Collaborative analytical work distributes workflow responsibilities across agents, tools, or reasoning modules and coordinates their intermediate results.This division of labor can keep agents embedded in human-guided workflows.Source: §6.2 · p. 27

6.3 Autonomous Data Workflows

• Autonomous Data Workflows execute dependent analytical operations with limited supervision, adapting later actions to intermediate results.They can combine data preparation, database operations, modeling, and execution feedback.Source: §6.3 · pp. 27–28

6.4 Knowledge-Intensive Discovery

• Knowledge-Intensive Discovery combines information seeking, heterogeneous evidence, and data analysis to reach domain-specific conclusions.Its iterative retrieval and analysis supports tasks whose conclusions depend on external evidence.Source: §6.4 · pp. 28–29

7 Benchmarking Autonomous Data Agents

• Benchmarking should assess both supported analytical capabilities and the reliability, quality, and efficiency of the overall analytical process.The landscape varies by assigned objective, data and execution complexity, and success criteria.Source: §7 · p. 29

7.1 Benchmark Objectives and Task Coverage

• Benchmarks progress from focused operations toward multi-step workflows coordinating data understanding, planning, tool use, and result interpretation.Success on an individual operation does not establish that an agent can complete a coherent workflow.Source: §7.1 · p. 29

7.2 Data, Environment, and Task Complexity

• Benchmark difficulty also depends on data complexity, knowledge requirements, interaction length, and environmental uncertainty.Greater data heterogeneity and longer interactions make reasoning harder to separate from retrieval, tool use, and state management.Source: §7.2 · pp. 29–30

7.3 Evaluation Criteria

• Evaluation criteria extend beyond final answers to execution, evidence quality, process coherence, efficiency, robustness, and failure attribution.Criteria vary by setting, including query faithfulness, transformation correctness, evidence sufficiency, and visual grounding.Source: §7.3 · pp. 30–31

7.4 Limitations of Current Benchmarking

• Outcome-based metrics can miss semantic, evidence-selection, and grounding failures, while process quality and failure attribution remain difficult to measure consistently.A valid query may misunderstand the intended schema, and a plausible multimodal answer may rely on the wrong visual evidence.Source: §7.4 · p. 31

• Prior surveys characterize autonomy, analytical capabilities, lifecycle stages, data modalities, and system architectures.These perspectives clarify agent responsibilities and system organization but do not necessarily compare operational mechanisms.Source: §8 · pp. 31–32
• Task- and lifecycle-based taxonomies describe analytical objectives, while similar planning, verification, and repair mechanisms can recur across different tasks.Autonomy level alone does not explain how agents interpret data states, validate results, or recover from failures.Source: §8 · p. 32
• The survey organizes Data Agents by perception, planning, execution, verification, and repair, then compares technical routes within each stage.This stage-wise view supports process-level analysis of how uncertainty, ambiguity, experience, and verification signals are handled.Source: §8 · p. 32

• The survey organizes Data Agent workflows into perception, planning, execution, verification, and repair, identifying 15 technical routes across these stages.It also identifies inactive semantic calibration, missing clarification, missing experience transfer, and a missing verification-repair repository as open problems.Source: §9 · p. 32
• Reliable Data Agents depend on how uncertainty, ambiguity, experience, and failure information are handled throughout the workflow, not only on stronger models or more tools.Source: §9 · p. 33
• The survey also summarizes task families, vertical application settings, and benchmarks for analytical capabilities and workflow reliability.Source: §9 · p. 33