Data Interpreter: An LLM Agent For Data Science

1 INTRODUCTION

• Data-science workflows span interconnected tasks from data processing to feature engineering and model training, requiring iterative refinement and real-time adjustment as data and requirements change.The field combines computer science, statistics, data visualization, and mathematics, and spans data gathering through modeling and decision-making.Source: §1 · pp. 1–2
• Data Interpreter reports a 25% performance boost on InfiAgent-DABench and a 26% improvement on MATH.The introduction also reports notable advancements over open-source frameworks on machine-learning and open-ended tasks; Figure 1 compares frameworks across data-science tasks using a comprehensive score.Source: Figure 1 · p. 2, §1 · p. 2
• Prior LLM approaches target individual data-science tasks, often in fixed pipelines, and struggle with end-to-end evaluation, real-time data changes, and evolving task dependencies.Complex machine-learning and data-transformation tasks require continuous updates and dynamic global planning.Source: §1 · p. 2
• Data Interpreter models tasks as nodes and dependencies as graph edges, enabling real-time task management as data and requirements evolve.Programmable Node Generation automates node generation, refinement, and verification to improve workflow robustness and precision.Source: §1 · p. 2

LLMs as Data Science Agents

• Prior data-science agents span code generation, interpreters, analysis, case-based reasoning, and tuning, but end-to-end frameworks for predictive pipelines remain insufficiently developed.The paper positions its unified framework as addressing this gap and reports benchmarking across diverse tasks and settings.Source: §2 · p. 3

Enhancing LLM with Tools

• Tool-using research advances from external tools and automatic selection to structured planning, while this work adds independent dynamic orchestration and combination of tools using execution experience.Prior work includes multi-agent systems, retrieval and tool selection, tool creation and instruction, and structured pipelines for planning and invocation.Source: §2 · p. 3

Graph-Based Planning for LLM Agents

• CoT, ToT, GoT, DSPy, and other graph-planning approaches structure or refine plans, but often struggle with multi-step, task-dependent data-science problems.The paper explores graph structures for LLM agents in data science, an area it describes as largely untapped.Source: §2 · p. 3

3.1 HIERARCHICAL GRAPH MODELING FOR COMPLEX TASK DECOMPOSITION

• Data Interpreter decomposes long data-science workflows into executable subproblems linked as a DAG, splitting difficult nodes further; LLM in-context learning makes this decomposition training-free.The method aims to optimize subprocess relationships to improve output performance.Source: §3.1 · pp. 4–5, Figure 2 · p. 4
• Task graphs track tasks and dependencies to construct relevant context dynamically, while action graphs represent independently executable and verifiable actions that can adapt to runtime feedback.The task graph avoids retrieving all context at once; action-graph adjustments can refine code or add verification when execution fails.Source: Figure 2 · p. 4, §3.1 · p. 5

3.2 TASK GRAPH: ITERATIVE GRAPH REFINEMENT

• Iterative graph refinement updates task plans and code in response to execution outcomes, enabling adaptation at both planning and code levels.The executor uses reflection to regenerate code after exceptions or failed checks; after bounded retries, failed tasks can trigger graph regeneration that preserves completed work and avoids redundant execution.Source: §3.2 · pp. 5–6

3.3 ACTION GRAPH: PROGRAMMABLE NODE GENERATION

• Programmable node generation retrieves task-relevant tools and integrates them with library operations and domain expertise in comprehensive, context-aware executable code.Tool ranking uses task descriptions, types, and dependencies; the system selects top-k tools or skips tool selection when none are suitable.Source: §3.3 · pp. 6–7

4.1 EXPERIMENTAL SETUP

• The evaluation covered 257 InfiAgent-DABench challenges, eight Kaggle ML datasets, 20 open-ended tasks, and level-5 MATH problems in four categories.The open-ended benchmark averaged results over three runs; the MATH categories were C.Prob, N.Theory, Prealg, and Precalc.Source: §4.1 · pp. 7–8

4.2 MAIN RESULT

• With gpt-4o, Data Interpreter improved InfiAgent-DABench accuracy by 25% over direct LLM inference.With gpt-4-0613, it scored 73.55 and outperformed AutoGen by 2.9%, but context-window overflow meant it did not surpass direct LLM invocation.Source: Table 1 · p. 8, §4.2 · p. 8
• Data Interpreter scored 0.95 on ML-Benchmark, versus AutoGen’s 0.86 and OpenDevin’s 0.88 comprehensive scores.It averaged 0.84 while using 27.9% of OpenDevin’s cost and scored above 0.9 on Titanic, House Prices, SCTP, and ICR.Source: §4.2 · p. 8, Table 2 · p. 9
• Data Interpreter achieved a 0.97 completion rate on open-ended tasks, improving over AutoGen by 110.8% and OpenDevin by 61.7%.Its OCR completion rate was 0.85; it alone executed every step in WPI, I2C, and T2I, while baseline login failures lowered ER completion.Source: §4.2 · pp. 8–9, Table 3 · p. 9
• On MATH, Data Interpreter achieved 0.82 accuracy in N.Theory, 0.29 in Precalc, and a 26.5% average relative improvement over AutoGen.It achieved the best results across all four tested categories.Source: Figure 4 · p. 9, §4.2 · p. 9

4.3 ABLATION STUDY

• On ML-Benchmark, iterative graph refinement improved performance by 0.48, and programmable node generation raised the comprehensive score by 10.6% to 0.94.The refinement enhanced dataset preparation and real-time tracking; the ablation sequence began with ReAct code execution and simplified prompts.Source: §4.3 · pp. 9–10, Table 4 · p. 10
• Data Interpreter improved task completion with GPT-4o and GPT-4o-mini, while smaller LLMs degraded on advanced coding and image or webpage tool tasks.Qwen-72B-Chat and Mixtral-8x7B performed comparably to GPT-3.5-Turbo on machine-learning tasks; smaller models still handled data loading and analysis effectively.Source: Figure 5 · p. 10, §4.3 · p. 10

5 CONCLUSION

• Data Interpreter continuously monitors data changes and adapts to dynamic environments through iterative task refinement and graph optimization.Its framework represents tasks with hierarchical graphs.Source: §5 · p. 10
• Hierarchical decomposition, fine-grained execution, validation, and iterative modifications enhance analysis, machine learning, and reasoning for complex multi-step tasks.The approach combines these mechanisms with the LLM’s planning and coding abilities.Source: §5 · p. 10
• Extensive evaluations show Data Interpreter outperforms various open-source frameworks in machine learning tasks, mathematical problems, and real-world applications.Source: §5 · p. 10

A LIMITATIONS

• The evaluation covers only entry-level Kaggle datasets and junior-data-scientist tasks, not large-scale data or complex tasks such as time series, multi-label, or multitable problems.The evaluated datasets are under 500MB, with hundreds of columns and tens of thousands of rows, and mainly involve classification and regression.Source: A LIMITATIONS · p. 15
• Data Interpreter lacks automatic self-improvement from numerical feedback across experiments, and its MATH evaluation is limited to level-5 problems by budget constraints.The authors plan to enable multiple experiments and derive dataset-specific insights from numerical feedback; they also plan more cost-effective evaluation across a wider range of math problems.Source: A LIMITATIONS · p. 15

B BROADER IMPACT

• Flexible tool integration can expose systems to malicious code, so the experiments prompt the agent to check code before generating new code and recommend LLMs with robust safety policies.The passage identifies unauthorized system penetration and web attacks as examples of potential harm.Source: B BROADER IMPACT · p. 15

C.1 PROGRAMMABLE NODE GENERATION

• The node-generation pipeline selects tools based on task metadata classification, then organizes multiple tools as needed to accomplish tasks.The paper illustrates this process with tools.Source: Figure 6 · p. 16, §C.1 · p. 16

C.1.1 AN EXAMPLE OF TOOL SCHEMA

• The example tool schema defines a feature-engineering class that adds categorical-column value counts as a new feature, with a required column parameter.The schema also describes fit, fit_transform, and transform methods for DataFrames.Source: §C.1.1 · p. 17

C.1.3 TOOL USAGE PROMPTS

• Open-ended tasks use zero-shot tool prompts, while machine-learning tasks use one-shot prompts.The prompts instruct the agent to produce executable notebook code, prioritize predefined tools for matching functionality, and avoid duplicating completed-task code; the one-shot prompt also says to copy DataFrames before processing.Source: §C.1.3 · pp. 17–19

C.1.2 TOOLS DETAILS

• The paper lists Data Interpreter’s tools in Table 5.The supplied passage and table caption identify the table as a listing of the framework’s tools, without providing their contents here.Source: Table 5 · p. 18

D.1 DATASET

• 257 InfiAgent-DABench problems cover seven data-analysis areas and combinations, with varying difficulty levels.The areas include summary statistics, feature engineering, correlation analysis, machine learning, distribution analysis, outlier detection, and comprehensive data preprocessing.Source: §D.1 · p. 19
• An easy InfiAgent-DABench example asks for mean and standard deviation directly from all values in a column, without removing outliers or modifying data.The output values are rounded to two decimal places.Source: §D.1 · p. 19
• Medium and hard examples combine feature engineering with correlation analysis, and missing-value imputation with before-and-after distribution analysis.The medium example creates FamilySize from SibSp and Parch and calculates its Pearson correlation with Survived; the hard example uses mean imputation and compares distribution statistics.Source: §D.1 · p. 20
• Eight ML-Benchmark tasks span three difficulty levels; each includes data, requirements, suggested steps, and metrics, while toy tasks require framework-led data splitting.The levels range from easy (level 1) to most complex (level 3).Source: §D.1 · p. 20
• 20 open-ended benchmark tasks test understanding user needs, decomposing complex tasks, and executing code across eight real-world task types.These include OCR, web search and crawling, automated email replies, web page imitation, text-to-image, image-to-HTML, background removal, and mini-game generation.Source: §D.1 · p. 20
• 12,500 MATH problems include 5,000 test problems across subjects and difficulty levels; evaluation follows Wu et al. (2023c) on four types, excluding level-5 geometry.The evaluated types are Counting and Probability, Number Theory, Prealgebra, and Precalculus; the dataset is attributed to Hendrycks et al. (2021).Source: §D.1 · p. 20

D.2 EVALUATION METRICS

• MATH benchmark performance is evaluated using accuracy, following the setting proposed by Wu et al. (2023c).Source: §D.2 · p. 20
• The ML-Benchmark reports completion rate (CR), normalized performance score (NPS), and comprehensive score (CS).Source: §D.2 · p. 21
• CR divides the sum of task-step scores by the maximum possible total, with missing or failed steps scoring 0, non-compliant successes 1, compliant successes 2, and optional steps excluded.Each step has a maximum score of 2 and a minimum of 0.Source: §D.2 · p. 21
• NPS uses each task’s evaluation metric, including accuracy, F1, AUC, or RMSLE, and reports raw accuracy, F1, and AUC values for comparisons across identical data tasks.Source: §D.2 · p. 21
• For loss-based metrics such as RMSLE, normalization scales values from 0 to 1, with higher NPS indicating better performance.Source: §D.2 · p. 21
• CS is a weighted sum of CR and NPS, assessing both task-requirement completion and generated-model performance.Source: §D.2 · p. 21
• For open-ended tasks, the paper sets NPS to 0 and equates CS with CR because unified performance standards are lacking.Source: §D.2 · p. 21

• Table 6 presents ML-benchmark Completion Rate and Normalized Performance Score results, while Table 8 presents ablation results for completion rate (CR) and normalized performance score (NPS).Source: §D.3.1 · p. 21

D.3.2 ABLATION STUDY

• The core-module ablation evaluates IGR and PNG using CR, NPS, and CS on the ML-Benchmark.The supplied passages describe the evaluation scope but do not provide ablation scores.Source: §D.3.2 · p. 21, Table 8 · p. 22

E.1 AN EXAMPLE OF TASK GRAPH

• The example starts with data exploration, then branches into correlation analysis and anomaly detection, with both tasks depending on exploration.The example concerns sensor readings used to predict whether industrial machines are normal or faulty.Source: §E.1 · p. 23
• The task graph lists feature engineering as task 4, dependent on tasks 2 and 3; task 2 performs correlation analysis and causal inferences, while task 3 implements anomaly detection.Source: §E.1 · p. 23
• After feature engineering, the graph develops a predictive model and evaluates its performance using eval_set.csv.Model development depends on task 4, and evaluation depends on model development.Source: §E.1 · p. 23
• A final task visualizes the analysis and prediction results, depending on both model development and evaluation.The example’s user requirement requests high-quality graphs.Source: §E.1 · p. 23

E.2 RUNTIME RESULTS OF TASK GRAPH

• Three runtime results demonstrate the task graph, action graph, and overall graph structure, as shown in Figure 7.Source: §E.2 · p. 23

E.3 ADDITIONAL RESULTS OF OPEN-ENDED TASKS

• The Data Interpreter’s results for open-ended tasks are organized across two figures: tasks 8, 9, 10, and 13 appear in Figure 8, while tasks 4, 14, and 15 appear in Figure 9.Source: §E.3 · p. 23
• Figure 7 provides runtime examples of Data Interpreter for machine learning, webpage imitation, and math problem solving.Source: Figure 7 · p. 24

E.4 RESULT OF DATA VISUALIZATION

• Figure 10 illustrates Data Interpreter’s results for data analysis and visualization.Source: §E.4 · p. 24, Figure 10 · p. 25
• Figure 8 shows webpage imitation by Data Interpreter.Source: Figure 8 · p. 24
• Figure 9 shows image background removal, text-to-image, and web search and crawling by Data Interpreter.Source: Figure 9 · p. 25

F.1 OPEN-ENDED TASK DETAILS

• The typical open-ended task illustrations include the necessary data and user requirements for each task.Source: §F.1 · p. 26
• Figures 11 to 14 also show the assessment pipeline for each illustrated task.Source: §F.1 · p. 26

• The ML-Benchmark dataset collects several typical datasets from Kaggle and machine learning.Source: §F.2 · p. 26
• Table 9 lists each dataset’s name, description, and standard user requirements.Source: Table 9 · p. 29
• Table 9 also reports dataset type, task type, difficulty, and the metric used.Source: Table 9 · p. 29