Source-linked AI summary
Data Interpreter: An LLM Agent For Data Science
Sirui Hong, Yizhang Lin, Bang Liu, Bangbang Liu, Binhao Wu, Ceyao Zhang, Chenxing Wei, Danyang Li, Jiaqi Chen, Jiayi Zhang, Jinlin Wang, Li Zhang, Lingyao Zhang, Min Yang, Mingchen Zhuge, Taicheng Guo, Tuo Zhou, Wei Tao, Xiangru Tang, Xiangtao Lu, Xiawu Zheng, Xinbing Liang, Yaying Fei, Yuheng Cheng, Zhibin Gou, Zongze Xu, Chenglin Wu
TL;DR
Data science agents must handle long-term interconnected workflows and changing data, but prior methods mainly target individual tasks. Data Interpreter uses hierarchical graph modeling with programmable node generation to decompose, refine, and verify work dynamically. Across data science, machine learning, and mathematical benchmarks, it reports consistent performance improvements, while evaluation remains limited to relatively small, entry-level datasets.
Problem
Prior approaches primarily focus on individual tasks and struggle to evaluate complete workflows or adapt to real-time changes in intermediate data and task dependencies.
Method
Data Interpreter combines hierarchical graph modeling with programmable node generation to decompose workflows and dynamically generate, refine, verify, and optimize subproblems.
Results
Data Interpreter consistently outperforms existing methods across benchmarks, including a 25% performance boost on InfiAgent-DABench and a 26% improvement on MATH.
Takeaways & Limitations
The framework supports complex multi-step data science, machine learning, mathematical, and real-world application tasks through adaptive workflow management.
Takeaways & Limitations
Evaluation is limited to relatively small, entry-level Kaggle datasets and does not yet cover large-scale data or complex tasks such as time series analysis, multi-label classification, or multitable problems.
Abstract
from arXiv · showhide
Large Language Model (LLM)-based agents have shown effectiveness across many applications. However, their use in data science scenarios requiring solving long-term interconnected tasks, dynamic data adjustments and domain expertise remains challenging. Previous approaches primarily focus on individual tasks, making it difficult to assess the complete data science workflow. Moreover, they struggle to handle real-time changes in intermediate data and fail to adapt dynamically to evolving task dependencies inherent to data science problems. In this paper, we present Data Interpreter, an LLM-based agent designed to automatically solve various data science problems end-to-end. Our Data Interpreter incorporates two key modules: 1) Hierarchical Graph Modeling, which breaks down complex problems into manageable subproblems, enabling dynamic node generation and graph optimization; and 2) Programmable Node Generation, a technique that refines and verifies each subproblem to iteratively improve code generation results and robustness. Extensive experiments consistently demonstrate the superiority of Data Interpreter. On InfiAgent-DABench, it achieves a 25% performance boost, raising accuracy from 75.9% to 94.9%. For machine learning and open-ended tasks, it improves performance from 88% to 95%, and from 60% to 97%, respectively. Moreover, on the MATH dataset, Data Interpreter achieves remarkable performance with a 26% improvement compared to state-of-the-art baselines. The code is available at https://github.com/geekan/MetaGPT.
1 INTRODUCTION
Data science workflows involve interconnected tasks that require iterative refinement and real-time adaptation, while prior LLM approaches largely target isolated tasks. Data Interpreter addresses this gap with hierarchical graph modeling and programmable node generation, and reports stronger benchmark performance.
- Data science spans data gathering, model building, and decision-making across computer science, statistics, visualization, and mathematics.
- Interconnected workflows require iterative refinements and real-time adjustments as data and requirements evolve.
- Prior LLM approaches focus mainly on individual tasks or fixed pipelines, limiting holistic evaluation and adaptation to changing intermediate data and task dependencies.
- Data Interpreter represents workflows as hierarchical graphs whose nodes are tasks and whose edges encode dependencies.
- Programmable Node Generation generates, refines, and verifies graph nodes in real time to improve subproblem execution robustness and precision.
- 25% performance boost on InfiAgent-DABench and 26% improvement on MATH demonstrate reported gains over existing methods across benchmarks.
2 RELATED WORK
Related work advances LLM agents through code generation, tools, and structured planning. Graph-based planning improves task decomposition and connectivity, but the surveyed approaches are presented as broader agent-planning methods rather than a complete data science workflow solution.
- LLMs as Data Science Agents: LLM data science agents evolved from code generation toward code interpreters using function-calling for more flexible problem solving.
- Enhancing LLM with Tools: External tools, retrieval, automatic tool selection, and multi-agent systems extend LLM capabilities beyond static task execution.
- Graph-Based Planning for LLM Agents: Chain-of-thought, tree, and graph methods structure complex reasoning into subtasks, nodes, or text-transformation graphs.
- Graph-Based Planning for LLM Agents: Recent graph-based approaches refine prompts, optimize connectivity, or dynamically generate edges to coordinate agents and improve planning.
3 METHODOLOGY
Data Interpreter structures end-to-end data science workflows as hierarchical, executable graphs that can be refined using runtime feedback. Its graph-based planning and programmable node generation support dynamic task adjustment, verification, and context-aware tool use.
- Hierarchical Graph Modeling: Hierarchical Graph Modeling decomposes complex data science problems into manageable tasks and actions connected through dependencies.The framework represents subprocesses as graph nodes and their relationships as edges, forming a directed acyclic graph for the overall workflow.
- Hierarchical Graph Modeling: The task graph organizes project requirements into task nodes, while the action graph further decomposes each task into independently executable and verifiable actions.Task nodes include descriptions, types, statuses, execution feedback, and dependencies; action graphs provide a more granular representation of executable code.
- Iterative Graph Refinement: The action graph uses runtime feedback to refine code or add verification when execution fails or produces unsuccessful checks.A stateful graph executor manages execution and debugging, while the action graph generator regenerates code using execution results.
- Iterative Graph Refinement: Data Interpreter dynamically adjusts its task graph instead of relying on a static one-time plan, updating planning and code in response to task outcomes.Continuous monitoring and iterative updates allow tasks to be added, removed, or modified as execution proceeds.
- Programmable Node Generation: Programmable Node Generation enables context-aware, task-specific tool use and integrates domain-specific expertise into task execution.The approach supports real-time adaptation and optimization of tool usage during execution.
4 EXPERIMENTS
Data Interpreter is evaluated across data analysis, machine learning, open-ended, and mathematical benchmarks, including ablations and base-LLM comparisons. It generally outperforms baselines, while performance depends on context capacity and coding proficiency.
- InfiAgent-DABench: 25% improvement over direct LLM inference was achieved on InfiAgent-DABench with GPT-4o, whereas GPT-4-0613 reached 73.55 and remained below direct invocation.The GPT-4-0613 limitation was attributed to context overhead exceeding the model’s maximum window.
- ML-Benchmark: 0.95 comprehensive score across ML-Benchmark tasks exceeded AutoGen’s 0.86 and OpenDevin’s 0.88.Data Interpreter also completed all mandatory processes and operated at 27.9% of OpenDevin’s cost while averaging 0.84.
- Open-ended tasks: 0.97 completion rate on open-ended tasks improved over AutoGen by 110.8% and over OpenDevin by 61.7%.The system completed all steps in WPI, I2C, and T2I, where baseline frameworks did not.
- MATH: 26.5% relative improvement over AutoGen was obtained on average across tested level-5 MATH categories.Accuracy reached 0.82 in N.Theory and 0.29 in Precalc, with improvements of 0.16 and 0.17 over AutoGen, respectively.
- Ablation study: 10.6% further improvement from Programmable Node Generation raised the ML-Benchmark comprehensive score to 0.94 after iterative graph refinement improved performance by 0.48.The ablation compared ReAct, iterative graph refinement, and the full Data Interpreter configuration.
- Base LLM comparison: Smaller base LLMs handled data loading and analysis effectively but degraded on advanced coding and image or webpage tool tasks.The comparison evaluated Data Interpreter with GPT-4o and GPT-4o-mini and reported failures for smaller models using multimodal tools or interfaces.
5 CONCLUSION
Data Interpreter is an LLM-based agent for data science that uses hierarchical graph representation to handle complex, multi-step reasoning. Evaluations report improvements across machine learning, mathematical, and real-world applications.
- Data Interpreter tackles data science challenges through hierarchical graph representation.The framework combines hierarchical decomposition, fine-grained execution, validation, and iterative modifications.
- The framework continuously monitors data changes and adapts to dynamic environments through task refinement and graph optimization.
- Data Interpreter improves data analysis and machine learning performance while enhancing reasoning for complex multi-step tasks.
- Extensive evaluations show that Data Interpreter outperforms open-source frameworks across machine learning, mathematical, and real-world applications.
A LIMITATIONS
The reported evaluation scope is limited by relatively small, entry-level datasets and by risks associated with flexible tool integration. The framework has not yet been evaluated on several challenging data settings, while code-safety checks mitigate but do not remove security concerns.
- The machine-learning evaluation uses entry-level Kaggle datasets smaller than 500MB, with hundreds of columns and tens of thousands of rows.The datasets mainly involve classification and regression tasks and are benchmarked against junior data-scientist capabilities.
- Data Interpreter has not yet been evaluated on large-scale data, time series analysis, multi-label classification, or multitable problems.
- Flexible tool integration can introduce security vulnerabilities when users provide malicious code for unauthorized system penetration or web attacks.The experiments prompt the agent to check code before generating new code, and the paper recommends LLMs with robust safety policies.
- The framework’s tools are selected from task metadata and organized by combining multiple tools as needed.
- Tool use employs zero-shot prompts for open-ended tasks and one-shot prompts for machine-learning tasks.
D EXPERIMENT DETAILS
The experiments cover data analysis, machine learning, open-ended real-world tasks, and mathematical problem solving through several benchmarks with varied task types and difficulty levels.
- InfiAgent-DABench contains 257 data-analysis problems spanning seven areas and combinations, with varying difficulty levels.
- Benchmark examples include summary statistics, feature engineering with correlation analysis, and preprocessing followed by distribution analysis.
- ML-Benchmark includes eight representative machine-learning tasks across three difficulty levels.Tasks provide data, descriptions, requirements, suggested steps, and evaluation metrics; toy tasks require data splitting during modeling.
- The open-ended benchmark comprises 20 tasks requiring user-need understanding, task decomposition, and code execution for real-world needs.
- The MATH dataset contains 12,500 problems, including a 5,000-problem test set across subjects and difficulty levels.Evaluation covers four problem types while excluding level-5 geometry problems from the test set.
D.2 EVALUATION METRICS
The evaluation metrics measure task completion, normalized model performance, and their combined score, while MATH uses accuracy and open-ended tasks equate comprehensive score with completion rate.
- MATH evaluation uses accuracy as its evaluation metric.
- Completion rate scores how fully task steps are completed relative to their maximum possible scores.Statuses range from missing or failed to success-compliant, with optional steps excluded from scoring.
- Normalized performance score accommodates task-specific metrics such as accuracy, F1, AUC, and RMSLE.Loss-based metrics are transformed so higher normalized values indicate better performance.
- Comprehensive score combines completion rate and normalized performance score as a weighted sum.
- For open-ended tasks, normalized performance score is set to 0, making comprehensive score equal to completion rate.
D.3.1 ADDITIONAL RESULTS OF ML-BENCHMARK AND MATH DATASET
This section reports additional ML-Benchmark and MATH results, using completion rate and normalized performance score for ML-Benchmark.
- ML-Benchmark results are reported using Completion Rate and Normalized Performance Score metrics.
D.3.2 ABLATION STUDY
The ablation study examines core modules and illustrates Data Interpreter's workflow through a dependent task graph spanning analysis, modeling, evaluation, and visualization.
- The ablation study reports core-module results on ML-Benchmark.
- Table 8 evaluates Iterative Graph Refinement and Programmable Node Generation using completion rate, normalized performance score, and comprehensive score.
- The illustrated workflow links exploration to correlation analysis and anomaly detection, then combines those outputs for feature engineering and predictive modeling.
- Model evaluation depends on the predictive model, while visualization depends jointly on modeling and evaluation.
E.3 ADDITIONAL RESULTS OF OPEN-ENDED TASKS
Additional results demonstrate Data Interpreter on open-ended tasks and data analysis or visualization, with examples spanning web, image, and other workflows.
- Figure 7 presents runtime examples covering machine learning, webpage imitation, and math problem solving.
- Figure 10 illustrates Data Interpreter's data analysis and visualization results.
F.2 ML-BENCHMARK DATASET DESCRIPTION
The ML-Benchmark comprises varied datasets and task requirements spanning machine learning, data processing, web interaction, email, OCR, and image-related tasks.
- The ML-Benchmark dataset collects typical datasets from Kaggle and machine learning, with detailed metadata provided in Table 9.
- The benchmark includes OCR on an English invoice image to extract the total amount and save it as a table using PaddleOCR.
- The benchmark also contains open-ended task categories represented by figures for OCR, email reply, webpage imitation, image processing, text-to-image, image-to-code, and mini-game generation.
- Other tasks include filtering emails and sending domain-specific automatic replies, removing image backgrounds with rembg, and additional open-ended cases.
- Webpage imitation renders https://pytorch.org/ with Selenium and WebDriver, converts the image into HTML, CSS, and JavaScript, and saves the webpage.
- Table 9 describes dataset names, descriptions, user requirements, dataset types, task types, difficulty, and evaluation metrics.