Source-linked AI summary
InfiAgent-DABench: Evaluating Agents on Data Analysis Tasks
Xueyu Hu, Ziyu Zhao, Shuang Wei, Ziwei Chai, Qianli Ma, Guoyin Wang, Xuwu Wang, Jing Su, Jingjing Xu, Ming Zhu, Yao Cheng, Jianbo Yuan, Jiwei Li, Kun Kuang, Yang Yang, Hongxia Yang, Fei Wu
TL;DR
LLM-based agents lack a comprehensive benchmark for end-to-end data-analysis tasks, which require interaction with data and execution environments. The paper introduces InfiAgent-DABench with closed-form evaluation, benchmarks 34 LLMs, and develops DAAgent; DAAgent-34B performs 3.9% better than GPT-3.5 on data analysis tasks.
Problem
A comprehensive benchmark for evaluating LLM-based agents on challenging end-to-end data-analysis tasks is missing.
Method
The paper builds InfiAgent-DABench with 257 questions from 52 CSV files, an agent framework, and format-prompting that makes open-ended questions closed-form for automatic evaluation.
Results
DAAgent-34B achieves 3.9% better performance than GPT-3.5, while benchmarking 34 LLMs reveals continuing challenges on data-analysis tasks.
Takeaways & Limitations
The benchmark provides a way to assess current data-analysis agents and exposes their capabilities and limitations in this domain.
Takeaways & Limitations
The benchmark excludes visualization questions because producing pictorial outcomes makes closed-form evaluation difficult.
Abstract
from arXiv · showhide
In this paper, we introduce InfiAgent-DABench, the first benchmark specifically designed to evaluate LLM-based agents on data analysis tasks. These tasks require agents to end-to-end solving complex tasks by interacting with an execution environment. This benchmark contains DAEval, a dataset consisting of 257 data analysis questions derived from 52 CSV files, and an agent framework which incorporates LLMs to serve as data analysis agents for both serving and evaluation. Since data analysis questions are often open-ended and hard to evaluate without human supervision, we adopt a format-prompting technique to convert each question into a closed-form format so that they can be automatically evaluated. Our extensive benchmarking of 34 LLMs uncovers the current challenges encountered in data analysis tasks. In addition, building on top of our agent framework, we develop a specialized agent, DAAgent, which surpasses GPT-3.5 by 3.9% on DABench. Evaluation datasets and toolkits for InfiAgent-DABench are released at https://github.com/InfiAgent/InfiAgent .
1. Introduction
InfiAgent-DABench addresses the lack of comprehensive end-to-end benchmarks for data-analysis agents by combining realistic CSV tasks with automatic closed-form evaluation. It benchmarks current LLMs and introduces a specialized agent trained for data analysis.
- 1. Introduction: Data-analysis tasks are practically useful yet challenging because agents must coordinate natural-language reasoning, code, data files, and execution environments.The benchmark therefore evaluates agent behavior beyond completing isolated code snippets.
- 1. Introduction: The benchmark contains 257 questions associated with 52 CSV files and converts open-ended tasks into closed-form answers for automatic matching.GPT-4 generates questions, constraints, and format requirements from CSV descriptions and expert-identified concepts.
- 1. Introduction: InfiAgent-DABench is the first benchmark designed specifically to evaluate LLM-based agents on data analysis tasks.It targets end-to-end abilities including planning, code writing, Python-sandbox use, self-debugging, and conclusion generation.
- 1. Introduction: Benchmarking 34 state-of-the-art LLMs shows that current models still face challenges handling data-analysis tasks effectively.The evaluation is supported by a human-assessed dataset and an agent framework for performing the tasks.
- 1. Introduction: DAAgent-34B achieves 3.9% better performance than GPT-3.5 on data analysis tasks.DAAgent is developed using the DAInstruct instruction-tuning dataset and the authors' agent framework.
2. InfiAgent-DABench Benchmark
InfiAgent-DABench combines a closed-form data-analysis dataset with an agent framework for evaluating LLM-based agents. It uses human assessment and filtering to ensure dataset quality, while DAInstruct supports training the specialized DAAgent.
- DAEval Dataset: DAEval contains realistic CSV files and closed-form questions designed for straightforward, model-free assessment.The questions are generated from file descriptions and expert-identified data-analysis concepts.
- DAEval Dataset: Format prompting converts open-ended questions into uniquely answerable forms with strict constraints and parseable @answer name[answer] outputs.Human assessment and filtering are used to ensure the resulting answers are unique and easy to parse.
- Agent Framework: The benchmark framework lets ReAct-style agents interact with files and execute Python code in an isolated sandbox.This supports both online and locally deployed models while avoiding safety issues from code execution.
- Dataset Quality: 85% of samples passed all human-assessed quality metrics and were retained in the final DAEval dataset.Assessment covers file suitability, question reasonableness, value, restrictiveness, alignment, and label correctness.
- Dataset Quality: GPT-4-generated questions outperform human-made questions on restrictiveness, alignment, and difficulty, with slightly better reasonableness and value.The comparison indicates that DAEval questions are comparable in quality to human-made questions.
- Instruction-Tuning Dataset: DAInstruct collects data-analysis questions, agent response trajectories, and filtered high-quality samples to train the specialized DAAgent.Responses are generated through iterative reasoning, code writing, execution, and refinement in the agent framework.
3. Experiments
The experiments benchmark diverse LLMs and agent frameworks on DAEval, using reformatting to enable automatic answer extraction and comparison. Results show persistent challenges, while DAAgent and strong open-source models improve over relevant baselines.
- 3.1.1. MODELS: The benchmark evaluates proprietary, open-source general, open-source code, and agent-framework models.The study includes GPT-4 for XAgent and AutoGen, and Qwen-72B-Chat for Qwen-Agent.
- 3.2. Evaluation: The evaluation prompts models with files, questions, and constraints, then reformats outputs before regular-expression extraction and label comparison.Ablation studies and manual checks examine the reformatting step.
- 3.3. Findings: 78.99% accuracy makes GPT-4 the best benchmarked model, while Qwen-72B-Chat trails it by 19%.The gap between proprietary and open-source models indicates substantial room for improvement in open-source data-analysis capability.
- 3.3. Findings: 59.92% makes Qwen-72B-Chat the strongest open-source model, 0.78% below GPT-3.5 and 3.50% above Gemini-Pro.
- 3.3. Findings: DAAgent-34B exceeds GPT-3.5 by 3.89% despite having fewer parameters.Comparisons with similarly sized Code Llama models show substantial gains from instruction-tuning on DAInstruct.
4. Related works
Prior work evaluates agents in web, tool-use, and code settings, but these benchmarks do not comprehensively assess end-to-end data-analysis agents. InfiAgent-DABench addresses this gap with broader data-analysis coverage and closed-form correctness evaluation.
- Existing benchmarks cover web environments, tool use, and code understanding or generation, but not comprehensive data-analysis agents.
- Unlike code-completion benchmarks, InfiAgent-DABench targets agents that must reason, plan, use tools, and self-debug across data-analysis tasks.
5. Conclusion
The paper introduces InfiAgent-DABench as a benchmark for LLM-based agents performing data analysis and develops DAAgent using DAInstruct. Together, these contributions provide evaluation and training resources while exposing current model capabilities and limitations.
- InfiAgent-DABench is the first benchmark specifically designed to evaluate LLM-based agents on data-analysis tasks.
- DAAgent is a specialized data-analysis agent trained on DAInstruct, an instruction-tuning dataset generated by GPT-4.
- The released benchmark and agent resources provide insights into current data-analysis capabilities and limitations.
A. Limitations and Future work
The benchmark excludes visualization because closed-form evaluation is difficult for pictorial outputs. The related-work material contrasts code and multimodal evaluation alternatives, while appendices document dataset-generation procedures and interview processing.
- A. Limitations and Future work: Visualization questions are excluded because their pictorial outcomes are difficult to convert into closed-form evaluation.
- A. Limitations and Future work: Code-only evaluation may be inaccurate because data analysis ultimately seeks accurate visual representations, not merely correct code.
- A. Limitations and Future work: Multimodal evaluation assesses visual outputs directly but remains costly and suboptimal for data-visualization assessment.
- A. Limitations and Future work: The paper leaves precise visualization evaluation for future research because neither proposed alternative matches closed-form assessment’s accuracy and convenience.
- B.2. Benchmarks for Code: Related benchmarks include code-generation datasets, but InfiAgent-DABench covers multiple data-analysis concepts and uses closed-form correctness without external evaluators.
- Appendices: The appendices describe prompts for description, question, constraint, and format generation, alongside expert-derived concepts and GPT-3.5-assisted interview summarization.
H. Detailed Information about Pilot Experiments
The pilot experiments assess generated and human-made questions and answers for solution quality, while also comparing question difficulty across single- and multiple-concept items.
- Examples: Table 13 illustrates generated Titanic questions containing one or more subquestions alongside question-specific constraints.The example file records passenger information such as survival status, class, age, and embarkation point.
- Quality Assessment: Three experts assess every DAEval sample’s files, questions, and labels using decomposed yes-or-no criteria, retaining only samples positive on all metrics.The assessment reports metric-level and overall positivity percentages before filtering.
- Difficulty Assessment: Difficulty assessment uses 216 anonymous pairs, separating single-concept from multiple-concept questions for expert comparison.The paired questions are sampled from human-made and GPT-4-generated questions.
J.2. Bad Case Analysis
Bad-case analysis identifies dataset samples whose questions or constraints fail alignment or restrictiveness requirements, alongside the human-authored comparison set used in assessment.
- Bad Cases: Two assessed failures show that constant target values can make a classifier question meaningless, while unspecified dates violate the requirement that constraints be sufficiently restrictive.The cases respectively fail Alignment and Restrictiveness.
- Assessment Set: Human experts contribute 54 questions across 9 CSV files, forming a 102-question assessment set when combined with GPT-4-generated questions on the same files.Each file contributes three single-concept and three multiple-concept human-written questions.
K. Method for Assessing Difficulty
The paper assesses question difficulty through repeated GPT-4 rankings, while constructing DAInstruct from iterative agent-framework conversations that generate and refine executable solutions.
- Difficulty Assessment: GPT-4 ranks 80 groups of five questions 100 times each using data-processing complexity, domain knowledge, and coding sophistication as difficulty criteria.Each question receives a score based on its ranking within the group.
- Instruction Data Collection: DAInstruct contains 5,131 data samples involving 631 CSV files, with ReAct conversations averaging 5.18 in length.
- Instruction Data Collection: GPT-4 constructs instruction data by iteratively proposing code, executing it, receiving results, and refining responses until it judges the solution conclusive.Prompt design directs the process and explicitly discourages fictitious execution results.
- Instruction Data Collection: A case study presents the same iterative dialogue pattern, with GPT-4 posing or answering questions and executing generated code until reaching a conclusive solution.
P. Implementation Details for Evaluation
Evaluation uses exact, automatically extracted answers and fixed sampling settings, with a separate reformatting step tested because models often violate the required output format.
- Evaluation Metric: Accuracy is the proportion of questions whose subquestions are all correct, evaluated by regular-expression extraction followed by exact matching.The experiments use temperature 0.2, top p 1.0, nucleus sampling, and frequency penalty 0.0.
- Prompting: Abab5.5-Chat receives an added instruction to output “I now know the final answer” before its final answer so it satisfies the framework format.
- Reformatting: A reformatting ablation tests whether post-processing is necessary for models that struggle to follow closed-form output requirements.Manual checks are used to ensure reformatting does not introduce external evaluation errors.
Q.1. Ablation Studies
The ablation study compares evaluation with and without GPT-3.5 reformatting and checks whether reformatting introduces additional errors. It reports the experimental setup and supporting prompt materials, but the supplied passages do not provide the ablation results themselves.
- Q.1. Ablation Studies: The ablation compares direct format-constrained prompting with a two-stage process that uses GPT-3.5 to reformat model responses.The comparison includes GPT-4, Qwen-72B-Chat, Qwen-14B-Chat, and Mixtral-7B-Instructv0.2, with GPT-4 as evaluator.
- Q.1. Ablation Studies: Manual inspection of 50 sampled responses found that GPT-3.5 reformatted every response into the required format without usually affecting evaluation results.The samples included both original and reformatted responses, and the authors checked whether reformatting introduced extra errors.
- Q.1. Ablation Studies: The evaluation setup uses prompts, format requirements, and tool-interaction instructions to standardize model answers for automatic assessment.The supplied materials include prompts for evaluation and reformatting, examples using a Python code sandbox, and structured answer formats for statistical outputs and outlier counts.