Source-linked AI summary
LongDS-Bench: On the Failure of Long-Horizon Agentic Data Analysis
Kewei Xu, Xiaoben Lu, Shuofei Qiao, Zihan Ding, Haoming Xu, Lei Liang, Ningyu Zhang
TL;DR
Existing benchmarks provide limited evaluation of agents managing evolving analytical states across long dependency chains. LongDS formulates this challenge as long-horizon analytical state management and evaluates agents on a realistic multi-turn benchmark, finding that performance degrades sharply and failures are dominated by state-management and cascading errors.
Problem
Existing data-analysis benchmarks provide limited evaluation of how agents update, perturb, roll back, and compose analytical states across long dependency chains.
Method
LongDS formulates long-horizon agentic data analysis as analytical state management and introduces a benchmark of 68 real-world workflow tasks spanning 2,225 turns with persistent environment states.
Results
Across five state-of-the-art models, even the best remains below 50% average accuracy, performance degrades sharply over longer interactions, and failures are dominated by long-horizon errors.
Takeaways & Limitations
Additional agent steps do not consistently improve accuracy, indicating that analytical state maintenance rather than interaction budget is the main limitation.
Takeaways & Limitations
LongDS is built from public Kaggle notebooks and datasets, which may not fully cover proprietary or production data-analysis scenarios and produces an imbalanced domain distribution.
Abstract
from arXiv · showhide
Real-world data analysis is inherently iterative, yet existing benchmarks mostly evaluate isolated or short interactive tasks, leaving agents' ability to track evolving analytical context over long horizons untested. We introduce LongDS, a benchmark for long-horizon, multi-turn data analysis where agents must maintain, update, restore, and compose evolving analytical states. LongDS comprises 68 tasks constructed from real-world Kaggle notebooks, spanning 2,225 turns across six domains including Geoscience, Business, and Education. Tasks are designed around state-evolution patterns (e.g., counterfactual perturbation, rollback, multi-state composition), with an average dependency span of 11.3 turns. Evaluating five state-of-the-art models, we find that the best model reaches only 48.45% average accuracy, performance drops nearly 47 points from early to late turns, and long-horizon errors account for 52%--69% of failures. Further analysis shows that additional agent steps do not necessarily improve performance, suggesting that the key bottleneck is maintaining a correct analytical state rather than increasing interaction budget. We release LongDS to support research on reliable long-horizon agentic data analysis. Code and data will be released at https://github.com/zjunlp/DataMind.
1 Introduction
Existing benchmarks largely miss whether agents can manage evolving analytical states across long dependency chains. LongDS addresses this gap with realistic multi-turn tasks and finds substantial degradation and state-management failures in current models.
- Existing benchmarks leave open whether agents can update, perturb, roll back, and compose analytical states across long dependency chains.
- 68 tasks and 2,225 turns across six domains form LongDS, with later requests depending on prior analytical states and an average dependency span of 11.3 turns.
- LongDS frames long-horizon agentic data analysis as analytical state management, including construction, inheritance, updates, counterfactual perturbations, rollbacks, and multi-state composition.
2 Preliminary
A multi-turn data analysis task unfolds in a persistent executable environment where each response depends on the current request, accumulated history, and evolving environment state. This setting captures analytical states that persist, change, and may later be restored across long horizons.
- A task consists of user requests over data files executed in a persistent environment such as a Jupyter notebook.
- The task specification includes data files, an initial executable environment, and a sequence of user requests.
- At each turn, the agent uses the current request, interaction history, and prior environment state to produce a response and updated environment state.The environment is not reset, so intermediate code states and results persist between turns.
- The target response depends on prior scopes, definitions, assumptions, and other analytical context accumulated in the interaction history.
- Later requests may depend on analytical states that are inherited, revised, temporarily perturbed, or restored from earlier turns.
3 LongDS Benchmark
LongDS is a curated benchmark for testing agents' management of evolving analytical states in realistic, persistent, long-horizon data-analysis workflows. It contains 68 tasks and 2,225 turns across six domains, with diverse state transformations, long dependencies, executable references, and validated evaluation.
- 3 LongDS Benchmark: LongDS evaluates evolving analytical states involving scopes, definitions, assumptions, and intermediate results across long-horizon interactions.Its state-evolution patterns include updates, local counterfactual perturbations, rollbacks, and multi-state composition.
- 3.1 Design Principles: LongDS is constructed from real-world notebooks through source filtering, state-annotated task construction, and three-stage refinement and validation.The pipeline includes expert review, Codex-based consistency checking, and final alignment checks.
- 3.2 Task Curation: 68 tasks span six application domains and comprise 2,225 turns, averaging 33 turns per task.The domains are Sports, Geoscience, Business, Social Good, Education, and Community.
- 3.2 Task Curation: Each task averages 5.8 rollback turns and 8.6 multi-state composition turns, with 2.9 dependencies per turn and an 11.3-turn average dependency span.These statistics characterize frequent state evolution and long-range dependency structure.
- 3.4 Evaluation Protocol: Each turn has executable reference code and a structured reference answer, while a judge model assesses semantic and numerical consistency with tolerance for training-related nondeterminism.The score is binary at the turn level and aggregated as a macro-average of task-level scores.
- 3.4 Evaluation Protocol: 93.11% agreement and Cohen’s κ of 0.8623 were found between human and LLM judgments in a blind audit.
4 Experiments
LongDS evaluates five state-of-the-art models across diverse long-horizon data-analysis tasks and finds substantial performance degradation as state tracking becomes more demanding.
- Experimental Setup: LongDS evaluates five state-of-the-art LLMs on long-horizon data-analysis tasks using a ReAct-style agent in a persistent Jupyter kernel.The evaluation allows up to 40 reasoning-action steps per turn and extracts final answers for automated semantic evaluation.
- Overall Performance: 48.45 average accuracy is achieved by Gemini-3.1-Pro, the best-performing model across the six-domain benchmark.GPT-5.4 and Claude-4.6-Sonnet follow with average scores of 43.50 and 41.56, respectively.
- Domain Variance: Performance varies across domains, with models scoring highest in Education and struggling in Geoscience, Business, and Sports.Relative rankings also shift across domains, so no model consistently maintains and applies analytical state across varying complexities.
- Degradation in Long-Horizon State Tracking: Nearly 47 percentage points separate accuracy in the first and last 10% of task progress, while broader dependencies and more complex state transitions further reduce performance.The decline is reported after normalizing turn positions within each task and is especially pronounced for Update, Counterfactual, and Rollback requests.
5 Deep Analysis
Deep analysis indicates that long-horizon failures arise primarily from maintaining analytical state, not simply from limited interaction effort, and that resetting state trades recovery against continuity.
- Efficiency and Performance Trade-off: More interaction budget does not necessarily improve long-horizon performance: Claude-4.6-Sonnet uses the most steps without achieving the highest accuracy.GPT-5.4 obtains the best cost-normalized efficiency because it uses fewer steps and trajectory tokens.
- Efficiency and Performance Trade-off: Correct analytical-state maintenance is more important than simply extending analysis, because additional steps may introduce state drift.Domain-level results show GPT-5.4 is most efficient in most domains, while Gemini-3.1-Pro leads in Geoscience and Sports.
- Error Analysis: Long-horizon errors constitute 52% to 69% of failures across models, with Cascade Error the largest category.State Management Error also contributes substantially, whereas Context Memory Error occurs less frequently.
- Agentic Behavior Decreases over Long Trajectories: Average agent steps per user turn decrease by 4.3 steps from early to late stages, reducing exploration, verification, and iterative refinement.The reduced exploration may make early state errors harder to detect and correct, contributing to downstream cascading failures.
- Persistent State and Reset Effects: Resetting the code environment slightly improves low- and medium-baseline cases but substantially hurts high-baseline cases.Reset trades recovery from corrupted states against preservation of useful accumulated analytical state.
6 Related Work
Prior benchmarks have advanced from isolated coding toward interactive and agentic analysis, but long-horizon management of evolving analytical state remains underexplored.
- Data Analysis Benchmarks and Agents: Existing data-science benchmarks mainly emphasize standalone coding, task completion, or workflow automation.These settings leave long-horizon analytical state management underexplored.
- Long-Horizon and Multi-Turn Agent Evaluation: Recent multi-turn and long-horizon evaluations study dynamic interaction, tool use, web or API operations, and extended workflow completion.They reveal degradation under sustained interaction and long-range consistency requirements, but do not specifically resolve the data-analysis state-management gap.
7 Conclusion
LongDS makes long-horizon analytical state management an explicit benchmark target and shows that current models still struggle as trajectories lengthen.
- Conclusion: LongDS evaluates whether agents can maintain, update, restore, and compose evolving analytical states across extended data-analysis interactions.The benchmark targets stateful environments rather than isolated requests.
- Conclusion: Current proprietary and open-source models show degrading performance over longer trajectories, with failures dominated by cascading and state-management errors.LongDS therefore provides a challenging testbed for developing more reliable data-analysis agents.
Limitations
LongDS has scope and construction limitations: its public-source composition may not represent proprietary or production analysis, and its task design only partially covers several analytical settings.
- Public Kaggle notebooks and datasets may not fully cover proprietary or production data-analysis scenarios.
- The benchmark has an imbalanced domain distribution, especially in Sports, where many candidate notebooks were filtered out because of large datasets or long-running computations.
- Quantitatively verifiable questions support reliable evaluation but only partially cover open-ended insight generation, visualization-heavy analysis, and presentation-oriented analytics.
- Semi-automated task construction with Codex-assisted generation and expert-guided refinement may preserve biases from source notebooks or the construction process.
Ethics Statement
The authors report that the benchmark follows applicable dataset, notebook, framework, competition, and usage restrictions and does not introduce new personal-data collection.
- The datasets, notebooks, and DSGYM framework are governed by their respective licenses, competition rules, and usage restrictions, which the authors state they comply with.
- The benchmark construction does not collect new personal data, and its tasks do not involve private or sensitive personal information.
- The authors foresee no substantial ethical or societal concerns arising from the work.
A Details of Benchmark
LongDS defines benchmark state continuity and situates its construction and evaluation within broader data-analysis and long-horizon agent research.
- Benchmark state design: State inheritance is the default continuity assumption rather than a separately annotated category in benchmark statistics.
- Benchmark state design: Table 1 summarizes LongDS state-evolution patterns, while Figure 7 shows those patterns and Table 3 reports task-level macro statistics.
- Benchmark sources: Table 8 documents the Kaggle competitions, public datasets, and notebooks used to construct LongDS, including task identifiers, notebook titles, and URLs for reproducible provenance.
- Related benchmarks: Prior benchmark work covers data-science coding, analytical reasoning, experimentation, agentic workflows, long-document analysis, spreadsheets, multimodal workflows, and cross-modal analysis.
- Related benchmarks: Interactive data-analysis benchmarks evaluate multi-round interaction, guided analysis, and analyst-agent collaboration.
- Data-analysis agents and systems: Data-analysis systems automate exploration, spreadsheet manipulation, visualization, notebook-centered analysis, data-science pipelines, and competition-style workflows, with later methods emphasizing planning, verification, supervision, and heterogeneous analytics.
- Long-horizon agent evaluation: General agent benchmarks study multi-turn interaction, tool use, web navigation, office workflows, API use, and long-horizon task completion.
- Evaluation setup: The DSGYM data-analysis agent follows a unified structured system prompt regulating its tool-use and reasoning format.
C.2 Codex Results on a Sampled Subset
Codex is evaluated on a domain-stratified sampled subset rather than the full benchmark; it performs best there, while long-horizon state management remains challenging.
- Codex is evaluated on a domain-stratified subset containing two tasks from each of LongDS’s six domains because turn-by-turn manual interaction is operationally costly.
- 65.55 is Codex’s highest average score on the sampled subset, where it leads in four of six domains.
- Claude-4.6-Sonnet and Gemini-3.1-Pro lead on Business and Geoscience, respectively, while Codex performs strongly in Education, Social Good, and Sports.
- The sampled-subset results suggest that a stronger code-centric agent can improve performance on some LongDS tasks, but long-horizon analytical state management remains challenging.
C.3 Error Analysis Details
The error analysis combines automated labeling with human validation and examines how dependency span and environment resets affect long-horizon performance. Human studies indicate generally reproducible error annotations and reliable turn-level correctness judgments.
- Error annotation: 3,207 incorrect turns are annotated by Codex across sampled tasks from the evaluated models, forming the error pool for subsequent analysis and validation.The sampled evaluation covers six domains, with three Sports tasks because only three are available.
- Human validation: 81.50% agreement and Cohen’s κ of 0.7535 occur in blind relabeling, while reference verification reaches 89.00% agreement, κ of 0.8715, and macro-F1 of 0.8898.The lower blind-relabeling macro-F1 indicates that some boundary cases remain difficult to distinguish.
- Dependency analysis: Accuracy declines as dependency span increases, showing that recovering and applying the correct analytical state becomes harder when the farthest dependency lies farther back.Dependency span measures the maximum distance to the farthest depended prior turn, complementing dependency breadth.
- Reset experiment: The reset experiment clears accumulated code state, variables, and intermediate results while retaining interaction history, enabling comparison with persistent execution on shared post-reset turns.Reset locations are selected from candidate turns 2, 4, 6, and 15 using a task-specific heuristic.
D Details of Evaluation
The evaluation details define automated and human-audited correctness assessment for LongDS. The protocol uses question-driven, exact factual checking with binary scoring and reports strong agreement between human judgments and the automated evaluator.
- Human audit: 450 cases are balanced across six domains, five models, and automated judge labels, then independently reviewed by three human annotators.The audit includes 225 cases judged correct and 225 judged incorrect by the automated evaluator.
- Human audit: 93.11% overall agreement, Cohen’s κ of 0.8623, and macro-F1 of 0.9311 indicate high agreement between blind human judgments and LLM-as-judge scores.The audit evaluates whether model responses correctly answer the task question using the question, ground-truth answer, and response.
- Agent protocol: The prompt requires continuous code execution, one analytical step per turn, and final responses enclosed in an answer tag.Variables and data loaded in previous steps remain available for subsequent analysis.
- Correctness criteria: The evaluator scores only information directly requested by the question, requires all required fields to be correct, and uses binary scoring.Extra information is ignored, but missing required fields count as incorrect.
- Correctness criteria: Numeric answers must match exactly after insignificant trailing-zero normalization, except when an explicitly specified tolerance applies.The protocol does not permit rounding or an unstated tolerance.