PaperlayerView exact revision on arXiv

Source-linked AI summary

The Tool Decathlon: Benchmarking Language Agents for Diverse, Realistic, and Long-Horizon Task Execution

Junlong Li, Wenshuo Zhao, Jian Zhao, Weihao Zeng, Haoze Wu, Xiaochen Wang, Rui Ge, Yuxuan Cao, Yuzhen Huang, Wei Liu, Junteng Liu, Zhaochen Su, Yiyang Guo, Fan Zhou, Lueyang Zhang, Juan Michelini, Xingyao Wang, Xiang Yue, Shuyan Zhou, Graham Neubig, Junxian He

arXiv:2510.25726v2cs.CLcs.AI

TL;DR

Existing language-agent benchmarks provide limited evidence about performance on diverse, realistic, long-horizon workflows. TOOLATHLON addresses this gap with a broad benchmark of real tools, initialized environments, fuzzy multi-application tasks, and deterministic evaluation, revealing substantial shortcomings in current models.

  • Problem

    Existing benchmarks often cover narrow domains or simplified tasks, limiting evaluation of agents on diverse, realistic, long-horizon workflows.

  • Method

    TOOLATHLON evaluates 108 realistic tasks spanning 32 applications and 604 tools, using initialized environments, fuzzy instructions, and deterministic execution-based evaluation.

  • Results

    Claude-4.5-Sonnet achieves the highest success rate at 38.6%, while DeepSeek-V3.2-Exp reaches 20.1% as the best open-source model.

  • Takeaways & Limitations

    The evaluation identifies long-context handling, tool-calling errors, and execution robustness as key challenges for practical language agents.

Abstract

from arXiv · show

Real-world language agents must handle complex, multi-step workflows across diverse Apps. For instance, an agent may manage emails by coordinating with calendars and file systems, or monitor a production database to detect anomalies and generate reports following an operating manual. However, existing language agent benchmarks often focus on narrow domains or simplified tasks that lack the diversity, realism, and long-horizon complexity required to evaluate agents' real-world performance. To address this gap, we introduce the Tool Decathlon (dubbed as Toolathlon), a benchmark for language agents offering diverse Apps and tools, realistic environment setup, and reliable execution-based evaluation. Toolathlon spans 32 software applications and 604 tools, ranging from everyday platforms such as Google Calendar and Notion to professional ones like WooCommerce, Kubernetes, and BigQuery. Most of the tools are based on a high-quality set of Model Context Protocol (MCP) servers that we may have revised or implemented ourselves. Unlike prior works, which primarily ensure functional realism but offer limited environment state diversity, we provide realistic initial environment states from real software, such as Canvas courses with dozens of students or real financial spreadsheets. This benchmark includes 108 manually sourced or crafted tasks in total, requiring interacting with multiple Apps over around 20 turns on average to complete. Each task is strictly verifiable through dedicated evaluation scripts. Comprehensive evaluation of SOTA models highlights their significant shortcomings: the best-performing model, Claude-4.5-Sonnet, achieves only a 38.6% success rate with 20.2 tool calling turns on average, while the top open-weights model DeepSeek-V3.2-Exp reaches 20.1%. We expect Toolathlon to drive the development of more capable language agents for real-world, long-horizon task execution.

1 INTRODUCTION

TOOLATHLON addresses the gap between narrow, simplified agent benchmarks and real-world workflows requiring diverse applications, realistic states, and long interactions. It evaluates 108 such tasks with deterministic scripts and shows substantial weaknesses in current models.

  • Motivation: Existing language-agent benchmarks cover limited domains and tools, whereas real workflows require switching across applications and handling complex environment states over long trajectories.Examples include coordinating databases, manuals, files, and email to resolve operational tasks.
  • Benchmark: TOOLATHLON spans 32 real-world applications and 604 tools across 108 tasks, with mostly multi-application scenarios grounded in realistic settings.Its domains range from daily affairs and education to technology and finance, and most tools come from MCP servers.
  • Evaluation: Each task is fully verifiable through a dedicated deterministic evaluation script that compares outcomes with static or dynamically generated ground-truth states.Dynamic examples include current NVIDIA shareholder information and real-time train schedules.
  • Environment realism: TOOLATHLON uses representative applications but supplements remote software with locally containerized open-source systems when realistic state setup is difficult.A Canvas course with dozens of students is one example of a state that is difficult to repeatedly recreate remotely.
  • Benchmark positioning: Compared with concurrent MCP benchmarks, TOOLATHLON emphasizes broader domains, cross-application tasks, realistic states, and execution-based verification rather than simplified or judge-scored interactions.MCPUniverse, for example, has six domains, 90% single-app tasks, synthetic states, and interactions shorter than eight turns.
  • Findings: 38.6% accuracy is achieved by Claude-4.5-Sonnet on TOOLATHLON, while DeepSeek-V3.2-Exp reaches a 20.1% success rate among open-source models.The benchmark is evaluated in isolated containers, enabling parallel execution; Claude-4.5-Sonnet completes all 108 tasks in about 70 minutes with 10 processes.

2 THE TOOLATHLON ENVIRONMENT AND EVALUATION FRAMEWORK

TOOLATHLON models agent tasks as interactions with partially observable, realistic environments and evaluates them through executed tool actions and resulting states. Its framework combines MCP-based tools, initialized environments, robust action-loop handling, and isolated container evaluation.

  • 2.1 TASK DEFINITION: Each task is formulated as a POMDP whose tools define actions and transitions, observations arrive sequentially, and execution-based rewards evaluate the resulting environment state.The formulation includes state, action, observation, transition, reward, and instruction spaces.
  • 2.2 ENVIRONMENT: TOOLATHLON sources tools from MCP servers and implements or refines servers when open-source versions lack required functionality or contain bugs.The benchmark begins by selecting valuable common applications, then locates or develops corresponding MCP servers.
  • 2.2 ENVIRONMENT: The benchmark combines remote applications such as Google Sheets, Google Calendar, Notion, and Gmail with locally containerized environments to support realistic state interaction.Remote environments reduce implementation effort but complicate reproducible state modification, motivating local deployment for some settings.
  • 2.4 EVALUATION: The agent framework supports explicit reasoning and tool calls while adding tool-error handling, overlong-response handling, and context-history management.These enhancements target robust evaluation of the action loop.
  • 2.3 INITIAL STATE SETUP: Tasks begin from pre-existing states rather than empty environments because state composition changes difficulty even when task descriptions are similar.Working with ten mixed useful and unrelated files is harder than working with a folder containing one relevant file.
  • 2.4 EVALUATION: Separate containers isolate task execution and enable parallel evaluation, including Claude-4.5-Sonnet on 108 tasks in about 70 minutes using 10 processes.Isolation prevents task interference while supporting scalable evaluation.

3 THE TOOLATHLON TASKS

TOOLATHLON tasks are sourced from genuine user demands and designed as multi-application workflows with concise, fuzzy instructions whose intent can be inferred from environment context. They undergo extensive implementation and quality checks before producing the 108-task benchmark.

  • 3.1 TASK SOURCING AND FUZZY TASK INSTRUCTION: Tasks are sourced or crafted to reflect real user demands and intentionally require interaction with multiple MCP servers.The sourcing principles aim to reproduce authentic human workflows.
  • 3.1 TASK SOURCING AND FUZZY TASK INSTRUCTION: TOOLATHLON instructions are fuzzy or ambiguous, but their intended outcomes can be deterministically inferred from existing data, examples, or document templates.Agents must infer intent, formulate plans, execute them, and handle unexpected tool-call errors.
  • 3.1 TASK SOURCING AND FUZZY TASK INSTRUCTION: Unlike step-by-step prompts in some benchmarks, TOOLATHLON prompts require agents to discover relevant information and decide how to complete the task from the environment.A Notion example requires finding job status and determining the appropriate record format from database examples.
  • 3.2 TASK IMPLEMENTATION: The benchmark contains 108 tasks after multiple rounds of quality checks, filtering, and refinement, with topic distribution reported in Figure 4 and Table 2.The task set is finalized only after several weeks of review and implementation work.
  • 3.2 TASK IMPLEMENTATION: Implementing a single TOOLATHLON task requires an average of 4–6 hours of work by a research graduate student in computer science.Task implementation includes realistic initial-state setup and an evaluation script.
  • 3.2 TASK IMPLEMENTATION: Five to six experienced authors conduct intensive quality checks requiring approximately 5 hours of labor per task per checking round.Reviews target correctness, solvability, unambiguity, and consistent standards across tasks.

4 EXPERIMENT

The evaluation benchmarks leading commercial and open-weight models on TOOLATHLON using repeated trajectory trials and multiple success metrics. Claude-4.5-Sonnet ranks first, but overall performance remains low across models.

  • Setup: The benchmark compares leading commercial models and best-performing open-weight models under the same TOOLATHLON evaluation.The evaluated commercial families include GPT, o-series, Claude, Gemini, and Grok models, alongside Qwen-3-Coder and DeepSeek-V3.2-Exp.
  • Setup: The evaluation reports average pass@1 success rate, standard deviation, pass@3, pass^3, and average turns across three trials.Pass@3 measures whether at least one trajectory succeeds, while pass^3 measures whether all three trajectories succeed.
  • Main Results: Claude-4.5-Sonnet ranks first but achieves a success rate below 40%.GPT-5, Claude-4-Sonnet, Grok-4, and Claude-4.5-Haiku form a second tier with Pass@1 scores above 26% and below 30%.
  • Main Results: Open-weight models score 20% or below, with DeepSeek-V3.2-Exp reaching 20.1%.This is lower than the performance of the leading proprietary models.
  • Main Results: Claude-4.5-Sonnet performs strongly across most categories, while GPT-5 is strongest on Daily tasks and Grok-4 on Tech tasks.These category-level results indicate differing strengths across task domains.

5 ANALYSIS

The analysis attributes TOOLATHLON difficulty to tool-use errors, long-horizon execution, and overlong outputs, with success generally declining as interaction demands increase. Models may also fail by stopping early or becoming trapped processing lengthy results.

  • Tool-Calling Errors: Incorrect tool names are more likely to affect final scores than tool execution errors.Execution errors may sometimes help models infer tool structure and adjust their later calls, while leading models produce few tool-name errors.
  • Tool-Calling Errors: The analysis distinguishes hallucinated non-existing tools from errors raised during tool execution.These are the two major tool-calling error categories examined in the benchmark.
  • Long-Horizon Challenges: Higher average-turn task groups generally have lower success rates across models.Claude-4.5-Sonnet retains an advantage across Easy, Medium, and Hard groups, but Medium and Hard are not significantly different.
  • Long-Context Challenges: Overlong tool outputs occur in approximately 15% to 35% of trajectories and usually reduce model success rates.Models often become trapped processing lengthy HTML or database outputs even when the underlying tasks are logically straightforward.
  • Performance and Expenses: Most models remain under $1 per task, while Claude-4.5-Sonnet ranks third in cost despite achieving the highest performance.Grok-4-Fast, Grok-Code-Fast-1, and DeepSeek-V3.2-Exp are identified as lower-cost alternatives.
  • Performance and Expenses: Most models generate between 5K and 10K output tokens, while some reasoning-focused models generate more.Claude and Grok models achieve strong results with fewer tokens, suggesting greater reliance on environment observation than extended internal reasoning.

6 RELATED WORK

Tool-agent benchmarks range from simulated tools to real applications with increasingly realistic workflows and environments. TOOLATHLON extends this trajectory by combining broad application diversity, authentic initial states, long-horizon orchestration, concise prompts, and deterministic evaluation.

  • Simulated and Constrained Benchmarks: Earlier benchmarks often use simulated tools, mock implementations, or language-model-based emulation instead of executing real APIs.τ-Bench, BFCL, and ACEBench are representative examples of this setting.
  • Simulated and Constrained Benchmarks: Other benchmarks connect agents to real APIs but use synthetic or constrained environments with artificially constructed initial states.AppWorld, MCPWorld, MCP-RADAR, MCPEval, and MCP-AgentBench exemplify this category.
  • Realistic Tool Benchmarks: Recent realistic-tool benchmarks still remain limited in domain diversity, environment-state realism, or prompt naturalness.LiveMCPBench, LiveMCP-101, MCPAtlas, MCPUniverse, and MCPMark introduce more authentic tools or workflows but retain such limitations.
  • TOOLATHLON: TOOLATHLON combines real tools with realistic environments across 32 applications and 604 tools.Its tasks span broad domains and often require long-horizon coordination across applications.
  • TOOLATHLON: TOOLATHLON uses authentic usage scenarios, cross-application tasks, concise fuzzy prompts, and deterministic script-based evaluation.These design choices require agents to infer intent and plan autonomously while enabling reliable outcome verification.

7 CONCLUSION

TOOLATHLON evaluates real-world, long-horizon language-agent tasks across many applications and tools. Its results show substantial limitations in current models and identify execution robustness, tool-calling errors, and long-context handling as key challenges.

  • Conclusion: Current models remain limited, with Claude-4.5-Sonnet achieving only a 38.6% success rate.The conclusion identifies substantial room for improvement in complex multi-step workflow execution.
  • Conclusion: TOOLATHLON analysis identifies long-context handling, tool-calling errors, and execution robustness as key challenges.These challenges are framed as important targets for improving practical language agents.
  • Benchmark Resources: The benchmark draws on MCP servers from multiple sources, including community-developed projects and further functional enhancements.Table 4 lists the complete server set and indicates whether servers access remote or local resources and whether they are writable.

B IMPLEMENTATION DETAILS OF AGENT FRAMEWORK

The agent framework extends the OpenAI-Agent-SDK with robust handling for tool errors, overlong responses, and context pressure, plus additional local tools for evaluation.

  • Robustness enhancements: Tool errors are returned to the agent as observations so execution can continue instead of terminating the loop.This applies when models invoke nonexistent tools or tool execution returns errors.
  • Robustness enhancements: Overlong tool outputs are truncated at 100K characters, while paging provides access to cached raw responses.The default page size is 10K characters.
  • Robustness enhancements: Context management tools expose token counts and turn numbers, support dropping historical turns, and automatically retain only the last 10 turns when limits are exceeded.Dropped and retained history remains searchable through the supporting tools.
  • Additional tools: The framework adds Python execution, web search, claim done, and sleep as local tools alongside MCP servers.These tools support computation, information retrieval, explicit completion, and waiting during task execution.

C.1 THE IMPACT OF TOOL CALL ERRORS ON FINAL SUCCESS RATES

Tool-calling errors are associated with lower final success rates for most evaluated models, indicating that misunderstanding or failed execution harms task completion.

  • Observed impact: For most models, trajectories with more frequent tool-calling errors have lower success rates than error-free trajectories.The errors include hallucinated or forgotten tool names and errors raised during tool execution.

C.2 CASE STUDY: TYPICAL CHALLENGES MODELS MET IN TOOLATHLON

Toolathlon case studies expose recurring failures in realistic long-horizon workflows: models struggle with fuzzy instructions, complex environment states, exhaustive processing, and sustained execution.

  • Fuzzy instructions: Models may fail to exploit available tools and resources when fuzzy instructions do not explicitly specify how or when to use them.They may also fail to pursue indirect methods after an initial attempt fails.
  • Complex environment states: Models often omit items in realistic environments containing many developers, products, files, or other records that must be examined.The benchmark tests whether agents can explore and act across complex pre-existing states rather than simplified ones.
  • Complex environment states: A repository-to-Notion task failed because the model did not inspect all files and folders, producing far fewer task items than expected.The task involved more than 10 developers and required comparing branches against an example template.
  • Long-horizon execution: Models can terminate prematurely during repetitive multi-year workflows, leaving the remaining work to the user.In the music-analysis example, the model stopped after completing only 1940 despite a 1940s-wide task.
  • Fuzzy instructions: Concise instructions can require agents to infer spreadsheet formats or repository files before making the requested updates.The k8s-safety-audit task requires reading Week1 and Week2 before filling Week3, while email-paper-homepage requires repository and file exploration.
  • Evaluation focus: The benchmark explicitly examines whether agents can infer actions from concise instructions by exploring the real environment with tool calls.This capability is tested in both the safety-audit and personal-homepage examples.

E.2 COMPLETE EXAMPLE TASK TRAJECTORIES

The trajectories illustrate Toolathlon’s long-horizon, multi-step tasks, including checkpoint evaluation, debugging, repository creation, and cross-application hiring workflows. Claude-4-Sonnet completed one task but failed another after omitting required README images.

  • HuggingFace Upload: Claude-4-Sonnet evaluated multiple checkpoints to identify the highest-scoring model, rather than receiving evaluation results directly.The agent inspected the workspace and ran evaluation scripts for checkpoints.
  • HuggingFace Upload: The HuggingFace Upload trajectory failed after 47 tool callings because required README images were omitted from the uploaded files.The agent identified the correct checkpoint but did not infer that the README depended on those images.
  • Notion HR: The Notion HR trajectory completed after 45 tool calls by matching candidates to open positions and emailing applicants without available head counts.The agent read resumes, updated the Notion table, and sent rejection emails.
  • HuggingFace Upload: 0.712 was the highest overall eval accuracy, achieved by checkpoint step 1000.The agent reported step 1000 as the best checkpoint after evaluating the available models.
Loading 2510.25726v2…