PaperlayerView exact revision on arXiv

Source-linked AI summary

The Tool Decathlon: Benchmarking Language Agents for Diverse, Realistic, and Long-Horizon Task Execution

Junlong Li, Wenshuo Zhao, Jian Zhao, Weihao Zeng, Haoze Wu, Xiaochen Wang, Rui Ge, Yuxuan Cao, Yuzhen Huang, Wei Liu, Junteng Liu, Zhaochen Su, Yiyang Guo, Fan Zhou, Lueyang Zhang, Juan Michelini, Xingyao Wang, Xiang Yue, Shuyan Zhou, Graham Neubig, Junxian He

arXiv:2510.25726v2cs.CLcs.AI

TL;DR

Language-agent benchmarks often cover limited domains and tools, despite real-world workflows spanning applications and complex environment states. Toolathlon evaluates agents on realistic, long-horizon tasks across 32 applications, and its best-performing model, Claude-4.5-Sonnet, achieves only 38.6% success.

  • Problem

    Existing language-agent benchmarks cover limited domains and tools, while real-world tasks involve diverse applications, complex environment states, and long interaction trajectories.

    Source: §1 · pp. 1–2

  • Method

    Toolathlon benchmarks agents on 108 realistic, often multi-application tasks across 32 applications and 604 tools, using dedicated deterministic evaluation scripts.

    Source: §1 · p. 2

  • Results

    Current models achieve limited success across the benchmark; Claude-4.5-Sonnet performs best with a 38.6% success rate.

    Source: §1 · p. 3, §7 · p. 10

  • Takeaways & Limitations

    The evaluation indicates substantial room to improve language agents' handling of complex multi-step workflows, including long-context handling, tool-calling errors, and execution robustness.

    Source: §7 · p. 10

Abstract

from arXiv · show

Real-world language agents must handle complex, multi-step workflows across diverse Apps. For instance, an agent may manage emails by coordinating with calendars and file systems, or monitor a production database to detect anomalies and generate reports following an operating manual. However, existing language agent benchmarks often focus on narrow domains or simplified tasks that lack the diversity, realism, and long-horizon complexity required to evaluate agents' real-world performance. To address this gap, we introduce the Tool Decathlon (dubbed as Toolathlon), a benchmark for language agents offering diverse Apps and tools, realistic environment setup, and reliable execution-based evaluation. Toolathlon spans 32 software applications and 604 tools, ranging from everyday platforms such as Google Calendar and Notion to professional ones like WooCommerce, Kubernetes, and BigQuery. Most of the tools are based on a high-quality set of Model Context Protocol (MCP) servers that we may have revised or implemented ourselves. Unlike prior works, which primarily ensure functional realism but offer limited environment state diversity, we provide realistic initial environment states from real software, such as Canvas courses with dozens of students or real financial spreadsheets. This benchmark includes 108 manually sourced or crafted tasks in total, requiring interacting with multiple Apps over around 20 turns on average to complete. Each task is strictly verifiable through dedicated evaluation scripts. Comprehensive evaluation of SOTA models highlights their significant shortcomings: the best-performing model, Claude-4.5-Sonnet, achieves only a 38.6% success rate with 20.2 tool calling turns on average, while the top open-weights model DeepSeek-V3.2-Exp reaches 20.1%. We expect Toolathlon to drive the development of more capable language agents for real-world, long-horizon task execution.

1 INTRODUCTION

Toolathlon evaluates language agents on diverse, realistic, long-horizon tasks that coordinate real-world applications and tools. Its authors report low success rates for leading models and provide deterministic, isolated evaluation.

1 INTRODUCTION

  • The introduction contrasts Toolathlon with benchmarks limited in domains, tools, cross-application tasks, realistic states, or interaction length.It notes that MCPUniverse spans six domains, with 90% single-app tasks and synthetic initial states, while MCPMark includes five apps and GAIA2 focuses on mobile tasks with simplified synthetic environments.Source: §1 · pp. 1, 3
  • Real-world tasks can require coordinating applications, diverse environment states, and long interaction trajectories, creating challenges for agent generalization.The introduction illustrates this with an administrative agent monitoring a Snowflake database, consulting a PDF manual, and emailing managers and customers.Source: §1 · p. 2
  • Toolathlon covers 32 real-world applications and 604 tools across 108 tasks, mostly requiring realistic coordination across multiple applications.Tasks use dedicated deterministic evaluation scripts that compare outcomes with static or dynamically generated ground-truth states.Source: §1 · p. 2
  • Toolathlon uses representative real software where feasible and locally deployed applications where needed to create complex, scalable environment states and authentic, concise, fuzzy prompts.Examples include poste.io in place of Gmail and WooCommerce in place of Shopify; prompts require models to infer intent and plan autonomously.Source: §1 · p. 2
  • 38.6% accuracy for Claude-4.5-Sonnet and a 20.1% success rate for DeepSeek-V3.2-Exp show that leading models still struggle on Toolathlon.The paper also identifies long-context modeling and robust tool-calling error tracking as major challenges for evaluated models.Source: §1 · p. 3
  • Toolathlon isolates task execution and evaluation in separate containers, enabling safe, parallel runs; evaluating Claude-4.5-Sonnet on all 108 tasks took 70 minutes with 10 processes.Each task includes initial-state setup when needed and an evaluation script.Source: §1 · p. 3

2 THE TOOLATHLON ENVIRONMENT AND EVALUATION FRAMEWORK

Toolathlon models tasks as partially observable, multi-step tool interactions in realistic environments with pre-existing state. It combines diverse MCP-based tools and applications with deterministic, execution-based evaluation of final environment states.

2.1 TASK DEFINITION

  • Toolathlon formulates each task as a POMDP whose states, observations, tools, transitions, rewards, and instructions define the agent’s interaction and execution-based evaluation.The reward function evaluates the resulting environment state.Source: §2.1 · p. 3

2.2 TOOLS, ENVIRONMENTS, AND FRAMEWORK

  • The benchmark builds 32 MCP servers from common and professional applications, combining remote services with locally containerized software and a framework for tool-based agent action loops.The framework adds tool-error handling, overlong-response handling, context-history management, and basic local tools.Source: §2.2 · p. 4

2.3 INITIAL STATE SETUP

  • 67% of tasks begin with initialized state, using scripts or preset workspace files to create realistic conditions rather than empty or overly simplified environments.The initial state can make otherwise similar instructions more difficult by mixing useful and unrelated files.Source: §2.3 · pp. 4–5

2.4 RELIABLE EXECUTION-BASED EVALUATION

  • Evaluation uses manually crafted deterministic scripts to verify final environment states, loads fewer than 10 selected MCP servers with all their tools, and runs tasks in isolated parallel containers.On Ubuntu 24.04 with 16 CPUs and 64 GB of memory, Claude-4.5-Sonnet evaluation on 108 tasks took about 70 minutes using 10 parallel processes.Source: §2.4 · p. 5

3 THE TOOLATHLON TASKS

Toolathlon’s tasks are sourced to reflect real user demands, multi-app workflows, and broad application coverage, while using fuzzy instructions that require agents to infer intent from environment state. Each task is implemented with evaluation and initial-state setup, then reviewed for correctness and solvability.

3.1 TASK SOURCING AND FUZZY TASK INSTRUCTION

  • The authors sourced 108 tasks through real-world or realistic user demands, multi-app orchestration, and a two-stage process targeting underrepresented apps.Tasks underwent multiple rounds of quality checks, filtering, and refinement before inclusion.Source: §3.1 · pp. 5–6
  • Toolathlon’s fuzzy instructions require agents to infer user intent from environment state, plan, execute, and handle unexpected tool-call errors.Unlike the MCPMark examples (The MCPMark Team, 2025), instructions may omit details such as the required format or where to find relevant information.Source: §3.1 · p. 6, Figure 3 · p. 6

3.2 TASK IMPLEMENTATION

  • Each task has an evaluation script and potential initial-state setup, using statically or dynamically collected ground-truth states and scripts to clear and refill states.Implementing a task requires an average of 4–6 hours from a computer-science research graduate student.Source: §3.2 · pp. 6–7
  • Five to six experienced authors review and revise task implementations for consistency, correctness, solvability, and unambiguity, followed by benchmark-wide cross-checking and bug fixing.Each review round requires approximately five hours of labor per task.Source: §3.2 · p. 7

4 EXPERIMENT

Claude-4.5-Sonnet ranks first on Toolathlon but remains below 40% success, while open-weight models reach at most 20%; performance also varies by task category and many models lack consistency across runs.

4.1 SETUP

  • The evaluation tests leading commercial and open-weight models with task-relevant MCP servers and common tools, allowing up to 100 turns per task.Because each MCP server exposes multiple related tools, models also see tools that are unnecessary for a given task.Source: §4.1 · p. 7
  • Models are evaluated three times, with average pass@1 and standard deviation, pass@3, Passˆ3, and average turns reported.Pass@3 measures whether at least one of three trajectories is correct; Passˆ3 measures whether all three are correct.Source: §4.1 · p. 7

4.2 MAIN RESULTS

  • Claude-4.5-Sonnet ranks first below 40%; GPT-5, Claude-4-Sonnet, Grok-4, and Claude-4.5-Haiku form a second tier, each with Pass@1 above 26% but below 30%; DeepSeek-V3.2-Exp leads open-source models at 20.1%.Source: §4.2 · p. 8
  • Higher reasoning effort did not improve thinking-oriented models, while Gemini-2.5 often neglects requirements or gives up prematurely on complex tasks.The results suggest that exploring new observations may matter more than extended internal reasoning in agentic tasks.Source: §4.2 · p. 8
  • Claude-4.5-Sonnet excels across nearly all categories, GPT-5 stands out in Daily tasks, and Grok-4 in Tech, while gaps between Pass@3 and Passˆ3 expose reliability challenges.The Pass@3–Passˆ3 differences indicate that capability coverage does not always translate into consistent task completion.Source: §4.2 · p. 8

5 ANALYSIS

The analysis examines tool-call errors, task difficulty and long-context challenges, and the relationship between model performance, output-token use, and cost.

5.2 THE LONG-CONTEXT CHALLENGES FOR LANGUAGE AGENTS

  • Higher average-turn task groups generally have lower success rates, but Medium and Hard show no significant difficulty difference, suggesting premature stopping may also cause failures.Claude-4.5-Sonnet and GPT-5 score higher on the Hard subset than on Medium.Source: Figure 6 · p. 8, §5.2 · p. 9

5.1 THE FAILURE OF CALLING TOOLS

  • Most models have lower success rates on trajectories with tool-calling errors than on error-free trajectories.The analysis compares success rates for trajectories with and without tool-calling errors, but the supplied passage gives no numerical differences.Source: §5.1 · p. 9

5.2 THE LONG-CONTEXT CHALLENGES FOR LANGUAGE AGENTS

  • 15%–35% of trajectories contain overlong tool outputs, and most models’ success rates decline on them, often while models struggle to process lengthy outputs.Only a few models maintain nearly unchanged performance, even on often logically straightforward tasks such as price comparison or data extraction.Source: Figure 7 · p. 9, §5.2 · p. 9

5.3 THE RELATIONSHIP BETWEEN PERFORMANCE AND EXPENSES

  • Claude-4.5-Sonnet has the highest performance but ranks third in cost; several lower-cost models offer alternatives, while most models use 5K–10K output tokens.Claude-4-Sonnet and Grok-4 are relatively expensive, while most other models cost under $1 per task. Reasoning-focused o4-mini and GPT-5(-high) generate more tokens; Claude and Grok-4 achieve strong results with fewer, while Gemini-2.5-Flash has the lowest token counts and lower accuracy.Source: §5.3 · p. 9

6 RELATED WORK

Related work progresses from simulated tools to real APIs and production-grade environments, but remains limited in authentic state, cross-application scope, task diversity, or naturalistic instructions. TOOLATHLON addresses these gaps with realistic environments, long-horizon orchestration, and deterministic evaluation.

6 RELATED WORK

  • Simulation-based benchmarks test function-calling accuracy or multi-turn tool selection, but omit challenges of executing real tools in unpredictable environments.Examples include τ-Bench, BFCL, and ACEBench; mock implementations or language-model emulation support efficiency and reproducibility.Source: §6 · p. 10
  • Benchmarks using real APIs or app simulations often start from artificial states or focus on single-app tasks, limiting authentic multi-application workflow complexity.Examples include AppWorld (Trivedi et al., 2024), MCPWorld (Yan et al., 2025), MCP-RADAR, MCPEval, and MCP-AgentBench (Guo et al., 2025).Source: §6 · p. 10
  • Production-grade MCP benchmarks add realistic tools, workflows, and outputs, yet remain limited in domain diversity, authentic state initialization, or naturalistic underspecified prompts.Examples include LiveMCPBench (Mo et al., 2025), LiveMCP-101 (Yin et al., 2025), MCPAtlas (Scale AI, 2025), MCPUniverse, and MCPMark.Source: §6 · p. 10
  • TOOLATHLON combines 32 applications and 604 tools with authentic initial states, long-horizon cross-app tasks, concise fuzzy prompts, and deterministic script-based evaluation.Source: §6 · p. 10

7 CONCLUSION · A MCP SERVER LIST AND SOURCE

TOOLATHLON evaluates agents on long-horizon tasks across 32 applications and 604 tools, and the best-performing model achieves only 38.6% success. Its MCP server set spans diverse domains, combining enhanced community projects with newly developed servers.

7 CONCLUSION

  • Claude-4.5-Sonnet achieves only 38.6% success on TOOLATHLON, which spans 32 applications and 604 tools.The conclusion identifies substantial room for improvement in handling complex, multi-step workflows.Source: §7 · p. 10
  • The analyses identify long-context handling, tool-calling errors, and execution robustness as key challenges for current models.The authors expect TOOLATHLON to drive development of more capable and robust agents for practical deployment.Source: §7 · p. 10

A MCP SERVER LIST AND SOURCE

  • The benchmark’s MCP servers span domains from entertainment and education to business productivity and software development, with most sourced from community projects..Source: A MCP SERVER LIST AND SOURCE · p. 14
  • For a substantial proportion of community-developed servers, the authors made functional enhancements, including output optimization, more robust error handling, and added tools.Source: A MCP SERVER LIST AND SOURCE · p. 14
  • Because available MCP server coverage remains insufficient, the authors developed new servers to extend supported tasks into more domains and plan to make them publicly available.Source: A MCP SERVER LIST AND SOURCE · p. 14
  • Table 4 lists all MCP servers used in TOOLATHLON and indicates whether each can access local or remote resources and whether its tools can modify them.Source: Table 4 · p. 14

B IMPLEMENTATION DETAILS OF AGENT FRAMEWORK

Built on OpenAI-Agent-SDK v0.0.15, the framework adds resilience features for tool errors and context limits, plus local tools.

B IMPLEMENTATION DETAILS OF AGENT FRAMEWORK

  • Tool-call errors are returned to the agent as observations so it can continue its trajectory, and the framework adds Python, web search, Done, and Sleep tools.This is intended to support operation in noisy environments where tool calls may fail.Source: B IMPLEMENTATION DETAILS OF AGENT… · pp. 14–15
  • Tool outputs are truncated at 100K characters, while a paging toolkit lets agents search cached raw outputs in 10K-character pages.The paging toolkit is available for all task evaluations.Source: B IMPLEMENTATION DETAILS OF AGENT… · p. 14
  • Agents can inspect token counts and turn numbers, drop searchable history, and rely on automatic clearing that preserves the last 10 turns’ preview and initial user input.The automatic clearing safeguard activates when context exceeds limits without model intervention.Source: B IMPLEMENTATION DETAILS OF AGENT… · p. 15

C EXTRA ANALYSIS FOR TOOLATHLON · C.1 THE IMPACT OF TOOL CALL ERRORS ON FINAL SUCCESS RATES

Tool call errors were negatively associated with task success for most models, with the relationship most pronounced in GPT-5. Errors included incorrect tool names and execution failures.

C.1 THE IMPACT OF TOOL CALL ERRORS ON FINAL SUCCESS RATES

  • For most models, tool-call error frequency was negatively correlated with task success, with the impact most pronounced in GPT-5.Errors included incorrect tool names caused by hallucination or forgetting, and errors raised during tool execution.Source: §C.1 · p. 15

C.2 CASE STUDY: TYPICAL CHALLENGES MODELS MET IN TOOLATHLON … E.1 EXAMPLES FOR FUZZY USER INSTRUCTIONS

The case studies show models failing to use available tools effectively, inspect complex environment states completely, and sustain long-horizon work. The prompt and fuzzy-instruction examples illustrate the autonomy required to infer and complete actions from sparse directions.

C.2 CASE STUDY: TYPICAL CHALLENGES MODELS MET IN TOOLATHLON

  • DeepSeek-V3.1 identified the ODC-BY license but failed to update the dataset pages after repeatedly trying server access or web search instead of using the provided token through terminal commands or Python.The task explicitly noted that the Hugging Face MCP did not support updates.Source: §C.2 · pp. 15–16
  • Claude-4-Sonnet’s Notion update contained 91 rows instead of 116 and omitted the analytics-dashboard task after it failed to inspect all repository files and folders.The task required examining branches for more than 10 developers and comparing them against a task template.Source: §C.2 · p. 16
  • Grok-Code-Fast-1 analyzed only 1940 of the requested 1940s singles, then claimed completion after 66 turns despite the intended instruction to finish before exiting.The task required a separate analysis sheet for each year from 1940 through 1949.Source: §C.2 · p. 16

D PROMPT

  • The evaluation prompt supplies a workspace directory, explains how to resolve relative paths, and allows completion by calling “claim done” or responding without a tool.It instructs agents to work independently without seeking confirmation or feedback.Source: D PROMPT · pp. 16–17

E.1 EXAMPLES FOR FUZZY USER INSTRUCTIONS

  • Toolathlon’s fuzzy-instruction examples require agents to infer the desired format from prior spreadsheet tabs or identify and edit relevant files in a GitHub repository.The Kubernetes audit task says only to update Week3; the homepage task says only to update the personal page based on acceptance emails.Source: §E.1 · pp. 17–18, Figure 10 · p. 17, Figure 11 · p. 17
  • The fuzzy-instruction examples assess whether agents can use tool calls to explore real environments and determine the actions required by concise, underspecified requests.The examples concern a cluster security audit and a personal homepage update.Source: §E.1 · pp. 17–18

E.2 COMPLETE EXAMPLE TASK TRAJECTORIES

This section presents example Claude-4-Sonnet trajectories for multi-step model-upload and Notion HR workflows. The examples trace task execution from locating relevant checkpoints, resumes, and HR records through upload and candidate-record and email actions.

Traj #1: HuggingFace Upload

  • 0.712 overall eval accuracy was assigned to step 1000, the correct checkpoint, but Claude-4-Sonnet failed the HuggingFace task by omitting README.md images.The task also involved choosing among 10 checkpoints and adding evaluation results to the README before uploading the relevant files.Source: §E.2 · pp. 18, 23, 25
  • Claude-4-Sonnet had to run evaluations because scores were not provided and identify and correct introduced syntax errors to determine the best checkpoint.Although it identified the correct checkpoint, the upload remained incomplete because it omitted images referenced by the README.Source: §E.2 · p. 18

Traj #2: Notion HR

  • Claude-4-Sonnet perfectly completed the Notion HR task after 45 tool calls, matching candidates to open roles from resumes and emailing rejections when positions lacked headcount.It read all resumes, entered the corresponding information in the Notion table, and sent rejection emails to candidates without a suitable available position.Source: §E.2 · p. 18

E.2 COMPLETE EXAMPLE TASK TRAJECTORIES

  • 0.550 was the updated MyAwesomeModel score for Math Reasoning in the README table.The agent replaced the {RESULT} placeholder with 0.550 before copying the README into the upload directory.Source: §E.2 · p. 28
  • After an initial 404, the agent corrected the Hugging Face repository name to include its username and successfully uploaded the model.The upload directory contained README.md, config.json, and pytorch model.bin, and the agent then declared the task complete.Source: §E.2 · pp. 29–30
  • The Notion HR task required resume-faithful candidate records, deletion of sample entries, and rejection emails only for applicants whose positions were closed.The requested email had to follow the supplied template, including line breaks.Source: §E.2 · p. 31
  • The agent found 10 resume PDFs and identified three databases on the HR Record page: Job Positions, Job Application Timeline, and Candidates.The trajectory then began examining position availability and candidate records; the supplied passage does not show completion.Source: §E.2 · pp. 31–32
  • After an invalid Notion block update, it switched to archiving pages individually and reported deleting all sample candidates.The first attempt returned a validation error; the agent then used a page-archive request.Source: §E.2 · p. 34
  • After an invalid candidate-page creation attempt, it checked the database structure, corrected the format, and reported adding all 10 candidates.The corrected request returned a page response before the agent said the resume-based entries were complete.Source: §E.2 · pp. 34–35
  • It targeted Angela Moore, Debra Smith, and Emily James for Securities Trader (0 positions), plus Martha Morales for Portfolio Manager (0 positions).These were the candidates identified for rejection emails because their applied positions had no openings.Source: §E.2 · p. 35
  • The agent reported completing the workflow after adding all 10 resume-based candidates and identifying applicants for rejection emails.It identified candidates whose positions had zero headcount for rejection emails.Source: §E.2 · p. 35

Related papers

Loading 2510.25726v2…