Source-linked AI summary
AsyncTool: Evaluating the Asynchronous Function Calling Capability under Multi-Task Scenarios
Kou Shi, Ziao Zhang, Shiting Huang, Avery Nie, Zhen Fang, Qiuchen Wang, Lin Chen, Huaian Chen, Zehui Chen, Feng Zhao
TL;DR
Existing benchmarks largely miss delayed tool feedback in concurrent, multi-task settings. AsyncTool evaluates agents in these environments and finds that delayed feedback causes clear performance degradation, while stronger temporal coordination supports better performance.
Problem
Existing evaluations provide limited coverage of concurrent tool-use tasks with delayed or out-of-order feedback and lack standardized metrics for these settings.
Method
AsyncTool combines concurrent dependent tasks, simulated tool-response delays, a diverse reconstructed dataset, multi-level evaluation, and efficiency-oriented metrics.
Results
Delayed feedback causes clear performance degradation, while stronger models better use idle periods, resume pending tasks, and avoid dependency violations, task neglect, and tool confusion.
Takeaways & Limitations
The findings highlight temporal coordination as important for future tool-using agents operating across delayed, concurrent tasks.
Takeaways & Limitations
The dataset’s scenarios and available tools are constrained by BFCLv3 and NESTFUL, while reconstruction with Gemini-2.5-pro increases construction and verification costs.
Abstract
from arXiv · showhide
Large language model (LLM)-based agents have shown strong capabilities in using external tools to solve complex tasks. However, existing evaluations often overlook the temporal dimension of tool use, especially the impact of tool response latency, and are usually limited to single-task settings. In real-world applications, multiple tasks often need to be executed concurrently, and overall efficiency depends on whether an agent can use idle time while waiting for tool responses. We refer to this capability as asynchronous tool calling. To evaluate it, we propose AsyncTool, a benchmark for assessing LLM-based agents in interactive multi-task tool-use environments with delayed tool feedback. AsyncTool presents multiple heterogeneous tasks simultaneously and simulates realistic tool response latency during execution. Using a hybrid data evolution strategy, we construct a diverse asynchronous multitasking dataset that covers multiple scenarios and tool-use patterns. We evaluate models at the step, sub-task, and task levels, and introduce efficiency-oriented metrics to measure task coordination and completion efficiency. Extensive experiments show that delayed tool feedback poses substantial challenges to current agents and leads to clear performance degradation. Models that better coordinate task switching, dependency tracking, and state maintenance achieve stronger performance on AsyncTool. Our analysis identifies key failure modes of current tool-using agents and provides practical insights for designing future systems with stronger temporal reasoning and coordination capabilities.
1 Introduction
ASYNCTOOL evaluates asynchronous tool calling when agents execute concurrent multi-step tasks with delayed feedback. It addresses benchmark gaps through diverse multitasking data, multi-level evaluation, and efficiency metrics focused on temporal coordination.
- Motivation: Concurrent tasks involving different tools incur latency, making sequential synchronous execution inefficient because it fails to use idle waiting time.Existing evaluation also lacks sufficient metrics and standardized protocols for concurrent tasks with delayed and out-of-order tool feedback.
- Benchmark: ASYNCTOOL benchmarks asynchronous tool calling in interactive multi-task environments with delayed tool feedback.It jointly considers delayed feedback, concurrent multi-task execution, and multi-step tool use.
- Dataset: A hybrid data-evolution strategy composes validated single-task trajectories into diverse tasks spanning task numbers, task types, scenarios, and dependency structures.The resulting dataset is designed for asynchronous multitasking evaluation.
- Evaluation: The evaluation protocol measures performance at the step, sub-task, and task levels, covering tool-call correctness and end-to-end task completion.This multi-level design captures both fine-grained and end-to-end performance.
- Findings: Efficiency-oriented metrics analyze task interleaving and completion under tool latency, exposing challenges in temporal coordination and dependency tracking.Stronger models use idle waiting periods to advance other tasks and resume pending tasks appropriately, while weaker models show dependency violations, task neglect, and tool confusion.
2 ASYNCTOOL
ASYNCTOOL evaluates asynchronous tool calling by requiring agents to coordinate multiple tasks under delayed tool feedback, including task switching, dependency-aware scheduling, and state tracking. It builds and validates a multitasking benchmark from existing tool-use resources, covering diverse dual- and tri-task configurations and evaluating completion at the task level.
- Interaction Paradigm: ASYNCTOOL simulates delayed tool feedback and concurrent execution, requiring agents to decide whether to wait for a pending call or switch tasks.This setting makes delayed feedback, task interleaving, and dependency-aware scheduling central challenges.
- Interaction Paradigm: Agents must invoke valid tools with correct arguments, track task states, respect intra-task dependencies, and coordinate tool-call scheduling across pending tasks.The benchmark treats tool use as temporal coordination and asynchronous task management rather than a purely sequential procedure.
- Dataset Construction: 358 validated instances comprise the high-quality Single-Task Dataset after model-assisted reconstruction, manual correction, and repeated trajectory verification.Gemini 2.5 Pro supports coarse reconstruction, while human annotation corrects invalid arguments, mismatched descriptions, execution errors, and ordering inconsistencies.
- Multi-Task Composition: 712 instances comprise the final Multitasking Dataset, created by weighted random sampling across dual-task and tri-task settings with within-class and cross-class combinations.These four configurations balance task quantity and task type while avoiding the excessive sample count produced by exhaustive combination.
- Evaluation Protocol: At the task level, completion is correct only when all corresponding subtask-level trajectory-completion and environment-consistency metrics are satisfied.The resulting task accuracy measures whether the agent completes and coordinates the entire multi-subtask task.
3 Experiment
The experiment evaluates 19 models on asynchronous tool calling under multi-task scenarios, finding GPT-4.1 strongest overall while revealing trade-offs between accuracy and efficient task interleaving. Closed-source models generally switch tasks compactly, whereas frequent switching alone does not ensure successful asynchronous execution.
- Experimental Setup: The benchmark evaluates 19 models, including prominent closed-source and open-source LLMs, for asynchronous tool calling under multi-task scenarios.The closed-source set includes Qwen-max, Kimi k2, Gemini 2.5 Pro, GPT-4.1, GPT-4o, and GPT-5; open-source models include LLaMA, Qwen, GLM4, and DeepSeek.
- Q1: Which Model is Better in Completing Multiple Tasks Asynchronously?: 38.06: GPT-4.1 achieves the highest asynchronous capability score, while DeepSeek-V3.1-Terminus performs comparably to closed-source models.This result is reported in the main asynchronous multitasking evaluation.
- Q1: Which Model is Better in Completing Multiple Tasks Asynchronously?: Closed-source models score consistently highly at the step level, while open-source models show notable discrepancies; sub-task scores are nearly double overload-model scores.Closed-source models also use fewer dialogue turns on average than open-source models, indicating greater efficiency in the same environment.
- Q2: How do Accuracy and Efficiency Trade off in Asynchronous Multi-Task Tool Use?: GPT-4.1 achieves the highest Overall score with a relatively low Same-task Streak, while Gemini 2.5 Pro and GPT-4o combine strong accuracy with compact task switching.A lower Same-task Streak indicates stronger interleaving, and high-performing models can use tool-waiting time to advance other tasks.
- Q2: How do Accuracy and Efficiency Trade off in Asynchronous Multi-Task Tool Use?: Frequent task switching alone does not guarantee successful asynchronous execution because models must track task states, dependencies, and tool outputs.Smaller open-source models can have low Same-task Streak but limited Overall scores, whereas DeepSeek-V3.1 can achieve competitive Overall scores with a higher Same-task Streak.
Q3: What Challenges do LLMs Encounter in ASYNCTOOL?
ASYNCTOOL exposes challenges in temporal reasoning, task coordination, and state management under delayed tool feedback. Performance degrades as asynchronous conditions, task counts, and response latencies increase, especially when models fail to interleave dependent work.
- Temporal reasoning and dependencies: Lower-performing models often issue the next function call without waiting for the current response, causing errors when calls are dependency-linked.Higher-performing models identify dependencies, execute sequentially, and use idle time to advance other tasks.
- Task coordination: Some models neglect earlier tasks by executing the most recently presented task first, particularly in tri-task settings and smaller models.This failure is almost absent in larger 70B-scale models.
- Performance degradation: Asynchronous multitasking produces model-dependent performance degradation and a simultaneous decline in the SC index compared with regular multitasking.Gemini 2.5 Pro has the largest SC drop among closed-source models, while Qwen3-8B has the greatest decrease among open-source models.
- Delayed feedback and interleaving: Delayed tool responses prevent agents from accessing information needed for subsequent steps, requiring high-performing models to shift attention and progress on other tasks.Relevant unavailable information includes file-operation confirmations, query results, and critical state data.
- Scaling and latency: Task quantity and response latency increase difficulty: four tasks make each task significantly harder, while longer latency causes delayed execution, more interactions, and greater difficulty.Random latency settings add evaluation randomness but provide a more realistic assessment of model performance.
4 Conclusion
ASYNCTOOL benchmarks asynchronous tool calling in interactive multi-task environments with delayed feedback, requiring agents to coordinate tasks, use idle waiting periods, and respect intra-task dependencies. It evaluates behavior across step, sub-task, and task levels using a diverse dataset built from validated single-task tool-use trajectories.
- Conclusion: ASYNCTOOL evaluates asynchronous tool calling in interactive multi-task environments with delayed tool feedback.It differs from conventional tool-use evaluations by introducing multiple concurrent tasks and delayed results.
- Conclusion: Agents must coordinate multiple tasks, utilize idle waiting periods, and respect intra-task dependencies when tool results are delayed.These requirements define the benchmark’s asynchronous multitasking setting.
- Conclusion: The benchmark uses a diverse asynchronous multitasking dataset constructed from validated single-task tool-use trajectories.This construction grounds the multitasking evaluation in validated tool-use behavior.
- Conclusion: ASYNCTOOL evaluates model behavior at the step, sub-task, and task levels.These evaluation levels capture behavior at progressively broader units of task execution.
Limitations · A Related Work · B Evaluation
AsyncTool extends tool-use evaluation to asynchronous multi-task settings by incorporating response latency and systematically assessing concurrent tool calling. Its scope remains constrained by datasets reconstructed from BFCLv3 and NESTFUL, which limit scenario and tool diversity.
- Limitations: AsyncTool introduces response delays into tool calls to enable fine-grained evaluation of agents executing multiple tasks asynchronously.The benchmark evaluates tool-using agents in an environment where tool feedback is delayed.
- Limitations: AsyncTool’s dataset is reconstructed from BFCLv3 and NESTFUL, constraining its task scenarios and available-tool range despite fine-grained annotation.The source benchmarks inherently limit the diversity of scenarios and tools represented.
- A Related Work: Prior multi-step tool-call benchmarks evaluate tasks requiring multiple tool calls in the correct order, creating comprehension and planning challenges.These benchmarks address increasingly complex tasks with richer dependencies.
- A Related Work: Related asynchronous-multitasking studies introduce time changes or multitasking evaluation tasks to assess models under temporal variation.This line of work establishes asynchronous multitasking as an emerging evaluation direction for large LLMs.
- A Related Work: AsyncTool is the first benchmark to systematically evaluate asynchronous tool calling in realistic multi-task scenarios with explicit tool-response latency.It differs from prior benchmarks by incorporating delayed feedback into the evaluation environment.
- A Related Work: Concurrent function-call generation and execution has been proposed to improve LLM operational efficiency.This approach enables models to generate and execute function calls concurrently.
B.1 Metrics · B.2 Compare ASYNCTOOLwith other benchmarks · B.3 Model Version
ASYNCTOOL evaluates tool-use responses through character, trajectory, and environment matching, with subset validation comparing predicted actions against golden truth. The section also situates ASYNCTOOL among existing benchmarks and specifies the evaluated model versions.
- B.1 Metrics: Character matching extracts function names and parameters, then compares them with the golden truth while accounting for incorrect calls under subset validation.Given a tool list T and query q, the model generates a series of predicted actions for comparison.
- B.1 Metrics: Subset validation verifies whether the golden action set Agt is contained in the predicted set Apred.The method explicitly checks Agt ⊆ Apred.
- B.1 Metrics: F1 function & Parameters uses metrics adapted from NESTFUL to evaluate model responses, while frequent task switching causes incorrect tool invocation and lower performance.The reported issue is reduced accuracy in recognizing different tasks during frequent switching.
- B.1 Metrics: Trajectory completion executes golden and predicted tool-call trajectories, compares their returned results, and applies the same subset-judgment method.The executor returns Rgt and Rpred for comparison.
- B.1 Metrics: Environment matching compares executor instances, including tool calls that alter the environment, to provide comprehensive evaluation despite most query calls causing no changes.This metric follows BFCL’s implementation.
- B.2 Compare ASYNCTOOLwith other benchmarks: Tab. 2 compares ASYNCTOOL with existing tool-use and asynchronous benchmarks.The passage identifies the comparison but provides no individual benchmark values.
- B.3 Model Version: The evaluated model versions are gpt-4o-2024-11-20, gpt-4.1-2025-04-14, gemini-2.5-pro-preview-05-06, GLM-4-32B-0414, and Qwen3-30B-A3B-Instruct-2507.These correspond to GPT-4o, GPT-4.1, Gemini 2.5 Pro, GLM-4-32B, and Qwen3-30B-A3B-Ins., respectively.
C Data … C.4 Supplementary data
The supplementary data materials document human validation of single-task trajectories, category definitions, data-length checks, and four-task sampling for ablation experiments. Together, these procedures target executable, consistent trajectories and characterize the constructed dataset.
- C.1 Human Annotation Instructions: Human annotators verified and refined single-task tool-use trajectories for consistency among task descriptions, tool calls, and execution results.The annotation goal was to ensure each trajectory was executable, deterministic, and consistent with its task description.
- C.1 Human Annotation Instructions: Annotators checked task intent, call validity, supported names and arguments, dependency relations, and final environment-state correctness.They also marked errors, corrected unambiguous cases, clarified ambiguous descriptions, and removed uncorrectable instances.
- C.1 Human Annotation Instructions: The annotation checklist emphasized incorrect initial-condition interpretations, missing prerequisites, invalid calls, dependency-order errors, and task–trajectory mismatches.Ambiguous descriptions that could yield multiple valid trajectories were also identified as a common error type.
- C.1 Human Annotation Instructions: Quality control repeated checks of trajectory validity, task–trajectory consistency, and final environment correctness until execution errors and obvious inconsistencies were eliminated.The annotation task used synthetic benchmark data and did not involve real users or private user information.
- C.2 Data details: The data categories include DM, FS, DG, MA, NO, SC, SM, TP, TB, TP*, DF, and MO, covering filesystem, message, numerical, social, string, ticket, trading, travel, data-format, and machine-operation tasks.Table 3 introduced the meanings of these categories, while Table 5 reported ASYNCTOOL’s data distribution.
- C.3 Data length: Input and output lengths were counted for each task to ensure that examples remained within a reasonable range, with the resulting distributions shown in Figure 7.The length analysis is presented as supplementary characterization of the task data.
- C.4 Supplementary data: 300 SIMILAR and 300 CROSS four-task examples were randomly selected after each combination for comparison in ablation experiments.The sampling addressed the large number of possible combination types and tasks, and the selected data were used only for ablations.
D Results · D.1 Detailed results · D.2 Other results
The results section organizes comprehensive findings by task type and reports outcomes across varying task counts. It also examines how the number of concurrent tasks affects open-source-model scores.
- D Results: The results are presented comprehensively and categorized by different types.This section introduces the paper’s comprehensive results by category.
- D Results: ASYNCTOOL’s data composition distinguishes similar task combinations from cross-task combinations and reports their counts and proportions.Here, # denotes number of and % denotes proportion of.
- D Results: Test outcomes are reported for varying numbers of tasks on the open-source model.The results specifically include experiments that vary the number of tasks.
- D.1 Detailed results: Detailed results are compiled and presented by task type in Figure 6.The figure organizes the detailed findings according to task type.
- D.2 Other results: A preliminary open-source-model experiment demonstrates the impact of task count on the score.The experiment focuses on how the number of tasks affects the measured score.
- D.2 Other results: Figure 7 presents the preliminary experiment examining task-count effects on score.This provides an additional result beyond the task-type breakdown.
D.3 Analysis of error cases
The error analysis identifies forgetting ongoing tasks as a frequent failure, producing many incompletions despite high subtask accuracy; open-source models are especially affected by frequent task switching.
- D.3 Analysis of error cases: Forgetting ongoing tasks frequently causes incompletion failures despite high accuracy in completing subtasks.The paper attributes this issue likely to insufficient prior training on multiple tasks.
- D.3 Analysis of error cases: Frequent task switching particularly challenges open-source models’ memory and localization, causing confusion in function calls and serving as a primary error source.
D.4 Analysis of model performance
Even advanced models perform suboptimally on AsyncTool, with performance varying significantly across dataset categories and difficulty growing nonlinearly as task counts increase. The benchmark emphasizes multitasking by making tasks involving three or more objectives account for over 60.
- Model limitations: Advanced models still perform suboptimally on the AsyncTool benchmark.The analysis examines the reasons for this limitation from two perspectives, including ablation experiments on mainstream large models.
- Task-scaling analysis: Difficulty grows nonlinearly as the number of tasks increases.This finding comes from ablation experiments on mainstream large models.
- Dataset-category analysis: Scores exhibit significant disparities across different dataset categories.Table 6 analyzes detailed AsyncTool results through dataset classification; bold marks the best overall performance and underlining the best within-group performance.
- Benchmark design: Tasks involving three or more objectives account for over 60.The benchmark was designed this way to encourage future models to develop greater capabilities.
D.5 Ablation results · E Use of Large Language Models
The ablation section reports supplementary AsyncTool evaluations under varied delay conditions, few-shot prompting, and interaction-round analysis. The paper also documents limited LLM use, standardized prompt reconstruction, and representative agent trajectories and failure modes.
- D.5 Ablation results: Supplementary AsyncTool results evaluate agents with a two-turn response delay.Bold marks the best overall performance, while underlining marks the best within the same group.
- D.5 Ablation results: Additional AsyncTool results test randomized delays of zero to one turn and one to two turns.Both tables distinguish best overall performance from the best result within each group.
- D.5 Ablation results: Few-shot ablations provide a successful trajectory in the prompt as a reference for AsyncTool evaluation.The corresponding results are reported specifically for few-shot settings.
- D.5 Ablation results: The study analyzes the average number of interaction rounds among models that successfully complete tasks.This analysis characterizes interaction length for successful task completion.
- E Use of Large Language Models: The authors use an LLM only for translation and language polishing, while completing the paper’s core research and content themselves.This section explicitly limits the model’s role in producing the paper.
- E Use of Large Language Models: The standardized system prompt instructs agents to handle multiple tasks, call one function per round, and switch tasks while calls remain incomplete.The prompt frames asynchronous execution as gradual completion across tasks.
- E Use of Large Language Models: Prompt reconstruction refines task descriptions and initial configurations into explicit solution paths, using reconstructed tool documentation and examples such as file operations.The reconstruction materials include a file-location, content-display, and backup-copy task, plus an example move-tool API specification.
- E Use of Large Language Models: Illustrative trajectories expose failures from insufficient temporal awareness, tool confusion, and neglecting a task.Examples include assuming Alpha Tech’s symbol before receiving the result, applying a file task’s tool to a trading task, and ignoring posting_11.