Source-linked AI summary
TaskBench: Benchmarking Large Language Models for Task Automation
Yongliang Shen, Kaitao Song, Xu Tan, Wenqi Zhang, Kan Ren, Siyu Yuan, Weiming Lu, Dongsheng Li, Yueting Zhuang
TL;DR
Task automation lacks a systematic, standardized benchmark for evaluating how LLMs decompose tasks and use external tools. TaskBench addresses this gap with Tool Graphs, back-instruct generation, and TaskEval across three automation stages. Experiments show that it reflects LLM capabilities across dimensions and correlates highly with human evaluation.
Problem
LLM task automation lacks a systematic and standardized benchmark covering task decomposition, tool selection, and parameter prediction.
Method
TaskBench uses Tool Graphs, back-instruct instruction generation, automated quality control, human verification, and TaskEval to evaluate the three automation stages.
Results
Experiments show that TaskBench reflects LLM capabilities across multiple dimensions and correlates highly with human evaluation.
Takeaways & Limitations
TaskBench provides a scalable, adaptable, and reliable benchmark for advancing LLM-based autonomous agents.
Abstract
from arXiv · showhide
In recent years, the remarkable progress of large language models (LLMs) has sparked interest in task automation, which involves decomposing complex tasks described by user instructions into sub-tasks and invoking external tools to execute them, playing a central role in autonomous agents. However, there is a lack of systematic and standardized benchmarks to promote the development of LLMs in task automation. To address this, we introduce TaskBench, a comprehensive framework to evaluate the capability of LLMs in task automation. Specifically, task automation can be divided into three critical stages: task decomposition, tool selection, and parameter prediction. To tackle the complexities inherent in these stages, we introduce the concept of Tool Graph to represent decomposed tasks and adopt a back-instruct method to generate high-quality user instructions. We propose TaskEval, a multi-faceted evaluation methodology that assesses LLM performance across these three stages. Our approach combines automated construction with rigorous human verification, ensuring high consistency with human evaluation. Experimental results demonstrate that TaskBench effectively reflects the capabilities of various LLMs in task automation. It provides insights into model performance across different task complexities and domains, pushing the boundaries of what current models can achieve. TaskBench offers a scalable, adaptable, and reliable benchmark for advancing LLM-based autonomous agents.
1 Introduction
TaskBench addresses the lack of systematic benchmarks for LLM task automation, which spans task decomposition, tool selection, and parameter prediction. It combines Tool Graphs, back-instruct data generation, quality control, and TaskEval to evaluate these capabilities.
- Task automation requires evaluating task decomposition, tool selection, and parameter prediction across complex instructions and tool dependencies.
- TaskBench uses Tool Graphs to represent relationships and dependencies among tools, enabling more realistic multi-step task scenarios.
- Back-instruct generates user instructions from sampled Tool Graph subgraphs, while structured sampling, self-critics, and human verification improve dataset quality.
- TaskEval quantitatively assesses LLMs across task decomposition, tool selection, and parameter prediction, providing more nuanced insights than current benchmark metrics.
- Experiments across different LLMs and additional dataset analyses show that TaskBench reflects capabilities across multiple dimensions and correlates highly with human evaluation.
2 Related Works
Prior tool-augmented benchmarks often rely on API documentation, templates, or simplified environments and incompletely model complex tool dependencies. TaskBench instead uses Tool Graphs and back-instruct to create more realistic, dependency-aware task automation evaluations.
- APIBench and ToolBench generate tasks from API documentation, but template-based sampling can limit their logical consistency.
- MetaTool focuses on whether a tool is needed, while ToolAlpaca does not adequately address dependencies between tools.
- AgentBench emphasizes agent-like behavior in simulated environments rather than tool interactions.
- TaskBench models real-world dependencies with a Tool Graph and aligns subgraphs with instructions through Back-Instruct.
3 TaskBench
TaskBench constructs realistic task-automation data by modeling tool dependencies as Tool Graphs, sampling subgraphs, and back-instructing LLMs to generate instructions and invocation structures. Quality control combines automated critics with human-supported verification, while dependency-aware graphs improve instruction naturalness and complexity.
- Tool Graph Construction: TASKBENCH simulates practical multi-task instructions across three tool-graph substructures: standalone nodes, sequential chains, and dependency-rich DAGs.The construction includes Hugging Face, multimedia, and Daily Life APIs domains.
- Tool Graph Construction: TaskBench models relationships among tools with a Tool Graph containing resource and temporal dependencies.Resource dependencies connect matching output and input types, while temporal dependencies encode execution order.
- Data Generation: Its dataset pipeline builds a tool repository graph, samples a node, chain, or DAG subgraph, and generates user instructions through back-instruct.The sampled structure preserves tool dependencies and determines the intended invocation pattern.
- Data Generation: Back-instruct then populates execution parameters and produces task-decomposition steps together with the final tool invocation graph.The initial sampled subgraph supplies an invocation skeleton but not the parameters required for execution.
- Quality Control: Quality control uses LLM-based or rule-based critics to check consistency between generated data and sampled tool-graph nodes and edges.The benchmark also incorporates human verification to assess logical consistency and alignment across task complexities.
- Dataset Quality: Back-Instruct achieved the highest Naturalness and Complexity scores, with edge information producing more natural and complex instructions than node-only prompting.The authors attribute this advantage to realistic resource and temporal dependencies represented in sampled subgraphs.
4 TaskEval
TaskEval evaluates task automation holistically across decomposition, tool selection, and parameter prediction using standardized metrics and prompting. Results show that model performance depends on task complexity and reasoning, while its automated scores correlate strongly with human judgments.
- Evaluation Framework: TaskEval measures task decomposition, tool selection, and parameter prediction as three stages of the complete task-automation process.A consistent prompt sequence decomposes requests, selects tools with parameters, and constructs a complete invocation graph.
- Task Decomposition: GPT-4 achieves approximately 10% higher Rouge-1 and Rouge-2 scores than other models for task decomposition.The performance gap widens as task complexity increases.
- Tool Selection: Edge prediction is approximately 20% lower in F1 than node prediction across models, making tool dependencies harder to capture than individual tools.Open-source models compete better on simpler node structures but show limitations with complex dependencies.
- Parameter Prediction: Parameter Name & Value F1 is generally lower for open-source models, while GPT-4 more robustly captures parameter names and values.Claude-2 and Gemini-Pro are competitive in some domains but remain below GPT-4.
- Influencing Factors: Code pre-training is associated with average improvements of 4.45% in tool prediction and 12.76% in parameter prediction across domains.The analysis identifies structured text as important for connecting automation stages.
- Validation: TASKEVAL scores correlate strongly with human evaluations, with average Kendall’s τ = 0.89 and Spearman’s ρ = 0.78.These correlations support alignment between the automated metrics and human judgments of task-automation quality.
5 Conclusion
The paper introduces TASKBENCH and TASKEVAL to evaluate LLM task automation across decomposition, tool selection, and parameter prediction. Its experiments characterize current LLM performance and influencing factors, while validating the benchmark’s effectiveness.
- Contributions: TASKBENCH aggregates tools and their interconnections into Tool Graphs and uses systematic metrics to evaluate task automation.The benchmark targets task decomposition, tool selection, and tool parameter prediction.
- Findings: Experiments report performance patterns for mainstream LLMs and analyze factors influencing autonomous task execution.The conclusion also states that the results validate TASKBENCH’s effectiveness for assessing task-automation performance.
- Future Work: Future work will expand domain coverage and develop more advanced metrics for studying LLM task automation and autonomous agents.
A.1 Case Study of Back-Instruct
The case study finds that Back-Instruct generates more comprehensive and interconnected tool usage than the compared instruction-generation methods. These examples reflect higher naturalness and complexity in the resulting instructions.
- Case Study: Back-Instruct produces examples with more comprehensive and interconnected tool usage than Back-Instruct w/o edges and Self-Instruct.
- Case Study: The generated examples reflect higher naturalness and complexity in instruction generation.
A.2.1 Error Analysis on TaskBench Dataset
Human evaluation found that TASKBENCH’s generated samples are not uniformly correct, with errors spanning instructions, parameters, and tool dependencies. The authors use these findings to motivate more detailed generation prompts and additional quality criteria.
- Nearly 12% of 148 randomly sampled TASKBENCH examples contained errors identified by the labeling team.The analysis found 18 erroneous samples across five error categories.
- Incorrect instructions include incomplete details or resources and impractical tasks that current tools cannot execute.
- Parameter errors arise from mismatched parameter types or incorrect values unsuitable for the task.
- Incorrect tool dependency errors involve linking or sequencing tools incorrectly for a task.
- The authors propose more detailed tool-use specifications and demonstrations, plus additional quality criteria beyond rule-based and LLM-based critics.
A.2.2 Error Analysis of Different LLMs in Predicting Tool Invocation Graph
The analysis compares tool-invocation-graph prediction failures across models and examines representative cases. GPT-4 has the fewest errors, while incorrect tool dependencies remain a recurring challenge and weaker models can omit required tools or compound errors.
- Incorrect tool dependency is the most common graph-prediction error across the evaluated models.
- GPT-4 demonstrates the fewest errors in all analyzed categories, indicating higher accuracy in predicting tool invocation graphs.
- GPT-3.5-turbo and Code Llama 13B produce progressively more prediction errors than GPT-4.
- In a representative audio-processing case, GPT-3.5-turbo and Code Llama 13B omitted the required Audio Downloader tool.
- Code Llama 13B additionally produced tool-parameter and tool-dependency errors in the representative case.
A.3 Metrics for Ranking Consistency
TaskBench evaluates ranking consistency against human judgments using Spearman’s ρ and Kendall’s τ. The reported correlations indicate strong agreement between the benchmark and human evaluation.
- Spearman’s ρ measures rank association, while Kendall’s τ compares concordant and discordant item pairs between two rankings.The rankings compared are human and TASKBENCH assessments of LLM task-automation capabilities.
- 0.89 Kendall’s τ and 0.78 Spearman’s ρ were the average values reported for TASKBENCH and human-evaluation rankings.These values are reported as evidence of very positive correlation between the two evaluations.
- The strong ranking agreement further validates TASKBENCH’s dataset-construction framework.
A.4 Analysis
The analysis examines how tool-graph complexity, demonstrations, data filtering, and evaluation prompts affect TASKBENCH construction and model assessment. Performance decreases with larger tool graphs, while demonstrations improve multiple automation dimensions.
- A.4.1 Different Number of Tools: 96.16% graph accuracy for single-node graphs fell to 39.31% for six-tool graphs and 25.00% for eight-node graphs.The analysis reports a clear downward trend in node-set, edge-set, and graph accuracy as tool count increases.
- A.4.1 Different Number of Tools: More tool nodes create harder automation cases because LLMs must process more extensive tool dependencies.The reported complexity affects predictive and analytical capabilities.
- A.4.2 Few-shot Setting: Increasing demonstrations significantly improves task decomposition, tool selection, and parameter prediction.The few-shot analysis evaluates these dimensions using the metrics defined for Rouge-L, node and edge F1, and parameter F1.
- A.4.2 Few-shot Setting: 20.78% and 21.82% improvements over zero-shot were observed for CodeLlama-13B in n-F1 and v-F1 under a 2-shot setting.
- A.5 Details about Back-Instruct and TaskBench: Rule-based and LLM-based critics filtered average proportions of 15.13% and 22.73%, leaving aligned samples of 61.76%, 62.71%, and 60.42% across three datasets.Human experts also revised and filtered the generated data.
- A.5 Details about Back-Instruct and TaskBench: Back-Instruct generates task steps, user requests, and tool-invocation graphs from tool nodes, edges, and their dependencies in strict JSON formats.The prompts require requests and task graphs to remain aligned with the supplied tool graph and tool parameters.