PaperlayerView exact revision on arXiv

Source-linked AI summary

ToolHop: A Query-Driven Benchmark for Evaluating Large Language Models in Multi-Hop Tool Use

Junjie Ye, Zhengyin Du, Xuesong Yao, Weijian Lin, Yufei Xu, Zehui Chen, Zaiyuan Wang, Sining Zhu, Zhiheng Xi, Siyu Yuan, Tao Gui, Qi Zhang, Xuanjing Huang, Jiecao Chen

arXiv:2501.02506v4cs.CL

TL;DR

Reliable evaluation of multi-hop tool use is needed to assess LLM comprehension, reasoning, and function-calling. ToolHop constructs a 995-query, 3,912-tool benchmark through query-driven tool creation, document refinement, and code generation, then evaluates 14 models across five families. GPT-4o leads with 49.04% accuracy, while model families exhibit different tool-use strategies and substantial room for improvement remains.

  • Problem

    Reliable datasets for evaluating LLM comprehension, reasoning, and function-calling in multi-hop tool use remain limited.

    Source: §1 · p. 1, §6 · p. 9

  • Method

    ToolHop uses query-driven tool creation, document refinement, and code generation to build 995 multi-hop queries with 3,912 locally executable tools and verifiable answers.

    Source: §1 · p. 2, §2.2 · p. 3

  • Results

    GPT-4o achieves 49.04% accuracy, while evaluation of 14 LLMs across five families reveals distinct tool-use strategies and substantial challenges.

    Source: §1 · p. 2

  • Takeaways & Limitations

    ToolHop provides a robust benchmark for comparing multi-hop tool use and offers insights for developing more effective approaches.

    Source: §1 · pp. 2–3, §7 · p. 9

    The work lacks an immediate strategy for enhancing multi-hop tool-use capabilities.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Effective evaluation of multi-hop tool use is critical for analyzing the understanding, reasoning, and function-calling capabilities of large language models (LLMs). However, progress has been hindered by a lack of reliable evaluation datasets. To address this, we present ToolHop, a dataset comprising 995 user queries and 3,912 associated tools, specifically designed for rigorous evaluation of multi-hop tool use. ToolHop ensures diverse queries, meaningful interdependencies, locally executable tools, detailed feedback, and verifiable answers through a novel query-driven data construction approach that includes tool creation, document refinement, and code generation. We evaluate 14 LLMs across five model families (i.e., LLaMA3.1, Qwen2.5, Gemini1.5, Claude3.5, and GPT), uncovering significant challenges in handling multi-hop tool-use scenarios. The leading model, GPT-4o, achieves an accuracy of 49.04%, underscoring substantial room for improvement. Further analysis reveals variations in tool-use strategies for various families, offering actionable insights to guide the development of more effective approaches. Code and data can be found in https://huggingface.co/datasets/bytedance-research/ToolHop.

1 Introduction

ToolHop addresses unreliable evaluation of multi-hop tool use by constructing a verifiable benchmark with query-driven tools and assessing current LLMs. Its evaluation reveals substantial performance limitations and distinct tool-use strategies across model families.

1 Introduction

  • Prior evaluations often lack genuine tool interdependencies and verifiable answers, introducing model bias and evaluation errors.Source: §1 · pp. 1–2
  • ToolHop expands each multi-hop query into a test case through tool creation, document refinement, and code generation.Source: §1 · p. 2, Figure 2 · p. 2
  • The benchmark evaluates 14 LLMs across five families to identify limitations and differences in multi-hop tool-use behavior.Source: §1 · pp. 2–3
  • The evaluated models achieved only 49.04% accuracy in multi-hop tool use, highlighting considerable room for improvement, while different model families exhibited distinct tool-use patterns and outcomes.Source: §1 · p. 2
  • Model families use different tool strategies: Qwen2.5 emphasizes parallel calls associated with hallucinations, whereas GPT leverages tool feedback.Source: §1 · p. 2
  • ToolHop contains 995 multi-hop queries and 3,912 locally executable tools for evaluating LLM tool use.The dataset supports diverse queries, meaningful interdependencies, detailed feedback, and verifiable answers.Source: §1 · p. 2

2 ToolHop

ToolHop formulates multi-hop tool use as sequentially dependent tool calls and constructs a dataset by creating, refining, and implementing tools from multi-hop queries. Its 995 queries and 3,912 executable tools span diverse domains, preserve meaningful dependencies, and support detailed feedback and verifiable answers.

2.1 Task Formulation

  • ToolHop models multi-hop tool use as sequential tool invocations whose intermediate answers become inputs to subsequent tools.Source: §2.1 · p. 3

2.2 Query-Driven Data Construction

  • The query-driven construction process creates interdependent tool documents, refines their functionality and parameter schemas, then generates verified executable functions.Source: §2.2 · pp. 3–4
  • 995 user queries and 3,912 locally executable tools form the ToolHop dataset, built from multi-hop questions decomposable into at least three atomic queries.Source: §2.2 · p. 4

2.3 Dataset Analysis

  • ToolHop covers 47 domains, including movies and television, academic subjects, and family relationships, to represent diverse real-world query types.Source: §2.3 · p. 4
  • Each query requires three to seven tools, preserving meaningful interdependencies and emphasizing multi-hop reasoning.Source: §2.3 · pp. 4–5
  • Document refinement increases average tool parameters from 3.49 to 5.91 and reduces simple string parameters by 12% in favor of structured types.Source: §2.3 · p. 5
  • Locally executable tools provide compiler-validated error feedback, while predefined queries and answers enable direct answer comparison and standardized final-answer validation.Source: §2.3 · p. 5

3 Experimental Setup

The experiments evaluate 14 LLMs from five model families on ToolHop using standardized function-call interfaces and deterministic decoding settings. The setup covers open- and closed-source models across multiple sizes.

3.1 Models

  • The evaluation includes 14 LLMs from the LLaMA3.1, Qwen2.5, Gemini1.5, Claude3.5, and GPT families.Source: §3.1 · pp. 5–6

3.2 Implementation Details

  • Open-source models use chat templates with greedy decoding, while closed-source models use APIs at temperature 0.Source: §3.2 · p. 6
  • All tools are implemented through the models’ function-call interfaces for consistency across evaluations.Source: §3.2 · p. 6

4 Main Results

Tool use improves LLM answer correctness on multi-hop queries, but substantial accuracy and invocation-error problems remain. Model families differ in their reliance on tools, scaling behavior, and sensitivity to invocation errors.

4.2 Evaluation Observations

  • 49.04% answer correctness is achieved by GPT-4o in mandatory tool use, despite tools increasing average LLM correctness by 12.29%.Source: §4.2 · p. 7
  • GPT-family accuracy improves by 23.59% with tool use, while 9.45% of mandatory-tool queries exhibit hallucinations.Source: §4.2 · p. 7
  • Mandatory and free-choice tool-use accuracy differs by only 0.62%, indicating that most evaluated models generally choose to use available tools.Source: §4.2 · p. 7
  • Qwen2.5-Instruct-72B improves correctness by 27.54% through tool use, whereas Claude3.5 performs especially well without tool reliance in direct answering.Source: §4.2 · p. 7
  • Larger models generally improve answer correctness and reduce invocation error in mandatory tool use across both open- and closed-source families.Source: §4.2 · p. 7

5 Further Studies

Further analysis links family-level performance differences to tool-call support, parallelism, reasoning behavior, and feedback use. These findings motivate better tool compatibility, controlled parallelism, and richer error-correction strategies.

5 Further Studies

  • LLaMA3.1 and Gemini1.5 underperform because incomplete tool support restricts mixed text-and-call reasoning or handling of union-type parameters.Source: §5 · p. 7
  • Qwen2.5-Instruct-14B and -32B use parallel calls on 70.1% and 75.08% of queries, while the 72B model uses 3.82%, avoiding parameter hallucinations associated with forced parallelism.Source: §5 · pp. 7–8
  • Claude3.5-Haiku applies chain-of-thought reasoning to 64.92% of direct-answer queries, supporting strong accuracy without full reliance on tool outputs.Source: §5 · p. 8
  • Detailed feedback improves GPT-family correction after invocation errors, whereas replacing it with a simple failure hint reduces final-answer correctness by 20.66%.Source: §5 · p. 8, Table 5 · p. 8
  • The authors recommend robust support for complex tools, safer optimization of parallelism, and strategies that exploit rich feedback for error correction.Source: §5 · p. 8

6 Related Works

Tool-use evaluations have progressed from single-step simulations toward process-oriented assessment, but reliable multi-hop evaluation remains difficult. ToolHop addresses this gap with query-driven construction and predefined verifiable answers.

  • Its query-driven approach extends locally executable tools through multi-hop queries while improving diversity and dependency consistency.Source: §6 · p. 8
  • Existing benchmarks often rely on manual verification, final-answer checks, or model-based process evaluation, limiting objectivity and reliability.Source: §6 · pp. 8–9
  • ToolHop focuses on multi-hop tool use and uses query-driven construction with predefined verifiable answers for accurate assessment.Source: §6 · p. 9

7 Conclusion

ToolHop introduces a query-driven benchmark for evaluating multi-hop tool use across diverse LLM families. Its analyses provide insights into model behavior and establish a foundation for future tool-based reasoning research.

  • ToolHop benchmarks 14 LLMs across five families using diverse queries, meaningful interdependencies, executable tools, detailed feedback, and verifiable answers.Source: §7 · p. 9
  • The benchmark’s family-level analyses offer actionable insights for improving multi-hop tool-use performance.Source: §7 · p. 9

Limitations

The dataset evaluates multi-hop tool-use performance but does not provide an immediate strategy for improving these capabilities. Its scalable construction scheme could support future training-data development.

  • The scalable construction scheme can be adapted to create training datasets, although improvement from targeted training remains a hypothesis.Source: Limitations · p. 9

A Comparison of Various Benchmarks

ToolHop is designed to address benchmark gaps in multi-hop tool-use evaluation through comprehensive dataset features and a query-driven construction process. The section also specifies prompts, domain classification, and the evaluated model families.

A Comparison of Various Benchmarks

B Prompt for Data Construction

C Prompt for Domain Classification

D Details for Models

  • The evaluation covers 14 open- and closed-source LLMs from five families and analyzes their performance in multi-hop tool use, including LLaMA3.1, Qwen2.5, Gemini1.5, and Claude3.5.Source: D Details for Models · p. 18

E Performance under Various Inference Frameworks

ToolHop remains challenging across ReAct, CodeAct, and built-in function-calling frameworks, although model families differ in their framework preferences and stability.

  • None of the evaluated frameworks effectively solves ToolHop’s multi-hop tool-use challenges.Performance varies across ReAct, CodeAct, and Free settings, but the overall difficulty persists.Source: E Performance under Various Inference… · p. 19, Table 11 · p. 19
  • Open-source models benefit more from ReAct because its step-by-step reasoning improves planning before tool invocation.Their direct function-calling behavior often invokes tools without explicit reasoning, leaving built-in capabilities room for improvement.Source: E Performance under Various Inference… · p. 19
  • Closed-source models are more stable across frameworks, with GPT favoring ReAct and Claude 3.5 favoring CodeAct.The passage attributes these preferences likely to differences in training data and methodology.Source: E Performance under Various Inference… · p. 19

F Case Study

Case studies illustrate family-specific tool-use behaviors, while the construction examples show how refinement and code generation produce richer, executable tools.

F Case Study

  • ToolHop case studies compare family-specific behaviors, including Qwen2.5 parallel calls, Claude 3.5 chain-of-thought reasoning, and GPT feedback refinement.The examples span direct-answer, mandatory-tool-use, and feedback-driven settings.Source: Figure 7 · p. 20, F Case Study · p. 20, Figure 8 · p. 21

G Examples of Tool Documents

  • Document refinement expands a preliminary tool by adding parameters and more diverse parameter types.The refined album-release tool accommodates album versions, release regions, languages, and output formats.Source: G Examples of Tool Documents · p. 24

H Examples of Code Implementations

  • Code generation turns refined tool documents into locally executable functions with validation, parameter-dependent release-date selection, feedback, and exception handling.The implementation checks required and matching parameters, retrieves album data, selects dates by version, region, or language, and supports date or text output.Source: H Examples of Code Implementations · pp. 26–27

Related papers

Loading 2501.02506v4…