PaperlayerView exact revision on arXiv

Source-linked AI summary

ToolBeHonest: A Multi-level Hallucination Diagnostic Benchmark for Tool-Augmented Large Language Models

Yuxiang Zhang, Jing Chen, Junjie Wang, Yaxin Liu, Cheng Yang, Chufan Shi, Xinyu Zhu, Zihao Lin, Hanwen Wan, Yujiu Yang, Tetsuya Sakai, Tian Feng, Hayato Yamana

arXiv:2406.20015v2cs.CLcs.AI

TL;DR

Tool-augmented LLMs can hallucinate when they use tools for tasks that cannot be solved with the available toolset, and existing benchmarks leave these failures incompletely assessed. ToolBH diagnoses such hallucinations across stages and toolset scenarios. Models remain challenged on the benchmark, and its error analysis identifies solvability hallucination as a primary difficulty.

  • Problem

    Tool-use hallucinations remain insufficiently understood because existing benchmarks generally assume complete, usable toolsets, despite models sometimes using wrong or non-existent tools when tasks are unsolvable.

    Source: §1 · pp. 1–2

  • Method

    ToolBH evaluates tool-use hallucinations by depth—solvability detection, solution planning, and missing-tool analysis—and breadth across three toolset scenarios.

    Source: §1 · p. 2, §3.1 · p. 3

  • Results

    Across ToolBH, models struggle with unsolvable tool use and primarily exhibit solvability hallucination; Gemini-1.5-Pro scores 45.3 overall.

    Source: §4.3 · p. 6, §6 · p. 9

  • Takeaways & Limitations

    ToolBH provides detailed error-pattern analyses in unsolvable tool-use scenarios that the authors say can guide future research.

    Source: §1 · p. 2

    The evaluation covers only seven models and excludes a broader range of open-source models, including cross-linguistic models, which may bias the findings.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Tool-augmented large language models (LLMs) are rapidly being integrated into real-world applications. Due to the lack of benchmarks, the community has yet to fully understand the hallucination issues within these models. To address this challenge, we introduce a comprehensive diagnostic benchmark, ToolBH. Specifically, we assess the LLM's hallucinations through two perspectives: depth and breadth. In terms of depth, we propose a multi-level diagnostic process, including (1) solvability detection, (2) solution planning, and (3) missing-tool analysis. For breadth, we consider three scenarios based on the characteristics of the toolset: missing necessary tools, potential tools, and limited functionality tools. Furthermore, we developed seven tasks and collected 700 evaluation samples through multiple rounds of manual annotation. The results show the significant challenges presented by the ToolBH benchmark. The current advanced models Gemini-1.5-Pro and GPT-4o only achieve total scores of 45.3 and 37.0, respectively, on a scale of 100. In this benchmark, larger model parameters do not guarantee better performance; the training data and response strategies also play crucial roles in tool-enhanced LLM scenarios. Our diagnostic analysis indicates that the primary reason for model errors lies in assessing task solvability. Additionally, open-weight models suffer from performance drops with verbose replies, whereas proprietary models excel with longer reasoning.

1 Introduction

Tool-using LLMs can hallucinate by choosing wrong tools or inventing tools when a task cannot be solved with the available toolset. ToolBH diagnoses these failures across multiple levels and toolset scenarios; the reported results show substantial model errors.

Benchmark motivation

  • ToolBH addresses a benchmark gap: prior tool benchmarks generally assume complete, usable toolsets, while MetaTool tests solvability without diagnosing broader hallucination phenomena.The benchmark evaluates hallucinations by depth and breadth, drawing on diagnostic procedures and prior hallucination benchmarks.Source: §1 · pp. 1–2

Benchmark design

  • ToolBH diagnoses hallucinations through solvability detection, solution planning, and missing-tool analysis across missing necessary, potential, and limited-functionality tool scenarios.The authors designed seven tasks and collected 700 query, toolset, and solution samples through multiple rounds of human annotation.Source: §1 · p. 2

Results

  • Llama-3-70B reaches only 32% of Gemini-1.5-Pro’s total score and 40% of GPT-4o’s, while open-weight models show more solvability hallucinations and proprietary models more instrumental reasoning errors.Proprietary models also use fewer non-existent tools, suggesting better tool recognition; open-weight models more often misjudge task complexity and feasibility.Source: §1 · p. 2

Contribution

  • ToolBH provides detailed analyses of error patterns in unsolvable tool-use scenarios, which the authors say can guide future research.The introduction identifies assessing whether a task is solvable with the available tools as a central source of hallucination.Source: §1 · pp. 1–2

2 Related Work

Prior tool benchmarks largely assume users provide all required tools, while hallucination benchmarks do not assess tool interaction. ToolBH is proposed to evaluate tool-use hallucinations across scenarios and investigate their causes.

2.1 Tool-based Benchmarks

  • Existing tool benchmarks evaluate planning, retrieval, and API use, but generally assume users provide every required tool with an explicit description; limited reliability tests do not explain why models return “None.”ToolBH evaluates models across diverse scenarios and further investigates hallucinations to help identify underlying causes.Source: §2.1 · pp. 2–3

2.2 Evaluating Hallucinations in LLMs

  • TruthfulQA, HaluEval, and FACTOR assess truthfulness or factuality without tool interaction, motivating a multi-level benchmark for hallucinations in tool-augmented LLMs.The paper draws on generative and discriminative approaches, as well as methods used in coding and medical diagnosis.Source: §2.2 · p. 3

3 The ToolBH Benchmark

ToolBH targets unsolvable tool-use tasks, where models may believe they can proceed despite missing capabilities. It evaluates these cases through staged diagnostics and varied toolset conditions, using a curated benchmark of 700 samples.

3.1 Design Philosophy; 3.2 In-depth: Multi-level Diagnostic Task

  • ToolBH focuses on unsolvable long-tail tool-use tasks and diagnoses them through progressively more detailed response stages, alongside scenarios defined by tool characteristics.The three diagnostic stages are solvability detection, solution planning, and missing-tool analysis.Source: §3.1 · p. 3, §3.2 · p. 3

3.2.1 Level-1: Solvability Detection

  • Level 1 treats solvability detection as binary classification: the model uses the query and tool descriptions to label the task solvable or unsolvable, scored by exact match.The model is expected to assess the task against the provided toolset.Source: §3.2.1 · p. 3

3.2.2 Level-2: Solution Planning

  • Level 2 asks models to divide requests into sub-goals and assign tools, using progress rate to score the predicted tool sequence against the ground truth until the first mismatch.An “UnsolvableQuery” tool represents sub-goals that the available tools cannot achieve.Source: §3.2.2 · pp. 3–4

3.2.3 Level-3: Missing-Tool Analysis

  • Level 3 requires a rationale for each subtask and, for unsolvable sub-goals, a description of the missing tool’s function; it evaluates progress and function matching.The matching procedure searches a tool-description database, then uses embedding cosine similarity when an exact description match is not found; sub-goal matching scores are averaged per sample.Source: §3.2.3 · p. 4

3.3 In-breadth: Tool-based Hallucination-inducing Tasks

  • The benchmark organizes toolset conditions into missing necessary tools, potential tools, and limited-functionality tools, with tasks designed to induce hallucinations in each.The conditions vary whether required tools are absent, tools in an environment are not provided, or available tools lack required functions or capabilities.Source: §3.3 · pp. 4–5, §3.3.1 · p. 5, §3.3.2 · p. 5

3.3.1 Missing Necessary Tools (MNT)

  • In missing-necessary-tool tasks, one ground-truth tool is removed; subtasks vary between single-step use and multi-step use, with or without tool repetition.These tasks become unsolvable when the toolset lacks a tool needed for completion.Source: §3.3.1 · p. 5

3.3.2 Potential Tools (PT); 3.3.3 Limited Functionality Tools (LFT)

  • Potential-tool tasks remove a required tool and introduce OS or web context, while limited-functionality tasks add requirements that exceed available functions or tool capabilities.Potential-tool examples include system commands and web technologies; limited-functionality examples include missing functions and stricter translation or resource requirements.Source: §3.3.2 · p. 5, §3.3.3 · p. 5

3.4 Data Curation

  • The curation process synthesizes and filters samples through repeated annotation rounds, then manually balances each of seven subtasks at 50 solvable and 50 unsolvable samples, totaling 700.The authors report selecting varied samples based on tool descriptions and task contexts before the final manual review.Source: §3.4 · pp. 5–6

4 Experiments

Experiments evaluate 14 proprietary and open-weight models on ToolBH under controlled settings. Gemini-1.5-Pro leads overall, while models also show challenges across diagnostic levels and toolset scenarios.

4.1 Baselines; 4.2 Evaluation Settings

  • Fourteen models—seven proprietary and seven open-weight—were evaluated at temperature 0.0, with proprietary APIs and open-weight inference using vLLM.Open-weight experiments used four NVIDIA RTX 6000 Ada 48G GPUs.Source: §4.1 · p. 6, §4.2 · p. 6

4.3 Main Results

  • 45.3 overall, Gemini-1.5-Pro leads the ToolBH leaderboard, outperforming competitors and exceeding GPT-4o by 22.4%.Among open-weight models, Llama-3-70B performs best but is comparable only to GPT-3.5-Turbo.Source: §4.3 · pp. 6–7, Table 2 · p. 7
  • All models struggle with L3 missing-tool analysis; in limited-functionality scenarios, GPT-4-0613 and GPT-4-Turbo score 0.0 on L3-MS.The authors suggest that L3's complex reasoning demands and extensive intermediate text may contribute to these errors.Source: §4.3 · p. 7

In evaluation breadth, all models face extra challenges in the LFT (limited functionality tools)

  • Potential tools are associated with lower L1-EM and L2-PR than missing-necessary-tool scenarios, while limited functionality brings sharper declines from L1 to L2 and L3.The authors suggest potential tools can confuse solvability judgments, while restricted tool functionality may lead models to misjudge solvability despite necessary tools being present.Source: In evaluation breadth, all models face… · p. 7
  • Performance does not track parameter count alone: training data and response strategy also matter, with open-weight scores declining for overly long replies while proprietary scores rise with length.The authors also identify data quality and long-context reasoning consistency as areas for attention.Source: In evaluation breadth, all models face… · pp. 7–8

5 Discussion

The discussion compares model performance across ToolBH's depth and breadth dimensions, examines recurring error patterns, and contrasts results on solvable and unsolvable tasks. Errors concentrate in judging solvability, while the open-weight/proprietary performance gap is especially large on unsolvable tasks.

5.1 Model Performance: A Depth and Breadth Perspective

  • Across depth, GPT-4-0613 (52.1) and GPT-4o (44.8) perform well in L1 but drop in L2 and L3; Gemini-1.5-Pro falls from L1 (62.6) to L2 (55.5), then rebounds at L3 (57.9).Across breadth, all models scored 40.0–55.0 in MNT, but scores fell to 20.0–30.0 in PT and LFT.Source: §5.1 · p. 8

5.2 Error Analysis

  • Solvability hallucination—failing to recognize an unsolvable task—is the most common error, particularly at Level 3; non-existent tools are frequent in Level-2 and LFT cases for open-weight models but rare for proprietary models.Figure 4 presents error counts for proprietary and open-weight models.Source: Figure 4 · p. 8, §5.2 · p. 8
  • Models also misidentify which subtasks are unsolvable, use unnecessary tools, or reason with tools in the wrong sequence; wrong tool reasoning is more prevalent in proprietary models.The paper attributes the smaller number and percentage of errors in open-weight models partly to their more frequent solvability hallucinations, which prevent subsequent tasks.Source: §5.2 · pp. 8–9

5.3 Solvable Tasks Versus Corresponding Unsolvable Tasks

  • 91.1% of proprietary-model performance for open-weight models on solvable tasks falls to 39.4% on unsolvable tasks after human curation of corresponding task pairs.The authors argue that unsolvable tests are important for comparing tool-using models, since recent benchmarks commonly assume complete, explained tool lists.Source: Table 3 · p. 9, §5.3 · p. 9

6 Conclusions

The conclusion presents ToolBH as a benchmark for diagnosing tool-use hallucinations across diagnostic depth and toolset breadth. Its analysis finds that solvability assessment is a primary difficulty and identifies training data, response strategy, and long-context reasoning consistency as relevant performance factors.

6 Conclusions

  • ToolBH diagnoses hallucinations in tool-augmented LLMs through multi-level evaluation and three scenarios that vary toolset characteristics.The conclusion frames this as a resource for studying real-world performance of tool-augmented LLMs.Source: §6 · p. 9
  • The primary error pattern is solvability hallucination: models misread queries or tool functions and mistakenly conclude that tasks are solvable.Source: §6 · p. 9
  • The conclusion identifies training-data quantity and quality, response strategy, and long-context reasoning consistency as factors to address in model performance.Source: §6 · p. 9

Limitations

The analysis is limited in model coverage and uses tool descriptions that omit API parameters, which may make benchmark results differ from real-world applications.

Limitations

  • The evaluation omits broader open-source families, including cross-linguistic models, and focuses on dense and mixture-of-experts architectures; the seven-model scope may bias findings.The authors plan to open-source the data and methods to simplify evaluation.Source: Limitations · p. 9
  • Tools are represented by names without API parameters, a design choice that could create discrepancies between benchmark results and real-world applications.The framework can incorporate parameterized APIs, allowing users to swap tools under its guidelines.Source: Limitations · pp. 9–10

Ethical Considerations

Synthetic data and a single-country annotation team leave potential ethical and regional biases that manual filtering may not fully remove.

Ethical Considerations

  • Synthetic data may contain ethical biases, and manual filtering does not guarantee their complete removal.A same-country annotation team may also hinder filtering biases prevalent in other regions; the authors recommend additional bias-mitigation strategies.Source: Ethical Considerations · p. 10

A.1 Design Considerations of the Multi-level Evaluation Methods

The appendix explains ToolBH’s staged diagnosis of tool-use hallucinations, alongside its sample-filtering and prompt-construction procedures.

A.1 Design Considerations of the Multi-level Evaluation Methods

  • Level 1 judges overall solvability, Level 2 decomposes the task to locate problematic steps, and Level 3 analyzes missing-tool functions for unsolvable subgoals.This progression moves from macro-level judgments to specific tool deficiencies that can inform toolset refinement.Source: §A.1 · p. 13

A.2 Filtering Criteria

  • Filtering combines similarity checks, code-based checks for clear requirements and tool coverage, and manual review for clarity, logic, and potential ethical bias.Synthetic samples with task or tool similarity scores greater than 0.8 are excluded.Source: §A.2 · p. 13

A.3 Generation Prompts

  • Prompts for the benchmark’s hallucination-inducing subtasks are detailed in Figures 9–15.Source: §A.3 · p. 13

A.4 Evaluation Prompts

  • Evaluation prompts wrap task descriptions and provided tools in XML tags and require tagged answers while discouraging redundant output.The three levels respectively assess solvability, prioritize subtasks using available tools, and identify missing tool functionalities.Source: §A.4 · p. 14

B Comparisons with Existing Benchmarks

ToolBH reports wider model-performance differences and more granular error diagnoses than prior benchmarks, then examines embedding robustness and response-length patterns.

B Comparisons with Existing Benchmarks

  • 57.7% is the GPT-3.5/GPT-4 performance gap on ToolBH, and GPT-3.5 averages 18 on ToolBH Level 1 versus 50.35 on MetaTool’s reliability task.The benchmark’s multi-level analysis identifies task-solvability judgment as a core issue, beyond common “Wrong Tools” errors, and supports more nuanced hallucination analysis.Source: B Comparisons with Existing Benchmarks · p. 14

C.1 Version for Tested Proprietary Models

  • The paper provides detailed versions of all tested proprietary models to ensure reproducibility of results, with supplied entries for Gemini-1.0-Pro, Gemini-1.5-Pro, GPT-3.5-Turbo, and GPT-4-Turbo.The listed versions are gemini-1.0-pro-002, gemini-1.5-pro-preview-0514, gpt-3.5-turbo-0125, and gpt-4-turbo-2024-04-09.Source: §C.1 · pp. 14–15

C.2 Different Embedding Models in Level-3 Matching Score

  • L3-MS scores show negligible variance across the tested proprietary text-embedding-004 and open-weight all-MiniLM-L6-v2 models, so all-MiniLM-L6-v2 is selected as the default.Source: §C.2 · p. 15

D.1 How Does Response Length Impact Performance?

  • The response-length analysis standardizes lengths and performance scores, then uses scatter plots and regression lines to inspect their relationships across evaluation levels and scenarios.Figures 5 and 6 show open-weight and proprietary models, respectively; rows represent performance indicators and columns represent tool-availability scenarios.Source: §D.1 · p. 15, Figure 5 · p. 16, Figure 6 · p. 17

D.1.1 Open-weight Models: Negative Correlation with Response Length

  • Open-weight models’ performance trends downward as responses lengthen across all evaluated levels and tool-availability scenarios.The authors suggest redundant conversational content may obscure critical steps and tool choices, while long-text consistency and logical coherence remain improvement targets.Source: §D.1.1 · pp. 15, 18

D.1.2 Proprietary Models: Positive Correlation with Response Length

  • Proprietary models’ longer responses are generally associated with higher scores, except for slight downward L3-MS trends in MNT and PT; they can nevertheless lose focus on long L3-MS missing-tool descriptions.The authors associate stronger contextual consistency in extended responses with pertinent information and improved task performance.Source: §D.1.2 · p. 18

D.2 Additional Error Analysis for LLMs

Additional error analysis examines hallucination patterns across model groups and tool scenarios, including models’ tendency to assume unavailable tools are present. The authors suggest these distributions may help guide targeted optimization.

D.2 Additional Error Analysis for LLMs

  • Open- and proprietary-model error distributions across Level-2 and Level-3 in three scenarios align with the analyses in Section 5.2.The analysis is intended to enrich understanding of tool-use hallucinations and inform targeted optimization.Source: §D.2 · p. 18

E.1 Potential Tools Failure

  • Models sometimes assume unavailable tools exist in textually fictionalized environments, such as trying unprovided Linux commands like “rm” and “ufw”.GPT-4o is also reported as susceptible to potential-tool hallucination.Source: §E.1 · p. 18
  • The authors suggest operating-system knowledge in training data may activate associations with restricted tools, challenging adherence to the provided toolset.This is offered as a possible explanation for outputs such as the restricted command “ufw”.Source: §E.1 · p. 18

E.2 Different Response Strategies

The analyses examine how response strategies, continued prompting, and changing task context relate to hallucination behavior. They report higher Level-3 solvability hallucinations, prompt-sensitive responses, and a contrast in contextual consistency between Gemini-1.5-Pro and GPT-4o.

E.2 Different Response Strategies

  • Solvability hallucinations are more prevalent in Level-3 than Level-2 across proprietary and open-weight models, while verbose replies are associated with lower accuracy regardless of parameter size.The authors suggest training data and response preferences may contribute to verbosity, and note Llama-3-8B exceeds Llama-2-70B in overall score.Source: §E.2 · pp. 18, 21

E.3 Analysis of Continual Instructions

  • GPT-4 can answer an initially unsolved problem correctly after an additional prompt, whereas GPT-4o still shows hallucinations in the continued-instruction analysis.The authors hypothesize that training data and multi-turn dialogue strategies may contribute to this behavior.Source: §E.3 · p. 21, Table 5 · p. 22

E.4 Consistency in Contextual Reasoning

  • Gemini-1.5-Pro preserves an unsolvable subgoal from Level-2 into Level-3, while GPT-4o bypasses it during subsequent planning.The authors hypothesize that Gemini’s solvability understanding is less affected by task-demand or reasoning-context changes; GPT-4o’s planning may be influenced by its generated context.Source: §E.4 · p. 21, Table 9 · p. 26
Loading 2406.20015v2…