PaperlayerView exact revision on arXiv

Source-linked AI summary

ToolFailBench: Diagnosing Tool-Use Failures in LLM Agents

Harsh Soni

arXiv:2607.04686v1cs.CLcs.AIcs.SE

TL;DR

Aggregate pass rates can make tool skipping, ignoring a tool result, and fabricating extra information look similar even though they are different failure modes. ToolFailBench tests 1,000 tasks across five professional domains, using contradictory tool-return traps and no-tool controls, and labels traces with a failure taxonomy and majority voting by a rule classifier and two LLM judges. Aggregate scores hide distinct failure mechanisms: most models remain disciplined on control tasks, while Llama-3.1 models show an Always-Call pattern; Llama-3.1-70B differs from Qwen2.5-72B by 89 percentage points on control-task accuracy.

  • Problem

    Aggregate pass rates can make tool skipping, ignoring a tool result, and fabricating extra information look similar even though they are different failure modes.

    Source: §1 · p. 1

  • Method

    ToolFailBench tests 1,000 tasks across five professional domains, using contradictory tool-return traps and no-tool controls, and labels traces with a failure taxonomy and majority voting by a rule classifier and two LLM judges.

    Source: §1 · p. 1, §3.2 · p. 3, §4.3 · p. 4

  • Results

    Aggregate scores hide distinct failure mechanisms: most models remain disciplined on control tasks, while Llama-3.1 models show an Always-Call pattern; Llama-3.1-70B differs from Qwen2.5-72B by 89 percentage points on control-task accuracy.

    Source: §6 · p. 7

  • Takeaways & Limitations

    Tool-use evaluation should distinguish whether agents call tools when needed, use returned evidence faithfully, and avoid tools when none is needed.

    Source: §1 · p. 2

    The single-turn format does not capture multi-turn failures such as tool chaining, recovery from earlier mistakes, or state updates across interactions.

    Source: §6 · p. 7

Abstract

from arXiv · show

Tool calling is central to modern language model agents, but aggregate benchmark scores often hide where tool use fails. A model that never calls a needed tool and a model that calls the tool but ignores the result can look similar under final task accuracy. We introduce ToolFailBench, a diagnostic benchmark for measuring tool-use failures across 1,000 tasks in finance, medicine, law, cybersecurity, and real estate. Tool-required tasks return values the model wouldn't guess, forcing it to trust the tool while control tasks attach the same tools but should be answered directly. We label each trace with Tool-Skip, Result-Ignore, Output-Fabrication, and Unnecessary-Tool-Use, using a rule classifier and two LLM judges aggregated by majority vote. Across 19 headline models, the best reaches 86.33% Clean Tool-Use Rate, showing that faithful tool use is not saturated. More importantly, models with similar aggregate scores fail in different ways: most stay disciplined on no-tool controls, while Llama-3.1 models show an Always-Call pattern, and at the same parameter scale Llama-3.1-70B and Qwen2.5-72B differ by 89 percentage points on control-task accuracy. Tool-use evaluation should measure not only whether agents call tools, but whether they use tool outputs correctly and avoid tools when none is needed.

1. Introduction

ToolFailBench targets a gap in aggregate tool-use evaluation: similar scores can conceal different failures in calling tools and using their returns. It tests these behaviors across required-tool and no-tool tasks, finding distinct model failure profiles.

1. Introduction

  • Aggregate scores can make tool skipping, ignoring a tool result, and fabricating information look similar despite representing different failures.Existing benchmarks measure function-call correctness, end states, API tool-use success, or long-horizon task execution.Source: §1 · p. 1
  • ToolFailBench tests 1,000 tasks across finance, medicine, law, cybersecurity, and real estate, using contradictory tool returns and no-tool controls.The tool-required tasks test whether models use returned values rather than plausible stored priors; controls test whether they answer directly without unnecessary calls.Source: §1 · p. 1
  • Its failure-mode taxonomy separates Tool-Skip, Result-Ignore, Output-Fabrication, and Unnecessary-Tool-Use, with labels assigned by majority vote across a rule classifier and two LLM judges.Tool-required and control tasks have distinct failure labels.Source: §1 · pp. 1–2
  • 89 percentage points separate Llama-3.1-70B and Qwen2.5-72B on control-task accuracy despite their same parameter scale, while Llama-3.1 models show an Always-Call pattern.Most models form a disciplined low-UTR cluster, and the best of 19 headline models reaches 86.33% Clean Tool-Use Rate.Source: §1 · pp. 1–2

2. Related Work

Prior tool-use work measures call validity, task completion, and diagnostic failure dimensions. ToolFailBench focuses on whether models faithfully use tool returns in their final answers.

2. Related Work

  • ToolFailBench focuses on whether a model uses returned tool evidence in its final answer, complementing benchmarks centered on tool selection and call validity.Prior examples include API-Bank, ToolLLM, and BFCL.Source: §2 · p. 2
  • End-to-end agent benchmarks test realistic task completion, but their final success scores can mix planning, retrieval, tool choice, execution, and answer integration.ToolFailBench is narrower by design, isolating faithful use of returned tool evidence.Source: §2 · p. 2
  • Diagnostic work decomposes tool-use capabilities and studies instability, output errors, hallucination, honesty, or overuse; ToolFailBench targets post-call faithfulness under controlled returns.It separates Tool-Skip, Result-Ignore, Output-Fabrication, and Unnecessary-Tool-Use.Source: §2 · p. 2

3. ToolFailBench

ToolFailBench isolates where the tool-use loop breaks by testing required-tool and no-tool tasks, assigning failure-mode labels, and reporting metrics across distinct task sets.

3. ToolFailBench

  • ToolFailBench evaluates whether agents use tool results in final answers, checking tool decisions, returned evidence, and answers across 1,000 tasks and assigning failure-mode labels.Labels combine a deterministic rule classifier with two independent LLM judges by majority vote.Source: §3 · p. 2

3.1. Benchmark Design

  • The benchmark has 1,000 tasks across five domains, split evenly by domain into 150 tool-required parametric traps and 50 no-tool controls.Trap returns differ from plausible prior values; “memorized prior” is a task-design term, not a claim about the model’s internal source of a value. Each task stores expected tools and answers, and traps store both returned and likely memorized values.Source: §3.1 · pp. 2–3, Figure 1 · p. 3

3.2. Failure Mode Taxonomy

  • For tool-required tasks, Tool-Skip means no valid executed call, Result-Ignore means the return is unused, and Output-Fabrication means invented structured information is added.On controls, Unnecessary-Tool-Use marks a call when none is needed, while Wrong-Answer marks an incorrect direct answer.Source: §3.2 · p. 3

3.3. System Prompt and Output Format

  • All models receive the same domain-specific system prompt, which specifies when to call tools, when to answer directly, and how to format answers.The uniform format supports cross-model and cross-domain comparisons and stable rule-classifier checks.Source: §3.3 · p. 3

3.4. Metrics and Detection

  • Metrics separate tool-required, called-tool, and control tasks; TSR and CTUR track needed calls, RIR and OFR post-call failures, and UTR and CTRL-Acc no-tool behavior.On Required tasks, TSR, CTUR, RIR, and OFR sum to 100%; trace labels are assigned by a rule classifier and two LLM judges, with ties resolved by the rule classifier.Source: §3.4 · pp. 3–4, Table 1 · p. 4

4. Experimental Setup

The experiments compare models across a shared task suite and inference protocol, then use two differently prompted LLM judges alongside deterministic rules to label traces.

4.1. Models

  • The headline comparison includes 19 models that completed the full suite with valid traces, spanning closed-API, open-weight, and reasoning-oriented systems.Twenty-two models were run; two appear as non-headline tool-skip cases and one was excluded for invalid raw-token output.Source: §4.1 · p. 4

4.2. Inference Protocol

  • Every model receives the same tasks, prompts, tool schemas, and decoding settings; the two-step protocol separates the tool decision from final answer writing.Temperature is 0 and the maximum is 1,024 tokens; valid calls trigger a mock return, after which the model answers with tool choice set to none.Source: §4.2 · p. 4

4.3. Judge Configuration

  • Two differently prompted LLM judges join the rule classifier in a majority-vote label, with ties resolved by the rule classifier.The Qwen judge is stricter for both Qwen-family and non-Qwen models, with mean gaps of -5.99 and -6.16 percentage points, respectively.Source: §4.3 · p. 4

5. Results and Analysis

ToolFailBench results show that similar aggregate scores can conceal distinct tool-use behaviors, while labeling robustness and result faithfulness also vary across evaluation settings. Most models remain disciplined on no-tool controls, but Llama-3.1 models show an Always-Call pattern, and finance has the highest median result-ignore rate.

5.1. Main Leaderboard

  • 86.33% ensemble CTUR is Grok-4.3’s highest score, with the strongest models forming a statistically close leading cluster.Their 95% Wilson confidence intervals overlap; Grok-4.3 is not significantly above Grok-4-1-Fast-Reasoning (p = 0.23) and is only marginally above Qwen2.5-32B-Instruct (p = 0.05).Source: Table 2 · p. 5, §5.1 · p. 5

5.2. Failure Profiles Reveal Different Model Behaviors

  • UTR stays at or below 1.61% and CTRL-Acc above 95% across the disciplined models, which rarely call tools for control questions.The cluster includes closed-API models, Qwen models, Gemma, GLM-4.7-Flash, QwQ-32B, and DeepSeek-V4-Flash.Source: §5.2 · p. 5, Figure 2 · p. 5
  • 77.73% UTR and 8.91% CTRL-Acc mark Llama-3.1-70B’s Always-Call pattern; Llama-3.1-8B reaches 98.39% UTR and 0.00% CTRL-Acc.The two models form a distinct regime, often calling tools even when questions should be answered directly.Source: §5.2 · pp. 5–6

5.3. Same Scale Does Not Mean Same Tool Discipline

  • Llama-3.1-70B and Qwen2.5-72B differ by 89 percentage points in control-task accuracy despite their similar parameter scale.Scaling Llama-3.1 from 8B to 70B improves CTUR and reduces RIR but does not fix control-task behavior; the authors treat the Always-Call pattern as family-specific.Source: §5.3 · p. 6

5.4. Labeling Robustness

  • Mean ensemble CTUR was 75.02%, versus 71.05% rule CTUR; semantic judging reduced surface-form brittleness without changing the leaderboard story.Exact rule checks can mark faithful answers as failures when they use the correct tool value without copying the expected string exactly.Source: §5.4 · p. 6
  • Three-way ties occurred in 1.5% of judgments, while the two LLM judges jointly overruled the rule label in 8.4%.Agreement was stable across strata; lower κ values for rare labels mainly reflected low prevalence rather than broad rater disagreement.Source: §5.4 · p. 6

5.5. Domain Effects

  • 12.24% median rule RIR in Finance versus Cybersecurity’s 0.68% shows that result ignoring varies by domain.The comparison is descriptive, not evidence that one domain is intrinsically harder: RIR can also depend on trap-value salience or recoverability.Source: §5.5 · pp. 6–7, Figure 3 · p. 6

6. Conclusion

ToolFailBench separates tool-use failure modes and shows that similar aggregate scores can conceal distinct model behaviors. Its single-turn, five-domain design and labeling pipeline limit the scope of its conclusions, while planned expansions target broader and longer tool-use settings.

6. Conclusion

  • 86.33% Clean Tool-Use Rate is the best result across 19 headline models, while similar aggregate scores conceal distinct failure profiles.Most models remain disciplined on no-tool controls, but Llama-3.1 models show an Always-Call pattern; Llama-3.1-70B and Qwen2.5-72B differ by 89 percentage points in control-task accuracy.Source: §6 · p. 7
  • ToolFailBench is a diagnostic benchmark, not a full simulation of deployed agents; its single-turn format omits tool chaining, recovery, and cross-interaction state updates.Its scope is also limited to five professional domains, its rule classifier and two LLM judges may share blind spots, and future train-test overlap is possible.Source: §6 · p. 7
  • The planned evaluation harness will give model developers failure-mode breakdowns, while planned dataset expansions add domains, query types, and multi-turn variants.These variants are intended to test whether the same failure modes persist in longer, more realistic tool-use settings.Source: §6 · p. 7

A. Benchmark Details

The benchmark uses domain-specific tools and tasks designed to test whether models follow controlled tool returns rather than substitute plausible prior values or invent unsupported details.

A.1. Domain Tools

  • Each domain uses a role-specific system prompt and nine tools, including domain-relevant finance, medicine, law, cybersecurity, and real-estate services.Examples include quote and filing tools for finance, drug and lab-reference tools for medicine, case-law and statute tools for law, CVE and threat-intelligence tools for cybersecurity, and listing and mortgage tools for real estate.Source: §A.1 · p. 9

A.2. Trap Values and Structured Fields

  • Tool-required tasks use mock returns that differ from plausible prior values, testing whether the final answer follows the return rather than an unsupported alternative.The benchmark measures observable behavior without claiming to identify whether the competing value comes from memorization, hallucination, or another internal mechanism; structured fields also let the rule classifier flag invented details as Output-Fabrication.Source: §A.2 · p. 9

B.1. Model Identifiers

The study reports 19 models with valid traces in its headline leaderboard and standardizes task and decoding settings across runs. Hardware differences affect serving throughput, while metrics use the same task set, trace format, and labeling pipeline.

B.1. Model Identifiers

  • 19 of the 22 listed model runs completed the full suite with valid traces and appear in the headline leaderboard; two are non-headline tool-skip cases and one has invalid raw-token output.The remaining runs are documented in Appendix D.Source: §B.1 · p. 9, Table 5 · p. 9

B.2. Inference Settings

  • All models use the same 1,000-task suite, task files, tool schemas, system prompts, and decoding settings, including temperature=0 and max tokens=1024.Seed=42 is used where supported; hardware affects throughput only, while metrics use the same tasks, trace format, and labeling pipeline.Source: §B.2 · p. 10

B.3. Temperature Ablation

The temperature ablation checks whether the headline conclusions depend on greedy decoding; Rule CTUR changes little, while degenerate repetition traces decrease under sampling.

B.3. Temperature Ablation

  • 75.47% Rule CTUR at temperature=0 shifts to 75.60% under nonzero temperature, while degenerate repetition traces fall from 34/1000 to 22/1000.The authors conclude that headline results are not driven by greedy decoding, though sampling can affect rare formatting or repetition failures.Source: §B.3 · p. 10

B.4. Leaderboard Confidence Intervals

Table 6 reports 95% Wilson score confidence intervals for ensemble CTUR. These intervals use valid ensemble-labeled tool-required traces, so their denominators may differ from 750.

B.4. Leaderboard Confidence Intervals

  • The intervals are computed over valid ensemble-labeled tool-required traces for each model.Source: §B.4 · p. 10
  • Denominators can differ slightly from 750 when a judge returns a null label.Source: §B.4 · p. 10
  • Table 6 reports 95% Wilson score confidence intervals for ensemble CTUR.The table's scope is ensemble CTUR.Source: Table 6 · p. 10

C. Judge Rubrics and Labeling Details

The appendix specifies trace-based rule labels and a three-rater majority-vote ensemble, then checks judge-family preference, prompt framing, and label agreement.

C.1. Rule Classifier

  • Tool-Skip and Unnecessary-Tool-Use depend on executed-call presence, while Result-Ignore and Output-Fabrication check the answer against returned values.Result-Ignore marks omission of the expected returned value; Output-Fabrication marks unsupported structured information.Source: §C.1 · p. 10

C.2. Judge Ensemble

  • The ensemble takes an unweighted majority vote across the rule classifier and two LLM judges, resolving ties with the rule classifier.Agreement is reported using pairwise Cohen’s κ and three-way Fleiss’ κ.Source: §C.2 · p. 11

C.3. Same-Family Judge Check

  • The same-family check tests whether Qwen-judge CTUR exceeds GLM-judge CTUR for Qwen-family models.Table 7 defines its gap as Qwen-judge CTUR minus GLM-judge CTUR, in percentage points.Source: §C.3 · p. 11, Table 7 · p. 11
  • The Qwen judge scores both model groups lower than the GLM judge, with nearly identical mean gaps and no evidence of same-family inflation.Source: §C.3 · p. 11

C.4. Judge Prompt Framing

  • The Qwen judge uses a decision-tree prompt, while the GLM judge requests evidence attribution before assigning a label.The comparison tests label stability across different prompt framings.Source: §C.4 · p. 11

C.5. Stratified Agreement

  • Result-Ignore and Unnecessary-Tool-Use have lower κ but high raw agreement, consistent with the low-prevalence κ effect.Source: §C.5 · p. 11
  • Table 8 stratifies inter-rater agreement by domain, failure mode, and model tier.Source: Table 8 · p. 11

D. Non-headline Runs

  • The study ran 22 models, while three runs were reported separately rather than included in the 19-model headline leaderboard.Those runs are described as useful for understanding tool-use and harness failure cases, but not as clean CTUR comparisons.Source: D. Non-headline Runs · p. 11

D.1. Tool-Style Answers Without Executed Tool Calls

This section documents how tool use is guided and classified, distinguishing executed calls from tool-like text and malformed-output exclusions. It specifies judge rubrics and procedures for classifying errors on tool-required and control tasks, including decision-tree and evidence-attribution rules.

D.1. Tool-Style Answers Without Executed Tool Calls

  • glm-4-9b and mistral-7b produced zero clean executed calls on tool-required tasks, and their plausible “Per <tool>” answers were labeled Tool-Skip.glm-4-9b is the clearer trace-level case; mistral-7b is treated qualitatively because parser behavior may have contributed to the missing calls.Source: §D.1 · p. 12

D.2. Invalid Raw-Token Output

  • The deepseek-r1-distill-llama-8b run was excluded from headline metrics because invalid raw-token output suggested a harness or formatting issue.It was excluded from the main comparison rather than assigned a leaderboard score.Source: §D.2 · p. 12

E.1. Prompt Setup

  • Each domain uses one shared system prompt across models, with no model-specific tuning or in-context examples containing task answers.Prompts specify when to call tools, when to answer directly, and how to format final responses; full prompt files are released with the harness.Source: §E.1 · p. 12

E.2. Domain Prompt Summary

  • Domain prompts share a high-level structure but use domain-specific roles, tool descriptions, and output fields, with nine tools per domain.Table 9 summarizes the role and tool categories for each domain prompt.Source: §E.2 · p. 12, Table 9 · p. 12

E.3. Output Template

  • Tool-required answers use a uniform template for returned values and bounded context, while control queries use a direct-answer template.The format supports cross-model comparison and rule-classifier checks that the final answer uses the tool-returned value.Source: §E.3 · p. 12

E.4.1. FINANCE

  • The finance prompt treats tool returns as ground truth for specific-value questions and directs models to answer conceptual questions from training without calling tools.It also instructs models to quote tool values exactly and not replace or blend them with remembered values.Source: §E.4.1 · p. 14

E.5. Judge Prompts

  • The two LLM judges share the same base rubric but use different overlays: a decision tree for Qwen and evidence attribution for GLM.The overlays are intended to reduce agreement arising solely from shared prompt framing.Source: §E.5 · p. 15, Table 10 · p. 15

E.5.1. SHARED BASE RUBRIC

  • The shared rubric requires tool calls and use of returned values for tool-required tasks, but direct answers without tools for controls.It defines Tool-Skip as no valid executed call, Result-Ignore as failure to reflect returned data, and Output-Fabrication as invented structured data; parametric traps make tool values differ from likely memorized values.Source: §E.5.1 · pp. 15–16
  • The shared rubric scores tool selection and result faithfulness on tool-required tasks, answer correctness on all tasks, and restraint on controls, while tolerating reasonable formatting differences.A result-ignore error includes different or missing values, or a generic answer despite specific tool data; control answers are checked against required ground-truth values.Source: §E.5.1 · p. 16

E.5.2. QWEN JUDGE OVERLAY: DECISION TREE

  • The Qwen overlay uses actual tool-call records to classify controls as unnecessary-tool-use or correct/wrong-answer, and tool-required tasks as tool-skip, output-fabrication, result-ignore, or correct.Tool-like strings in the answer body do not count as executed calls; tool-required answers are checked for fabricated fields and fidelity to returned values, allowing formatting differences.Source: §E.5.2 · p. 17

E.5.3. GLM JUDGE OVERLAY: EVIDENCE ATTRIBUTION

  • The GLM overlay grounds classifications in tool-call, faithful-use, fabrication, memorization, and control ground-truth evidence, mapping those signals to the corresponding failure modes.A missing expected call indicates tool-skip; fabricated values or contradictions indicate output-fabrication or result-ignore, while matching tool values support a correct classification.Source: §E.5.3 · p. 18
Loading 2607.04686v1…