Reducing Tool Hallucination via Reliability Alignment

1. Introduction

• Tool hallucinations can impair task performance and reliability, produce misleading final outputs, and increase computational costs through faulty or redundant calls.The introduction notes particular risks in database operations, robotic control, and scientific experimentation.Source: §1 · p. 1
• Tool hallucinations include selection errors—choosing an unrelated tool or calling it at the wrong time—and usage errors, such as incorrect or fabricated parameters.The proposed detection approach combines rule-based methods with LLM evaluators.Source: §1 · pp. 1–2
• Reliable Pass Rate discounts task outcomes influenced by hallucinated calls, while Benefit-cost Utility rewards task completion and penalizes hallucinations and excessive tool costs.These measures address how tool hallucinations can cascade into final task outcomes.Source: §1 · p. 2
• RelyToolBench adds subsets for tool-task mismatches and parameter deficiencies to assess selection and usage hallucinations.The benchmark is derived from StableToolBench (Guo et al., 2024).Source: §1 · p. 2
• Relign expands tool-use actions to include deferring invocation, asking users for clarification, or switching tools when preconditions are unmet.The introduction says decisive actions can otherwise lead to invalid calls when parameters are insufficient or a tool is unsuitable.Source: §1 · p. 2
• Relign combines supervised fine-tuning, direct preference optimization, and reliability-focused data synthesis to reduce tool hallucinations and improve model reliability.Experiments on RelyToolBench are reported to demonstrate these improvements.Source: §1 · p. 2

2.1. Tool Hallucination

• Tool hallucinations comprise selection errors, including wrong or mistimed calls, and usage errors, including malformed calls or fabricated parameter content.Selection subtypes include tool-type and timing hallucinations; usage subtypes include format and content hallucinations.Source: §2.1 · pp. 2–3
• The evaluation combines rule-based checks for structural errors with an LLM evaluator for tool relevance and parameter authenticity; human assessment reports 92.7% accuracy.The framework also assesses whether calls are redundant or premature and whether parameter values are grounded in the user query.Source: §2.1 · p. 3

2.2.1. RELIABLE PASS RATE

• The Reliable Pass Rate (RePR) subtracts task hallucination rate from pass rate, discounting successful outcomes affected by tool hallucinations.Tasks with an original pass rate of zero are excluded from result-hallucination assessment; an LLM checks whether identified hallucinated calls correlate with the final result.Source: §2.2.1 · pp. 3–4
• Across datasets, RePR is significantly lower than pass rate because some result hallucinations are not directly identified by evaluation models.Figure 3 compares reliable and original pass rates across the original, missing-parameter, and unmatched-tools subsets.Source: §2.2.1 · p. 4, Figure 3 · p. 4

2.2.2. BENEFIT-COST UTILITY

• Benefit-Cost Utility combines task outcome with penalties for hallucinations and excessive tool calls, capturing both task quality and tool-use efficiency.The task reward is 20 for success and 0 for failure; hallucinations incur a -10 penalty, and excess calls are penalized up to 10.Source: §2.2.2 · p. 4

2.3. Construction of RelyToolBench

• RelyToolBench extends StableToolBench with missing-parameter and unmatched-tool subsets to test reliability under incomplete information and tool-selection mismatches.The benchmark includes data statistics in Table 1.Source: §2.3 · p. 4

3. Reliability Alignment

• Relign addresses calls made under unmet conditions, such as unknown parameters or task-mismatched tools, by adding ChangeTools and TalkToUser actions and training with synthesized alignment data.The paper identifies inadequate modeling of decision conditions as a primary cause of tool hallucinations.Source: §3 · p. 4

3.1. Alignment Goal

• At task level, Relign ranks successful trajectories above failure or abandonment, and failure or abandonment above hallucinated outcomes.The authors report that models with high tool-hallucination rates tend to favor hallucinated outputs over acknowledging failure, risking user misbelief that a task succeeded.Source: §3.1 · p. 5
• At step level, correct calls outrank indecisive actions, which outrank hallucinated calls; reducing excessive calls can also improve efficiency.The paper notes that required call counts are typically fixed for many tasks and that excessive calls tend to introduce more hallucinations.Source: §3.1 · p. 5

3.2. Alignment Methods

• Relign first uses SFT to teach indecisive actions, then synthesizes model-based preference data for DPO training.The two-stage process is intended to teach handling of indecision and align behavior with task-level preferences.Source: §3.2 · p. 5

3.2.1. RELIABLE SFT

• Reliable SFT trains ChangeTools for tool-type hallucinations and TalkToUser for tool-content hallucinations using modified ToolBench samples, with training data limited to single-tool tasks.The samples simulate hallucinations by replacing tool sets or removing tool parameters.Source: §3.2.1 · p. 5

3.2.2. RELIABLE DPO

• Reliable DPO samples multiple responses per trajectory step, classifies them, and trains preferences ordering correct calls above indecision above hallucinated calls.It also includes preferences for correct calls over hallucinated calls; the paper says the resulting ranking supports choosing reliable sequences or abandoning a task or seeking clarification.Source: §3.2.2 · p. 6

4.1. Experimental Setup

• Experiments use ToolBench, StableToolBench, and RelyToolBench, compare RLHF, StepTool, SFT, and DPO baselines, and train reliability data from 10,000 ToolBench samples.The setup describes three model families and curated SFT traces plus DPO preference pairs generated through multi-round environment interactions.Source: §4.1 · p. 6

4.2. Main Results

• Relign improves task success while reducing tool hallucinations and tool calls across Original, Missing Parameter, and Unmatched Tools subsets.The reported comparison also finds that LLaMA 3.1 Relign outperforms several similarly sized closed-source models and approaches GPT-4o.Source: §4.2 · pp. 6–7, Figure 5 · p. 7, Table 2 · p. 7

4.3. Tool Hallucination Analysis

• Relign reduces all hallucination types, with the largest improvement in Content Hallucination on Unmatched Tools and a smaller improvement in Timing Hallucination.The authors attribute the smaller Timing Hallucination improvement as likely related to the difficulty of learning sequential dependencies.Source: §4.3 · p. 7, Figure 6 · p. 8

4.4. Comparison with Existing Metrics

• RePR is lower than Pass Rate, with a larger gap for unaligned base models; Relign improves task success under both metrics.The authors note that some tasks counted as passing exhibit result hallucinations, motivating their refined metric.Source: §4.4 · p. 8, Table 3 · p. 8

4.6. Discussion

• Increasing training data from 1k to 5k raises Reliable Pass Rate, but tool and task hallucination rates remain stable or slightly increase.The authors suggest ToolBench’s lack of edge-case failure examples may contribute to excessive hallucinated calls instead of indecisive actions.Source: Figure 7 · p. 8, §4.6 · p. 8
• With training data fixed at 5k, scaling model size from 1.5B to 14B improves performance and hallucination reduction.The reported changes include declining tool hallucinations and excessive tool calls, alongside increasing Reliable Pass Rate.Source: Figure 8 · p. 8, §4.6 · p. 9

4.5. Out-of-distribution Generalization to APIBench

• On APIBench, Relign improves performance under two tool-retrieval setups, supporting generalization to other tool-use tasks.The paper also suggests that reducing tool hallucinations helps models learn better tool-use strategies.Source: Table 4 · p. 8, §4.5 · p. 8

5.1. Tool Learning

• Tool-learning research spans in-context demonstrations and training on tool-enriched datasets, while evaluation often emphasizes tool-call accuracy and task success over hallucinations and efficiency.The passage names prior work on tool learning, evaluation, hallucinations, and efficiency as examples of these research strands.Source: §5.1 · p. 9

5.2. Mitigating Halucinations

• Prior hallucination-mitigation work covers several strategies, while the paper distinguishes its approach by systematically defining tool-hallucination types and proposing reliability alignment.The related-work discussion also describes using LLMs as evaluators for tool hallucinations.Source: §5.2 · p. 9

6. Conclusion

• Relign expands tool-use decisions to include switching tools or seeking user clarification, allowing models to make more informed decisions and avoid hallucinations.The conclusion frames reliability-aware training as important for tool-augmented LLMs and as a basis for future research on real-world tool-use decisions.Source: §6 · p. 9

A. Human Evaluation for Hallucination Detection

• Human annotators confirmed 46/50 no-hallucination labels, 45/50 parameter-value hallucination labels, and 48/50 tool-relevance hallucination labels, for 92.7% overall accuracy.The authors interpret these results as evidence of evaluator reliability and alignment with human judgment.Source: A. Human Evaluation for Hallucination… · p. 12

B. Prompt for Checking Tool Relevance

• The tool-relevance checking prompt is shown in Table 5.Source: B. Prompt for Checking Tool Relevance · p. 12
• Table 5 is labeled as the prompt for checking tool-relevance hallucination.Source: Table 5 · p. 12

C. Prompt for Checking Content Hallucination

• The tool-content hallucination checking prompt is shown in Table 6.Source: C. Prompt for Checking Content… · p. 12

D. Prompt for Checking Hallucinated Answer

• The prompt for checking hallucinated answers is shown in Table 7.Source: D. Prompt for Checking Hallucinated… · p. 12

• The missing-parameter dataset generation prompt is shown in Table 8.Source: E. Prompt for Data Generation · p. 12
• Table 6 contains the prompt used to check content hallucination.Source: Table 6 · p. 13
• Table 7 contains the prompt for checking relevance between a provided answer and hallucinated tool calls.Source: Table 7 · p. 13

• The StableToolBench task evaluation prompt is shown in Table 9.Source: F. Prompt for evaluating StableToolBench · p. 13
• Table 8 contains the prompt for generating the missing-parameter dataset.Source: Table 8 · p. 14
• Table 9 contains the prompt used to evaluate StableToolBench tasks and call tools.Source: Table 9 · p. 14

• Relign successfully corrects the parameter hallucination in the missing-parameter subset, compared with the Llama3.1 baseline.The case study is shown in Figure 9.Source: Figure 9 · p. 15, G. Case Study · p. 15