PaperlayerView exact revision on arXiv

Source-linked AI summary

Understanding and Mitigating Hallucination Escape in Tool-Using LLM Agents

Peigui Qi, Kunsheng Tang, Yide Song, Weiming Zhang, Nenghai Yu

arXiv:2610.04409v1cs.CLcs.AI

TL;DR

Tool-mitigation gains are often measured only on tuned configurations, leaving performance under other configurations and queries unassessed. EscapeGuard detects conflicts between intrinsic tool tendencies and the runtime configuration, then redirects attention to current tool definitions. Across benchmarks, it reduces tool-selection hallucination and suppresses escape.

  • Problem

    Existing tool-mitigation evaluations focus on the configuration used for tuning, leaving performance across other configurations and queries unmeasured.

    Source: §1 · p. 2

  • Method

    EscapeGuard detects conflicts between intrinsic tool-use tendencies and the runtime configuration, then strengthens attention to relevant tool definitions for conflicted inputs.

    Source: §1 · p. 2, §5.1 · p. 7

  • Results

    Across six benchmarks, EscapeGuard reduces tool-selection hallucination by 9.0 pp, lowers the cross-configuration mean by 23.7 pp, and achieves 89.1% net improvement in paired-query evaluation.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The findings support evaluating tool-agent mitigations beyond a single tool configuration to assess reliability across changing configurations.

    Source: §7 · p. 9

    The six studied tool configurations and architectures are representative but not exhaustive; tendency conflict accounts weakly for argument-value errors, with correspondingly modest AVHR improvements.

    Source: §6 · p. 9

Abstract

from arXiv · show

Large language models (LLMs) increasingly serve as autonomous agents that invoke external tools. However, this capability introduces tool hallucination, selecting incorrect tools or generating invalid calls. Existing mitigation methods report substantial improvements, yet we identify a previously overlooked failure mode that we term Hallucination Escape. These methods reduce hallucination on the tool configuration they are tuned on but increase it on other configurations, canceling out the gain. We further investigate this phenomenon and find that hallucination rises sharply when a model's intrinsic tool-use tendencies conflict with the current tool configuration, and that existing methods reinforce rather than suppress these tendencies, which in turn contributes to hallucination escape. Building on these findings, we propose EscapeGuard, a training-free inference-time method that combines conflict-aware gating with configuration-derived attention enhancement to mitigate tool hallucination while preventing hallucination escape. Across six benchmarks on various models, EscapeGuard reduces tool-selection hallucination by 9.0 pp and suppresses hallucination escape, lowering the cross-configuration mean by 23.7 pp and achieving an 89.1% net improvement in paired-query evaluation. We hope this work can encourage evaluation beyond a single tool configuration and pave the way for more reliable tool-using LLM agents.

1 INTRODUCTION

Tool hallucinations can trigger consequential invalid calls, and this paper identifies a failure in existing mitigations: gains on tuned configurations may be lost on others. It traces this escape to reinforced intrinsic tool-use tendencies and introduces EscapeGuard, which uses runtime configuration information to reduce hallucination without escape.

Motivation

  • Tool hallucination includes selecting nonexistent or irrelevant tools and generating invalid arguments, which can trigger irreversible operations on real systems.Existing mitigation approaches include training-based behavior changes and steering-based interventions on internal representations.Source: §1 · p. 1

Phenomenon

  • Hallucination escape occurs when mitigation reduces hallucination on its tuned configuration but increases it on others enough to cancel the gain.Existing methods average a 7.5 pp reduction on tuned configurations but a 13.0 pp increase across the other five.Source: §1 · p. 2

Investigation

  • Models favor intrinsic tool and argument patterns, and hallucination rises sharply when those tendencies conflict with the runtime configuration.Existing methods reinforce these tendencies, with stronger reinforcement associated with larger escape ratios.Source: §1 · p. 2

Solution

  • 9.0 pp lower tool-selection hallucination, a 23.7 pp lower cross-configuration mean, and 89.1% net improvement in paired-query evaluation are reported for EscapeGuard.The results span six benchmarks and various models.Source: §1 · p. 2
  • EscapeGuard detects high-conflict inputs and strengthens attention to runtime tool definitions only for those inputs, adapting its corrective signal as tools or queries change.It is a training-free inference-time method.Source: §1 · p. 2

2 BACKGROUND AND RELATED WORK

Prior work addresses tool hallucination through training-based behavior changes and steering-based interventions. The reviewed methods report gains but evaluate on the tool configuration used for tuning.

LLM-based Agents and Tool Use

  • Toolformer, ReAct, and ToolLLM exemplify the expansion of LLM agents from learned tool calls to reasoning-and-action and real-world API use.As tool use expands, erroneous invocations remain a risk.Source: §2 · p. 2

Tool Hallucination and Mitigation

  • Training-based methods reshape behavior, while steering-based methods intervene on internal activations; both are evaluated only on tuned-on configurations.Examples include Relign, PALADIN, and Gorilla for training, and LinSteer and PRISMS for steering.Source: §2 · p. 2
  • Figure 2 illustrates tool-configuration variants using a weather API, with each variant retaining a valid solution.Source: Figure 2 · p. 3

3 PHENOMENON: HALLUCINATION ESCAPE

Existing mitigations can improve results in their tuned setting yet worsen hallucination under other configurations or paired queries. The section evaluates this escape across configuration changes, measures its effects, and formalizes the failure mode.

3 PHENOMENON: HALLUCINATION ESCAPE

  • Prior evaluations varied tool availability but kept tool definitions fixed, whereas this study varies names and argument specifications while preserving a valid solution.Source: §3 · p. 3, §3.1 · p. 3

3.1 EVALUATION SETUP

  • The evaluation varies six tool configurations and paired queries, testing five C1-trained methods on two models with BFCL and Seal-Tools data.It includes about 1,000 examples per configuration and 120 related BFCL query pairs.Source: §3.1 · p. 3
  • Cross-configuration evaluation reports TSHR and, when argument schemas change, OHR; cross-query evaluation reports repair, transfer damage, and net improvement.Source: §3.1 · p. 3

3.2 EVALUATION RESULTS

  • All five methods reduce tuned C1 TSHR but leave the cross-configuration mean above base, with Var 62.8% higher on average.They raise C2 TSHR by 2–9 pp, C3 and C6 TSHR by 1–6 pp, and C5 OHR by 2–7 pp.Source: Table 1 · p. 4, §3.2 · p. 4
  • Figure 3 reports cross-query hallucination escape as a percentage, but its caption does not establish a numerical rate, repair-error rate, or effect on net improvement.Source: Figure 3 · p. 4
  • Methods change TUHR by no more than 1.5 pp in any configuration, leaving tool-usage hallucination largely untouched.Base TUHR reaches 24%–28% on C4 and C5; shared-subset analysis confirms changes of at most 0.8 pp.Source: §3.2 · p. 4

3.3 DEFINITION OF HALLUCINATION ESCAPE

  • Hallucination escape requires improvement in some configurations, deterioration in others, and an overall mean at least as high as the base model.All five evaluated methods meet this definition across configurations and queries.Source: §3.3 · p. 4

4 INVESTIGATION: THE MECHANISM OF HALLUCINATION ESCAPE

This section examines whether intrinsic tool-use tendencies persist without runtime configuration, how conflicts relate to hallucination, and whether existing methods reinforce those tendencies. Its evidence links stronger conflict to higher hallucination and preference reinforcement to larger escape ratios.

4.1 INTRINSIC TENDENCIES AND HALLUCINATION

  • Selection priors averaged 0.61–0.78 and usage priors 0.57–0.71; hallucination rose with tendency strength under conflict but stayed uniformly low under alignment.The conflict association was strongest for tool selection and argument names, and modest for argument values.Source: §4.1 · p. 5, Figure 4 · p. 5

4.2 INTERNAL EVIDENCE OF TENDENCY CONFLICT

  • Layerwise probes reached 85%–87% tool-selection and 77%–79% argument-name conflict accuracy, while stronger conflict scores tracked higher hallucination.For LLaMA-3.1-8B, tool-selection hallucination rose from 19.6% to 81.2% across the extreme conflict-score quintiles.Source: §4.2 · p. 6, Table 2 · p. 6

4 INVESTIGATION: THE MECHANISM OF HALLUCINATION ESCAPE

  • Existing methods appear to reinforce intrinsic tendencies, motivating a conflict-aware approach to mitigate hallucination without increasing it on other configurations.The section’s investigation tests whether those preferences persist without runtime configuration, conflict with it, and are reinforced by mitigation.Source: §4 · p. 5, §4.3 · p. 6

4.3 ANALYZING HALLUCINATION ESCAPE IN EXISTING METHODS

  • 76%–93% of queries retained their preferred tool after mitigation, while prior strength rose 0.04–0.14 across methods and stronger reinforcement was associated with escape ratios up to 2.42.The same pattern held across queries: reinforced preferences overgeneralized to similar queries requiring other tools.Source: §4.3 · p. 6, Table 3 · p. 6

5 SOLUTION: ESCAPEGUARD

ESCAPEGUARD detects conflicts between a model’s tool-use tendencies and the runtime configuration, then conditionally redirects attention to relevant tool definitions. It reduces tool hallucination across benchmarks and configurations, while its evaluation tests baseline comparisons, robustness, and component contributions.

5.1 DESIGN OF ESCAPEGUARD

  • ESCAPEGUARD prepares tool-name, description, and schema spans, gates intervention on conflict, then enhances attention to selection definitions or the selected tool’s schema.Low-conflict inputs retain the original computation; the intervention derives from the current configuration and adapts to tool renaming, replacement, and schema changes.Source: §5.1 · p. 7

5.2 EXPERIMENTAL SETUP

  • Evaluation spans six tool-use benchmarks and six model backbones, with all five baselines compared on LLaMA-3.1-8B and Qwen-3.5-9B.The reported evaluation perspectives are overall effectiveness, baseline comparison, cross-configuration robustness, and ablation.Source: §5.2 · p. 8, §5.3 · p. 8

5.3 EXPERIMENTAL RESULTS

  • Across six models, ESCAPEGUARD reduces TSHR by 4.3–14.7 pp for every model-benchmark pair and consistently lowers TUHR and ANHR.The overall-effectiveness results also report a modest 4%–6% inference-latency overhead.Source: Table 4 · p. 8, Table 5 · p. 8, §5.3 · p. 8
  • On four LLaMA-3.1-8B benchmarks, ESCAPEGUARD reduces TSHR by 9–12 pp on average, versus 3–6 pp for the five baselines.It achieves the lowest TSHR, TUHR, and ANHR across those benchmarks.Source: §5.3 · p. 8, Table 6 · p. 9
  • Cross-configuration mean hallucination falls by 24.2 pp for LLaMA and 23.3 pp for Qwen, with 87.8%–90.4% net improvement in paired-query evaluation.Hallucination decreases across all six configurations; under C1–C3, rates converge to 5.6%–8.6%.Source: Table 7 · p. 9, Table 8 · p. 9, §5.3 · p. 9
  • Ablations find contributions from conflict gating, stage-specific enhancement, correct conflict layers, and input-dependent enhancement over fixed steering vectors.The ablations use LLaMA-3.1-8B and Qwen-3.5-9B.Source: §5.3 · p. 9

6 DISCUSSION

The discussion identifies limits in the evaluated scope and in the tendency-conflict mechanism’s coverage of argument-value errors. Distractor injection is also a setting where gains are smallest.

6 DISCUSSION

  • The six studied configurations and architectures are representative, not exhaustive, so the scope does not establish behavior across broader settings.The authors identify extension to broader architectures as future work.Source: §6 · p. 9
  • Tendency conflict accounts weakly for argument-value errors, and ESCAPEGUARD’s AVHR improvements are correspondingly modest.Source: §6 · p. 9
  • Gains are smallest under distractor injection (C6), likely because confusable tools dilute the enhanced-attention signal.Improving distractor robustness is identified as future work.Source: §6 · p. 9

7 CONCLUSION

The paper concludes that ESCAPEGUARD reduces tool hallucination while preventing hallucination escape across models. Its appendices document how configuration variants, query pairs, and evaluation sets were constructed and checked.

7 CONCLUSION

  • ESCAPEGUARD reduces tool-selection hallucination by 9.0 pp and suppresses hallucination escape across models and benchmarks.The method is training-free and redirects attention to the runtime configuration after detecting tendency conflicts.Source: §7 · p. 9

A.1 CONFIGURATION CONSTRUCTION RULES

  • Configurations vary tool names, descriptions, and argument names, with C5 combining changes and C6 adding distractors; mitigation methods are calibrated on C1.C2–C6 are out-of-distribution test scenarios relative to that standard naming style.Source: §A.1 · p. 12

A.2 CONFIGURATION CONSTRUCTION EXAMPLES

  • Five examples illustrate C1–C6 configuration changes for queries across finance, communication, translation, meteorology, and travel.Source: §A.2 · p. 12

A.3 CONFIGURATION QUALITY CONTROL

  • Two authors independently reviewed and reconciled generated variants, then automated checks verified preserved solutions and configuration-specific construction rules.All queries passed the automated checks.Source: §A.3 · p. 18

B QUERY PAIR CONSTRUCTION RULES

  • The paired-query set contains 30 pairs per domain across four domains; each pair shares a tool list but requires a different correct tool.Pairs are semantically related, and their functional descriptions distinguish the correct tools.Source: B QUERY PAIR CONSTRUCTION RULES · p. 18

C.1 CONTROLLED AND PAIRED-QUERY EVALUATION

  • Controlled evaluation uses about 1,000 source queries across C1–C6, while paired evaluation uses 120 pairs totaling 240 queries.The paired-query set is drawn from the BFCL V3 single-turn multi-tool subset.Source: §C.1 · pp. 18–19

C.2 BEHAVIORAL ANALYSIS SET

  • Behavioral analysis samples 300 queries equally from BFCL, Seal-Tools, and Glaive Function Calling V2, retaining cases with identifiable tools and structured schemas.Each query is evaluated in aligned, tool-selection-conflict, and argument-level-conflict conditions.Source: §C.2 · p. 19

C.3 PROBING CORPUS AND DATA SPLITS

The probing corpus combines queries from BFCL, Seal-Tools, and Glaive Function Calling V2, then partitions source queries for probe fitting and testing.

C.3 PROBING CORPUS AND DATA SPLITS

  • The 1,200-query corpus retains 300 behavioral queries and adds 900 queries: 300 from BFCL, 200 from Seal-Tools, and 400 from Glaive Function Calling V2.The corpus composition counts refer to source queries before condition variants are constructed.Source: §C.3 · p. 19, Table C.1 · p. 19
  • Of 1,200 source queries, 240 are held out for testing; the remaining 960 split into 720 training and 240 validation queries.Source: §C.3 · p. 19
  • Probe fitting repeats over five stratified random splits, keeping the test set fixed and stratifying by dataset source and conflict type.Source: §C.3 · p. 19

D.1 BASIC METRICS

This appendix defines the hallucination metrics and cross-configuration summaries, then checks the results under a uniform selection metric and separates tool-selection from tool-usage changes.

D.1 BASIC METRICS

  • Selection metrics count unavailable or functionally incorrect tools as hallucinations, distinguishing fabrication, misselection, and correct selection.The usage rates are conditional on correct selection; ANHR and AVHR need not sum to TUHR because one call can contain both error types.Source: §D.1 · p. 20
  • Overall hallucination combines incorrect selection with hallucinated argument usage after correct selection; percentage OHR is computed from TSHR and TUHR.Usage rates are undefined when their conditional denominator is zero.Source: §D.1 · p. 20

D.2 CROSS-SCENARIO AGGREGATION

  • The six-configuration mean summarizes designated per-configuration metrics, while variance captures whether hallucination is distributed unevenly across configurations.The designated metric is TSHR for C1–C3 and C6, and OHR for C4–C5; this mean is not pooled OHR.Source: §D.2 · p. 20

D.3 QUERY TRANSFER METRICS

  • Query-transfer metrics use target and related-query pairs sharing a tool list, with Net Improvement discounting repairs that introduce collateral errors.The shared configuration argument is omitted from the formulas, which use proportions and can be converted to percentages by multiplying by 100.Source: §D.3 · pp. 20–21

D.4 TSHR-ONLY CROSS-SCENARIO AGGREGATION

  • 25.7/24.2 pp reductions in mean TSHR and an approximately 60% variance reduction for EscapeGuard confirm escape mitigation under a uniform metric.Existing methods instead raise mean TSHR above the base model by 0.4–2.4 pp for LLaMA and 0.4–2.0 pp for Qwen, while increasing variance by 62.7%–165.5%.Source: §D.4 · p. 21, Table D.1 · p. 21

E.1 TOOL-USAGE HALLUCINATION UNDER CONFIGURATION CHANGES

  • In both models, mitigation methods change TUHR by no more than 1.5 pp in any configuration, leaving the usage component largely stable.The base TUHR rises under C4 and C5 because of argument renaming; the observed marginal C4 OHR reductions therefore stem primarily from small TSHR improvements.Source: §E.1 · p. 21, Table E.1 · p. 22

E.2 SHARED-CORRECT SUBSET ANALYSIS

A fixed shared-correct subset checks that limited tool-usage sensitivity is not an artifact of changing denominators; separate elicitation measurements show models favor recurring tools and argument structures.

E.2 SHARED-CORRECT SUBSET ANALYSIS

  • On the shared-correct subset, every TUHR change remains within ±0.8 pp, confirming limited sensitivity is not driven by denominator shifts.The analysis recomputes TUHR over queries correctly selected by both the base model and the mitigation method.Source: §E.2 · p. 22, Table E.3 · p. 22, Table E.4 · p. 22

F.1 INTRINSIC PRIOR STRENGTH

  • Mean selection priors range from 0.61 to 0.78 and mean usage priors from 0.57 to 0.71 across six models, indicating recurring preferences without runtime configuration.Selection prior strength is based on the frequency of the most commonly generated tool; usage prior strength compares generated argument-name sets with the most frequent set.Source: §F.1 · pp. 22–23, Table F.1 · p. 22

F.2 TENDENCIES AND HALLUCINATION

The appendix relates intrinsic tool-use tendencies to hallucination under conflict, describes how probes are selected, and tests whether their conflict scores predict errors within conflicting inputs.

F.2 TENDENCIES AND HALLUCINATION

  • Under selection conflict, hallucination rises monotonically with prior strength, while aligned-condition rates remain uniformly low across prior-strength groups.The reported per-seed quintile granularity is roughly 1.7 pp; estimates average three random seeds.Source: §F.2 · p. 23
  • Argument-name hallucination has a stronger association with intrinsic usage prior than argument-value hallucination; under conflict, ANHR rises monotonically while AVHR shows a weaker positive trend.Under aligned conditions, both error rates remain flat across prior-strength groups.Source: §F.2 · pp. 23–24, Table F.7 · p. 24, Table F.8 · p. 24

G.1 PROBE TRAINING AND LAYER SELECTION

  • Probes use last-input-token hidden states with L2-regularized logistic regression; the representative layer is the median selected across five validation-based seed runs.When layers are within 0.2 percentage points of the best validation accuracy, the earliest is selected.Source: §G.1 · p. 24

G.2 EXTENDED LAYERWISE PROBING RESULTS

  • Selection conflict is most distinguishable in middle-to-late layers, while argument-name conflict generally peaks later; selection probes also achieve higher accuracy and hallucination-prediction AUROC.Source: §G.2 · p. 24

G.3 PREDICTING HALLUCINATION WITHIN CONFLICTING INPUTS

  • Within conflicting inputs, stronger probe conflict signals correspond to higher hallucination likelihood across every evaluated backbone, with a stronger effect for tool selection than argument naming.The analysis groups conflicting examples into five equal-sized score groups.Source: §G.3 · p. 24, Table G.1 · p. 25

G.4 CONTROL EXPERIMENTS

Controls indicate that probe signals reflect more than configuration novelty, and the appendix documents frozen probes, evaluation isolation, metric coverage, and supplementary benchmark comparisons.

G.4 CONTROL EXPERIMENTS

  • Randomly permuting conflict labels reduces probe accuracy to approximately chance, and controls indicate the probes rely on interactions between tendency and runtime tool information.Source: §G.4 · p. 25, Table G.2 · p. 25

G.5 DISENTANGLING CONFLICT FROM CONFIGURATION NOVELTY

  • Novelty-matched Bnovel receives lower conflict scores and classification accuracy than Btool, while conflict-score correlations with hallucination persist within novelty-matched terciles.The reported correlations are Spearman ρ=0.51–0.67; surface-form novelty contributes to but does not dominate the probe signal.Source: §G.5 · pp. 25–26, Table G.3 · p. 26
  • Fully ruling out distributional shift as a confound would require changing tendency strength while holding configuration fixed, which is not straightforward in a pretrained model.Source: §G.5 · p. 26

G.6 PROBE FREEZING AND § 5 DATA ISOLATION

  • The probes are frozen before EscapeGuard evaluation, and intervention regions are centered on model-specific conflict layers identified by probing.Evaluation examples are kept disjoint from the probing corpus to prevent test examples from influencing the conflict detector.Source: §H.1 · p. 26, §G.6 · p. 26

H.2 METRIC APPLICABILITY AND DATA ISOLATION

  • Metric coverage depends on benchmark annotations: some benchmarks support all four metrics, RelyToolBench supports TSHR and TUHR, and MetaTool supports TSHR only.Evaluation pools exclude overlap with the probing corpus.Source: §H.2 · pp. 26–27, Table H.2 · p. 27

H.3 FULL USAGE-LEVEL RESULTS

H.4 SUPPLEMENTARY GENERALIZATION ON GLAIVE

  • Glaive Function Calling V2 is supplementary evaluation on a held-out subset without query or normalized-schema overlap with probe-training examples.The results provide supplementary evidence for the main-text findings.Source: §H.4 · p. 27, Table H.6 · p. 28

H.5 FULL BASELINE COMPARISON ON QWEN-3.5-9B

  • On Qwen-3.5-9B, EscapeGuard consistently improves TSHR, TUHR, and ANHR, while its AVHR advantage remains limited.This matches the pattern reported for LLaMA-3.1-8B.Source: §H.5 · p. 28, Table H.7 · p. 28

H.6 CROSS-CONFIGURATION PERFORMANCE EVALUATION DETAILS

EscapeGuard reduces hallucination across tested tool configurations and models while leaving intrinsic tool-use priors unchanged; the appendix also reports stable hyperparameter ranges, modest inference overhead, and residual error patterns.

H.6 CROSS-CONFIGURATION PERFORMANCE EVALUATION DETAILS

  • Table 7 reuses the § 3 evaluation set, keeping variants from each query together for evaluation and statistical resampling.This makes the changes in parentheses comparable with Table 1.Source: §H.6 · p. 28
  • Across four additional models, EscapeGuard reduces hallucination under every tested configuration and brings C2 to a level comparable with C1.The cross-model evaluation uses C1, C2, and C5 as representative standard, tool-replacement, and mixed-change configurations.Source: §H.6 · p. 28, Table H.8 · p. 28

H.7 MECHANISM VALIDATION

  • EscapeGuard leaves the intrinsic prior unchanged while increasing reliance on the current tool configuration during generation.Its conflict gate avoids intervention when tool configuration is absent, unlike unconditional steering; configuration-span attention and current-setting output margins are expected to rise on high-conflict inputs.Source: §H.7 · p. 29, Table H.9 · p. 29

H.8 HYPERPARAMETER SENSITIVITY

  • For both main backbones, τ values 0.5–0.7 and λ values 1.4–2.4 yield TSHR within 1 pp of the respective optimum.The tested ranges are τ ∈ {0.4, 0.5, 0.6, 0.7, 0.8} and λ ∈ {0.6, 1.0, 1.4, 1.8, 2.4, 3.0}.Source: §H.8 · p. 29, Table H.10 · p. 29, Table H.11 · p. 29

H.9 EFFICIENCY AND PROBE RETRAINING

  • EscapeGuard adds 4.1%–5.9% to base inference latency, with attention modification applied only when the conflict gate activates.The probe forward pass contributes approximately 1.2–1.8 ms; the gate activates on approximately 35–45% of queries, and probes use fewer than 0.01% of total model parameters. Probe retraining takes approximately 15–25 minutes per model checkpoint.Source: §H.9 · pp. 29–30, Table H.12 · p. 30

H.10 ABLATION STUDY

  • The ablation study removes or replaces components and averages TSHR and TUHR across four benchmarks and two main backbones.The reported table presents ablation averages, with lower values better.Source: §H.10 · p. 30, Table H.13 · p. 30

H.11 ERROR-TYPE BREAKDOWN UNDER C4/C5

  • AVHR accounts for most residual TUHR under C4 and C5 for both models, indicating residual OHR is dominated by argument-value errors.The paper describes argument-value errors as the error type least associated with tendency conflict.Source: §H.11 · p. 30, Table H.14 · p. 30
Loading 2610.04409v1…