Source-linked AI summary
Internal Representations as Indicators of Hallucinations in Agent Tool Selection
Kait Healy, Bharathi Srinivasan, Visakh Madathil, Jing Wu
TL;DR
Tool-calling hallucinations can produce incorrect tool choices, malformed parameters, and tool bypass, threatening reliable agent operation. The paper detects these errors from last-layer internal representations during the same generation pass, using unsupervised labels and lightweight classification. Across models, it reports accuracies from 72.7% to 86.4% with real-time operation and minimal overhead, while noting limits from reference-agreement labeling and function-equivalence matching.
Problem
Tool-calling hallucinations create inappropriate selections, malformed parameters, and bypass behavior, while existing detection methods often require multiple generations or external validation.
Method
The paper trains lightweight classifiers on last-layer representations from tool-call generation, using masked-call prediction and reference agreement to obtain unsupervised labels.
Results
Accuracies range from 72.7% to 86.4% across multiple models, with GPT-OSS-20B achieving 86% precision in hallucination detection and real-time single-pass operation.
Takeaways & Limitations
Internal representations contain discriminative information for real-time tool-calling hallucination detection with minimal computational overhead.
Takeaways & Limitations
Reference-agreement labeling inherits dataset-call and standardization problems, while function equivalence beyond string matching remains difficult without additional context.
Abstract
from arXiv · showhide
Large Language Models (LLMs) have shown remarkable capabilities in tool calling and tool usage, but suffer from hallucinations where they choose incorrect tools, provide malformed parameters and exhibit 'tool bypass' behavior by performing simulations and generating outputs instead of invoking specialized tools or external systems. This undermines the reliability of LLM based agents in production systems as it leads to inconsistent results, and bypasses security and audit controls. Such hallucinations in agent tool selection require early detection and error handling. Unlike existing hallucination detection methods that require multiple forward passes or external validation, we present a computationally efficient framework that detects tool-calling hallucinations in real-time by leveraging LLMs' internal representations during the same forward pass used for generation. We evaluate this approach on reasoning tasks across multiple domains, demonstrating strong detection performance (up to 86.4\% accuracy) while maintaining real-time inference capabilities with minimal computational overhead, particularly excelling at detecting parameter-level hallucinations and inappropriate tool selections, critical for reliable agent deployment.
Introduction
Tool-calling hallucinations can select inappropriate tools, generate malformed or incomplete parameters, bypass tools, and create security and reliability risks. The paper proposes real-time detection from LLM internal representations using unsupervised labeling and lightweight classifiers.
- Motivation: Tool-calling hallucinations include inappropriate tool selection, malformed parameters, incorrect chaining, tool bypass, and semantically incorrect invocations.These errors can cause system failures, data corruption, or incorrect data access.
- Motivation: Rigid structural and semantic constraints make tool-call hallucinations difficult to detect reliably.Calls must use available functions, satisfy parameter types, interpret queries correctly, and include required arguments.
- Approach: The method leverages internal LLM representations during tool-call generation for real-time hallucination detection.It uses unsupervised labeling by masking ground-truth calls, predicting functions, and training lightweight classifiers on contextualized embeddings.
- Contributions: Internal representations contain discriminative information for detecting tool-calling hallucinations in reasoning tasks.The paper presents this as a key contribution of the approach.
- Contributions: The approach provides effective real-time detection with minimal computational overhead.This capability is identified as a central evaluation outcome.
Related Work
Prior hallucination detection methods commonly rely on external knowledge or multiple generations, while tool-use research has focused more on learning and benchmarking API calls than detecting incorrect usage. This work builds on evidence that internal representations encode information useful for real-time reliability monitoring.
- Hallucination Detection: Earlier hallucination detectors typically analyze generated text using external knowledge sources or multiple model generations.Consistency-based methods assess whether repeated outputs agree or contradict verified information.
- Hallucination Detection: Non Contradiction Probability and Semantic Similarity detect hallucinations through consistency across multiple sampled responses.NCP scores contradiction against curated beliefs, while semantic similarity measures meaning alignment between outputs.
- Tool Use: Toolformer and ToolLLM advanced automated tool learning but did not address detection of incorrect tool usage.Related benchmarks evaluate API-call accuracy, multi-step reasoning, and complex tool interactions.
- Internal Representations: Prior work showed that hidden representations can encode veracity and support unsupervised real-time hallucination detection.These findings motivate applying internal-state classifiers to structured tool calling.
- Agent Reliability: Agent research identifies reliability and consistent performance across diverse environments as practical deployment challenges.Evaluation frameworks and benchmarks have been developed to assess agent capabilities and behavior.
Problem Formulation
The paper formulates tool-calling hallucination detection as binary classification over last-layer representations during generation. Hallucinations cover incorrect function selection, inappropriate calls, invalid or missing arguments, and bypassing tool use.
- Problem Formulation: The task maps an LLM’s last-layer representation during tool-call generation to a binary correct-versus-hallucinated decision.The available functions form the toolkit, while the model generates a function and its arguments from a query and context.
- Detection Pipeline: The training pipeline masks generated tool calls, predicts them again from context, and labels agreement with the reference call.A lightweight classifier is trained on final-layer representations and later gates execution using real-time scores.
- Hallucination Types: Function selection error occurs when the generated function is not in the available function set.This represents invocation of a nonexistent function.
- Hallucination Types: Semantic, parameter, and completeness errors cover inappropriate calls, invalid arguments, and missing required parameters.Parameter validity is defined relative to the generated function’s valid argument domain.
- Hallucination Types: Tool bypass error occurs when the model generates an output without using the required tool.The framework unifies these five theoretical categories into binary detection of incorrectness.
Method
The method detects tool-calling hallucinations by classifying final-layer representations collected during the same forward pass that generates each tool call. It extracts features from function, argument, and closing-delimiter tokens, then applies a lightweight classifier and an inline decision policy.
- Detection framework: Final-layer representations from the generation pass are used to distinguish correct tool calls from hallucinated calls.The predicted call and hidden-state features are produced during one forward pass.
- Label construction: Ground-truth and predicted calls are compared by function-name matching and canonicalized arguments to assign binary hallucination labels.A differing function name or mismatched normalized arguments receives label y_i = 1.
- Final-layer feature extraction: Features combine the initial function-name token, all argument tokens, and the closing delimiter from the final transformer layer.The three semantic regions are concatenated, with an optional linear projection.
- Classifier: A lightweight feed-forward network maps the feature vector to a hallucination probability and applies a validation-selected threshold.The classifier uses ReLU and sigmoid components, with binary crossentropy as the training objective.
- Inline deployment: The system operates inline: flagged calls can be blocked, confirmed, replaced, or repaired, while unflagged calls proceed normally.Feature extraction reuses the generation pass, keeping computational overhead negligible for latency-sensitive deployment.
Experimental Setup
The evaluation uses multiple open-source LLMs, specialized tool-calling agents, and comparisons with multi-sampling consistency baselines. It measures classification performance while testing the proposed single-pass efficiency advantage.
- Models: Three open-source models—Qwen7B, GPT-OSS-20B, and Llama-3.1-8B—provide architectural and scale diversity for evaluation.The models range from 7B to 20B parameters and use different attention, normalization, positional-encoding, and activation designs.
- Baseline Methods: NCP and semantic similarity are baseline detectors that compare tool calls and parameters across three agent invocations.NCP measures consistency with verified facts or premises, while semantic similarity uses cosine similarity.
- Baseline Methods: 5× computational overhead is required by both baselines because they use multiple forward passes instead of the proposed single-pass approach.The comparison targets the trade-off between consistency-based detection and inline inference cost.
- Tasks and agents: The evaluation uses five specialized agents covering arithmetic, finance, health, sustainability, and digital commerce tasks.These agents are built from the Glaive dataset and target domains where tool-calling precision matters.
- Data processing: Each model receives 2,411 tool-calling instances, with canonicalized tool names and parameters used for consistent evaluation.Instances contain the query, context, ground-truth call, predicted call, and cached final-layer representations.
- Training and evaluation: The protocol uses a 60%/20%/20% train/validation/test split, validation-based threshold selection, temperature scaling, and precision, recall, F1-score, and accuracy reporting.Dedicated two-layer MLP classifiers are trained with model-specific input dimensions and regularization.
Baseline Comparison
The approach is compared with sampling-based NCP and semantic-similarity baselines for tool-calling hallucination detection. Internal LLM states provide competitive detection with higher recall and avoid repeated sampling during inference.
- NCP and semantic similarity detect tool-calling hallucinations by generating multiple samples and measuring consistency.
- Internal-state detection achieves higher recall than both baselines across all evaluated models, while the baselines achieve higher precision and overall accuracy.
- Single-pass internal representations eliminate the repeated sampling or post-hoc scoring required by black-box baselines.
Main Results
The final-layer representation approach is evaluated across GPT-OSS-20B, Llama-3.1-8B, and Qwen-7B. Results show strong classification performance and suitability for real-time deployment using one forward pass.
- Table 2 reports classification results for GPT-OSS-20B, Llama-3.1-8B, and Qwen-7B using final-layer features.
- 86% accuracy and 86% precision are achieved by GPT-OSS-20B, with 53% recall and macro-averaged F1=0.79.
- Llama-3.1-8B shows more balanced class performance, with 73% versus 71% precision and a 1375:1036 class ratio.
- Final-layer representations contain enough information to distinguish correct from hallucinated tool calls while requiring only one forward pass.
Ablation Study on Feature Extraction Methods
The ablation study evaluates ten feature-extraction strategies on Qwen-7B using identical training and evaluation settings. Simple sequence-wide aggregation performs well, while greater feature complexity provides no consistent advantage.
- Feature Extraction Setup: Ten feature-extraction methods are evaluated, converting variable-length hidden-state sequences into fixed-size vectors.
- Feature Extraction Methods: The study compares mean, max, and min pooling, token-based features, statistical summaries, attention-weighted pooling, and multi-scale combinations.
- Evaluation Setup: All methods are tested on Qwen7B with the digital commerce assistant dataset, identical 70-30 splits, optimization settings, and evaluation metrics.
- Ablation Findings: Mean pooling provides the best balance of performance and simplicity, while higher-dimensional aggregation does not necessarily improve results.
- Ablation Findings: Sequence-wide aggregation consistently outperforms single-token approaches, indicating that hallucination signals are distributed across the generated sequence.
- Recommendation: Last-layer mean pooling is recommended as the default because it combines performance, computational efficiency, and implementation simplicity.
Conclusion and Limitations
The paper reports strong accuracy across multiple models and practical real-time operation from single-pass last-layer representations. It also identifies limitations in reference-based labeling and function-equivalence assessment.
- 72.7% to 86.4% accuracy is achieved across multiple models using internal representations for tool-calling hallucination detection.
- GPT-OSS-20B achieves 86% precision while maintaining balanced performance across classes, and the method operates in real time from one forward pass.
- Reference-agreement labels inherit dataset-call and standardization issues, while function equivalence beyond string matching remains difficult without additional context.