PaperlayerView exact revision on arXiv

Source-linked AI summary

Internal Representations as Indicators of Hallucinations in Agent Tool Selection

Kait Healy, Bharathi Srinivasan, Visakh Madathil, Jing Wu

arXiv:2601.05214v1cs.AI

TL;DR

Tool-calling hallucinations can produce incorrect tool choices, malformed parameters, and tool bypass, threatening reliable agent operation. The paper detects these errors from last-layer internal representations during the same generation pass, using unsupervised labels and lightweight classification. Across models, it reports accuracies from 72.7% to 86.4% with real-time operation and minimal overhead, while noting limits from reference-agreement labeling and function-equivalence matching.

  • Problem

    Tool-calling hallucinations create inappropriate selections, malformed parameters, and bypass behavior, while existing detection methods often require multiple generations or external validation.

    Source: Introduction · p. 1

  • Method

    The paper trains lightweight classifiers on last-layer representations from tool-call generation, using masked-call prediction and reference agreement to obtain unsupervised labels.

    Source: Introduction · p. 1, Problem Formulation · p. 2, Final-Layer Feature Extraction · p. 4

  • Results

    Accuracies range from 72.7% to 86.4% across multiple models, with GPT-OSS-20B achieving 86% precision in hallucination detection and real-time single-pass operation.

    Source: Conclusion and Limitations · p. 7

  • Takeaways & Limitations

    Internal representations contain discriminative information for real-time tool-calling hallucination detection with minimal computational overhead.

    Source: Introduction · p. 1

    Reference-agreement labeling inherits dataset-call and standardization problems, while function equivalence beyond string matching remains difficult without additional context.

    Source: Conclusion and Limitations · p. 7

Abstract

from arXiv · show

Large Language Models (LLMs) have shown remarkable capabilities in tool calling and tool usage, but suffer from hallucinations where they choose incorrect tools, provide malformed parameters and exhibit 'tool bypass' behavior by performing simulations and generating outputs instead of invoking specialized tools or external systems. This undermines the reliability of LLM based agents in production systems as it leads to inconsistent results, and bypasses security and audit controls. Such hallucinations in agent tool selection require early detection and error handling. Unlike existing hallucination detection methods that require multiple forward passes or external validation, we present a computationally efficient framework that detects tool-calling hallucinations in real-time by leveraging LLMs' internal representations during the same forward pass used for generation. We evaluate this approach on reasoning tasks across multiple domains, demonstrating strong detection performance (up to 86.4\% accuracy) while maintaining real-time inference capabilities with minimal computational overhead, particularly excelling at detecting parameter-level hallucinations and inappropriate tool selections, critical for reliable agent deployment.

Introduction

Tool-calling hallucinations can select inappropriate tools, generate malformed or incomplete parameters, bypass tools, and create security and reliability risks. The paper proposes real-time detection from LLM internal representations using unsupervised labeling and lightweight classifiers.

Motivation

  • Tool-calling hallucinations include inappropriate tool selection, malformed parameters, incorrect chaining, tool bypass, and semantically incorrect invocations.These errors can cause system failures, data corruption, or incorrect data access.Source: Introduction · p. 1
  • Rigid structural and semantic constraints make tool-call hallucinations difficult to detect reliably.Calls must use available functions, satisfy parameter types, interpret queries correctly, and include required arguments.Source: Introduction · p. 1

Approach

  • The method leverages internal LLM representations during tool-call generation for real-time hallucination detection.It uses unsupervised labeling by masking ground-truth calls, predicting functions, and training lightweight classifiers on contextualized embeddings.Source: Introduction · p. 1

Contributions

  • Internal representations contain discriminative information for detecting tool-calling hallucinations in reasoning tasks.The paper presents this as a key contribution of the approach.Source: Introduction · p. 1
  • The approach provides effective real-time detection with minimal computational overhead.This capability is identified as a central evaluation outcome.Source: Introduction · p. 1

Related Work

Prior hallucination detection methods commonly rely on external knowledge or multiple generations, while tool-use research has focused more on learning and benchmarking API calls than detecting incorrect usage. This work builds on evidence that internal representations encode information useful for real-time reliability monitoring.

Hallucination Detection

  • Earlier hallucination detectors typically analyze generated text using external knowledge sources or multiple model generations.Consistency-based methods assess whether repeated outputs agree or contradict verified information.Source: Hallucination Detection in Large… · pp. 1–2
  • Non Contradiction Probability and Semantic Similarity detect hallucinations through consistency across multiple sampled responses.NCP scores contradiction against curated beliefs, while semantic similarity measures meaning alignment between outputs.Source: Hallucination Detection in Large… · p. 2

Tool Use

Internal Representations

  • Prior work showed that hidden representations can encode veracity and support unsupervised real-time hallucination detection.These findings motivate applying internal-state classifiers to structured tool calling.Source: Internal Representation Analysis · p. 2

Agent Reliability

  • Agent research identifies reliability and consistent performance across diverse environments as practical deployment challenges.Evaluation frameworks and benchmarks have been developed to assess agent capabilities and behavior.Source: Agent Systems and Reliability · p. 2

Problem Formulation

The paper formulates tool-calling hallucination detection as binary classification over last-layer representations during generation. Hallucinations cover incorrect function selection, inappropriate calls, invalid or missing arguments, and bypassing tool use.

Problem Formulation

  • The task maps an LLM’s last-layer representation during tool-call generation to a binary correct-versus-hallucinated decision.The available functions form the toolkit, while the model generates a function and its arguments from a query and context.Source: Problem Formulation · pp. 2–3

Detection Pipeline

  • The training pipeline masks generated tool calls, predicts them again from context, and labels agreement with the reference call.A lightweight classifier is trained on final-layer representations and later gates execution using real-time scores.Source: Figure 1 · p. 3

Hallucination Types

  • Function selection error occurs when the generated function is not in the available function set.This represents invocation of a nonexistent function.Source: Problem Formulation · p. 3
  • Semantic, parameter, and completeness errors cover inappropriate calls, invalid arguments, and missing required parameters.Parameter validity is defined relative to the generated function’s valid argument domain.Source: Problem Formulation · p. 3
  • Tool bypass error occurs when the model generates an output without using the required tool.The framework unifies these five theoretical categories into binary detection of incorrectness.Source: Problem Formulation · p. 3

Method

The method detects tool-calling hallucinations by classifying final-layer representations collected during the same forward pass that generates each tool call. It extracts features from function, argument, and closing-delimiter tokens, then applies a lightweight classifier and an inline decision policy.

Detection framework

  • Final-layer representations from the generation pass are used to distinguish correct tool calls from hallucinated calls.The predicted call and hidden-state features are produced during one forward pass.Source: Method · p. 3

Label construction

  • Ground-truth and predicted calls are compared by function-name matching and canonicalized arguments to assign binary hallucination labels.A differing function name or mismatched normalized arguments receives label y_i = 1.Source: Data Collection and Label Generation · p. 4

Final-layer feature extraction

  • Features combine the initial function-name token, all argument tokens, and the closing delimiter from the final transformer layer.The three semantic regions are concatenated, with an optional linear projection.Source: Final-Layer Feature Extraction · p. 4

Classifier

  • A lightweight feed-forward network maps the feature vector to a hallucination probability and applies a validation-selected threshold.The classifier uses ReLU and sigmoid components, with binary crossentropy as the training objective.Source: Classifier Architecture and Training… · p. 4

Inline deployment

  • The system operates inline: flagged calls can be blocked, confirmed, replaced, or repaired, while unflagged calls proceed normally.Feature extraction reuses the generation pass, keeping computational overhead negligible for latency-sensitive deployment.Source: Inference Protocol · p. 4

Experimental Setup

The evaluation uses multiple open-source LLMs, specialized tool-calling agents, and comparisons with multi-sampling consistency baselines. It measures classification performance while testing the proposed single-pass efficiency advantage.

Models

  • Three open-source models—Qwen7B, GPT-OSS-20B, and Llama-3.1-8B—provide architectural and scale diversity for evaluation.The models range from 7B to 20B parameters and use different attention, normalization, positional-encoding, and activation designs.Source: Target Models · p. 4

Baseline Methods

  • NCP and semantic similarity are baseline detectors that compare tool calls and parameters across three agent invocations.NCP measures consistency with verified facts or premises, while semantic similarity uses cosine similarity.Source: Baseline Methods · p. 4
  • 5× computational overhead is required by both baselines because they use multiple forward passes instead of the proposed single-pass approach.The comparison targets the trade-off between consistency-based detection and inline inference cost.Source: Baseline Methods · p. 4

Tasks and agents

  • The evaluation uses five specialized agents covering arithmetic, finance, health, sustainability, and digital commerce tasks.These agents are built from the Glaive dataset and target domains where tool-calling precision matters.Source: Dataset Construction · p. 5, Agent Categories · p. 5

Data processing

  • Each model receives 2,411 tool-calling instances, with canonicalized tool names and parameters used for consistent evaluation.Instances contain the query, context, ground-truth call, predicted call, and cached final-layer representations.Source: Agent Categories · p. 5

Training and evaluation

  • The protocol uses a 60%/20%/20% train/validation/test split, validation-based threshold selection, temperature scaling, and precision, recall, F1-score, and accuracy reporting.Dedicated two-layer MLP classifiers are trained with model-specific input dimensions and regularization.Source: Training Configuration · p. 5

Baseline Comparison

The approach is compared with sampling-based NCP and semantic-similarity baselines for tool-calling hallucination detection. Internal LLM states provide competitive detection with higher recall and avoid repeated sampling during inference.

  • NCP and semantic similarity detect tool-calling hallucinations by generating multiple samples and measuring consistency.Source: Baseline Comparison · p. 5
  • Internal-state detection achieves higher recall than both baselines across all evaluated models, while the baselines achieve higher precision and overall accuracy.Source: Baseline Comparison · p. 5
  • Single-pass internal representations eliminate the repeated sampling or post-hoc scoring required by black-box baselines.Source: Baseline Comparison · p. 5

Main Results

The final-layer representation approach is evaluated across GPT-OSS-20B, Llama-3.1-8B, and Qwen-7B. Results show strong classification performance and suitability for real-time deployment using one forward pass.

  • Table 2 reports classification results for GPT-OSS-20B, Llama-3.1-8B, and Qwen-7B using final-layer features.Source: Main Results · p. 6
  • 86% accuracy and 86% precision are achieved by GPT-OSS-20B, with 53% recall and macro-averaged F1=0.79.Source: Main Results · p. 6
  • Llama-3.1-8B shows more balanced class performance, with 73% versus 71% precision and a 1375:1036 class ratio.Source: Main Results · p. 6
  • Final-layer representations contain enough information to distinguish correct from hallucinated tool calls while requiring only one forward pass.Source: Main Results · p. 6

Ablation Study on Feature Extraction Methods

The ablation study evaluates ten feature-extraction strategies on Qwen-7B using identical training and evaluation settings. Simple sequence-wide aggregation performs well, while greater feature complexity provides no consistent advantage.

Feature Extraction Setup

Feature Extraction Methods

Evaluation Setup

  • All methods are tested on Qwen7B with the digital commerce assistant dataset, identical 70-30 splits, optimization settings, and evaluation metrics.Source: Advanced Aggregation Methods · p. 7

Ablation Findings

  • Mean pooling provides the best balance of performance and simplicity, while higher-dimensional aggregation does not necessarily improve results.Source: Advanced Aggregation Methods · p. 7
  • Sequence-wide aggregation consistently outperforms single-token approaches, indicating that hallucination signals are distributed across the generated sequence.Source: Advanced Aggregation Methods · p. 7

Recommendation

  • Last-layer mean pooling is recommended as the default because it combines performance, computational efficiency, and implementation simplicity.Source: Advanced Aggregation Methods · p. 7

Conclusion and Limitations

The paper reports strong accuracy across multiple models and practical real-time operation from single-pass last-layer representations. It also identifies limitations in reference-based labeling and function-equivalence assessment.

  • 72.7% to 86.4% accuracy is achieved across multiple models using internal representations for tool-calling hallucination detection.Source: Conclusion and Limitations · p. 7
  • GPT-OSS-20B achieves 86% precision while maintaining balanced performance across classes, and the method operates in real time from one forward pass.Source: Conclusion and Limitations · p. 7
  • Reference-agreement labels inherit dataset-call and standardization issues, while function equivalence beyond string matching remains difficult without additional context.Source: Conclusion and Limitations · p. 7

Related papers

Loading 2601.05214v1…