Source-linked AI summary
The Calls are Coming from Inside the Model: Investigating Probe-based Detection of Tool-Calling Errors in LLMs
Eric Yeats, Brendan Kennedy, Loc Truong, John Buckheit, Jung Lee, Jesse Friedbaum, John Emanuello, Henry Kvinge
TL;DR
Tool-calling errors are difficult to monitor from outputs alone, motivating hidden-state detection. This paper trains linear probes across 18 BFCL-evaluated LLMs and finds effective detection, especially for larger models and intermediate layers, with transfer to unseen error types.
Problem
Tool-calling errors are structurally and semantically diverse, while hidden-state detection has been studied mainly for factual question answering and summarization.
Method
The paper trains regularized linear probes on frozen last-token hidden states from individual layers to classify correct versus incorrect tool-calls.
Results
Probing is effective across 18 models, with larger and middle-layer representations performing better and probes transferring to held-out error types.
Takeaways & Limitations
Hidden-state probes can support monitoring that detects varied tool-calling errors, including error types unseen during probe training.
Takeaways & Limitations
The BFCL data use clean schemas and 750 examples per model, and all studied models are open-weight, limiting coverage of production and closed-model settings.
Abstract
from arXiv · showhide
The hidden states of large language models (LLMs) are known to capture rich information relating to model knowledge and behavior that can be hard to extract from examination of input and output alone. As LLM-based systems increasingly interface with the external world, one area of concern is detecting incorrect or improper use of tools. Motivated by this, we study the effectiveness of using linear probes to detect incorrect tool-calls, measuring probe efficacy across 18 tool-calling LLMs evaluated on the Berkeley Function Calling Leaderboard. Overall, we find that probing is an effective means to catch a range of different tool-calling errors, including errors arising from using an argument that has the wrong value but the correct type, which might not be recorded by standard logging frameworks. Important factors in success include model size, probing layer, and model post-training type. We also show that probes are capable of generalizing to novel types of errors, which is critical in real world deployments.
1 Introduction
Tool-calling reliability matters for agentic systems, but hidden-state probing has mainly been studied for factual and summarization errors. This work extends probing to tool-call errors and tests its effectiveness and generalization across diverse models.
- Tool-calling systems must select appropriate tools, provide correct arguments, and chain calls reliably.
- Hallucination-detection research suggests that LLM hidden states contain signals predictive of model errors.
- Tool-calling errors are structural and semantic, span diverse categories, and remain underexplored with hidden-state methods.
- Prior work used last-layer probes on only three models and did not test variation across sizes, architectures, training regimes, or broader error notions.
- This study evaluates probes across 18 BFCL models and tests transfer from observed error types to held-out types.
2 Methodology
The method extracts final-token hidden states from each recorded layer and trains independent regularized linear probes to classify tool-call correctness. Performance is assessed with threshold-independent detection metrics.
- The final generated tool-call token provides one hidden-state vector per recorded layer and tool-call.
- A linear probe is a lightweight classifier trained on frozen hidden states to predict whether a tool-call is correct.
- Labels mark correct calls as 0 and incorrect calls as 1 in the probe-training dataset.
- Each layer receives an independently trained ℓ2-regularized logistic probe after feature standardization.
- Probe quality is measured with AUROC and AUPR, while FPR@90%TPR characterizes monitoring costs at a fixed threshold.
3 Experimental Setup
The experiments evaluate tool-call error detection across BFCL models, error categories, and hidden layers. Errors are grouped into a collective positive class against correct calls for stratified probe evaluation.
- The study evaluates 18 LLMs on four BFCL splits totaling 750 examples per model and records every fifth transformer layer.
- The BFCL AST checker identifies syntax, type, hallucinated-name, unnecessary-call, missing-parameter, incorrect-value, and correct outcomes.
- Figure 1 compares AUROC across normalized layer depth for approximately 8B and 70B models, with means and ±1σ shading.
- Incorrect-value and unnecessary-tool-call errors are difficult to detect through traditional code checks or constrained decoding.
- All error categories form the positive class, correct calls form the negative class, and examples use stratified 70% train and 30% test splits.
4 Results
Probes detect tool-calling errors effectively, with stronger performance in larger models and middle-to-late layers. They also transfer across held-out error types, supporting detection of broader tool-calling incorrectness.
- Effect of Model Size and Probing Layer: Most models exceed AUROC 0.80 and normalized AUPR 3.5 across most layers.
- Effect of Model Size and Probing Layer: Larger models improve over smaller models by approximately 0.06 AUROC and 2 normalized AUPR on average.
- Effect of Model Size and Probing Layer: Probe effectiveness peaks in middle-to-late layers, suggesting the final layer is not always optimal for tool-call error detection.
- Tool-Calling Accuracy vs Probe Efficacy: Within the approximately 8B group, R2 = 0.005, while larger models with about 85% tool-calling accuracy show approximately +0.08 higher probe AUROC.
- Behavior of Probes on Novel Error Types: Transfer across disjoint error-type splits reaches AUROC > 0.75 and normalized AUPR > 6 for many models in both directions.
- Behavior of Probes on Novel Error Types: Large models improve transfer by approximately 0.05-0.1 AUROC and 10-40% normalized AUPR relative to smaller models.
5 Conclusion
The paper finds that linear probes can detect tool-calling errors, with effectiveness associated with model size, probing layer, and post-training regime. Middle-layer probes may capture abstract correctness features that generalize to unseen error types.
- Probe effectiveness is associated with model size, probing layer, and post-training regime.
- Middle-layer probes capture features associated with abstract tool-calling correctness and can generalize to error types unseen during training.
- The findings support using probes as part of a robust monitoring strategy for LLMs.
A Limitations
The study is constrained by the scope and scale of its benchmark, incomplete ablations of probe design choices, and confounding in cross-model comparisons. Its findings may not extend to more complex production settings or establish causal effects.
- BFCL uses relatively clean schemas and may not represent longer contexts, multi-turn conversations, or ambiguous production tool-calls.With 750 examples per model, rare error categories may yield noisy AUPR estimates.
- The study does not compare linear probes with nonlinear probes or alternative token-aggregation strategies.These design choices may interact with model size or finetuning type in ways the analysis does not capture.
- Cross-model findings are correlational rather than causal because model size and finetuning type are confounded with other model differences.These differences include base model, training-data composition, and alignment procedure.
B Metric Background
The study uses AUROC and normalized AUPR to evaluate probe-based error detection, while also collecting operating-cost results at fixed thresholds. Experiments cover 18 models and use a two-stage generation and hidden-state extraction process.
- Metric Background: AUROC measures whether incorrect tool-calls receive higher probe scores than correct ones and is invariant to class imbalance.Its random-performance value is 0.5, while 1.0 indicates perfect discrimination.
- Metric Background: AUPR summarizes precision-recall tradeoffs across decision thresholds, with a random baseline equal to the positive-class rate.The study normalizes AUPR by that rate for cross-model comparison.
- Evaluation: The evaluation covers 18 tool-calling LLMs and 750 BFCL examples per model across four splits.The splits are simple_python, simple_java, simple_javascript, and multiple.
- Implementation: The pipeline separates rapid response generation from teacher-forced hidden-state extraction with a HuggingFace implementation.vLLM generates tool-calls, while transformers encodes prompts and responses without autoregressive generation.
- Implementation: Hidden states are collected from the final token of every fifth layer, and probes use normalized, ℓ2-regularized logistic regression.Training uses C = 1, 2000 iterations, and the liblinear solver.
D Tool-call Performance Results
The study catalogs tool-call performance and error patterns across models, with tables organized by model size and evaluation setting. Most models consistently produce incorrect-value and type errors, while high-error models are excluded from probe analyses.
- Error patterns: Most models consistently generate incorrect value errors and type errors, although error-type prevalence varies somewhat between models.
- Tool-call accuracy: Table 1 reports tool-call accuracy by model and evaluation setting, with models sorted by parameter count.Single denotes single-tool use, while Multi requires selecting and correctly employing one tool from multiple candidates.
- Analysis inclusion: Models with high error rates of at least 20% are excluded from probe-based analyses and shown with grayed-out names.
- Error patterns: Table 2 breaks down tool-call errors across all evaluated domains, including simple_python and simple_javascript.The breakdown distinguishes fully correct calls from error categories.
E Effect of Model Finetuning on Probe Efficacy
The paper compares probe efficacy between general instruction-tuned and tool-finetuned ∼8B models. General instruction-tuned models perform slightly better in the correlational comparison, and a controlled same-base-model experiment reports the same direction, while causal interpretation remains limited.
- Comparative results: +0.05 AUROC and +0.5 normalized AUPR favor general instruction-tuned over tool-finetuned ∼8B models.The comparison uses probes trained and applied within each post-training group.
- Model groups: Tool-finetuned models are explicitly trained on function-calling or agent-action datasets, unlike the general instruction-tuned comparison group.
- Interpretation: The reported finetuning-related findings are correlational, and training-data volume or training methodology could also influence probing results.
- Controlled experiment: In a controlled same-base-model comparison, Llama-3.1-8B-Instruct probes achieve AUROCs of 90–95%, versus 72–83% for Hermes-3-Llama-3.1-8B.FPR@90%TPR is 15–35% for Llama-3.1-8B-Instruct and 50–60% for Hermes-3-Llama-3.1-8B.
F Label Space Transfer Experiments
Label space transfer tests whether probes trained on some tool-call error categories can separate different, unseen error categories from correct calls. The experiments use balanced disjoint error splits and report AUROC and normalized AUPR for transfer in both directions.
- Experimental design: The experiments divide six error categories into disjoint three-category splits, with correct tool-calls always treated as the negative class.Split A contains incorrect value, missing parameter, and syntax errors; Split B contains unnecessary tool-call, hallucinated tool or parameter, and type errors.
- Experimental design: The C/D partition pairs Split C—missing parameter, type error, and syntax error—with Split D—incorrect value, unnecessary tool-call, and hallucinated tool or parameter.The aggregate positive rate is approximately balanced between the subsets for most models to support fair transfer comparisons.
- Results: Figure 5 reports AUROC and normalized AUPR for C-to-D and D-to-C label-space transfer, using means with ±1σ shaded regions.The figure compares probe performance when trained on one split and evaluated on the other.
- Results: For the C/D partition, probes on larger models appear as good or better at separating novel errors from correct tool-calls than probes on smaller models.This pattern is reported across the transfer efficacy metrics shown in Figure 5.
G Results on False Positive Rate (FPR) at 90% True Positive Rate (TPR)
The study uses FPR@90%TPR to quantify the operational cost of monitoring tool-call error probes at a fixed score threshold. Larger models have lower false-positive rates than smaller models in later layers, while the figures show how detection varies across layer depth.
- Results: For 70B models, later-layer FPR@90%TPR averages approximately 34% with a 14% standard deviation.FPR@90%TPR characterizes the operational cost of monitoring at a set probe score threshold.
- Results: For ∼8B models, later-layer FPR@90%TPR averages between 50-60% with a 10% standard deviation.Lower FPR@90%TPR indicates lower operational cost at the specified true-positive rate.
- Figure guides: Figure 6 plots normalized AUPR against hidden activation layer depth for ∼8B and 70B models, with solid means and ±1σ shaded areas.The x-axis is hidden activation layer depth and the y-axis is normalized AUPR.
- Figure guides: Figure 8 plots FPR@90%TPR against normalized probing layer depth, where lower values are better, using means and ±1σ shaded areas.The figure directly displays how false-positive operating cost changes with normalized layer depth.