Source-linked AI summary
Do Agents Know When They Succeed? Calibrating Agent Confidence from Internal Representations
Priyanka Mary Mammen, Emil Joswin, Srujananjali Medicherla
TL;DR
Agentic systems need confidence estimates for multi-turn tasks whose failures involve planning, tool use, and dynamic environmental interaction. This paper derives confidence from internal residual-stream representations using LTD and ARP, and reports consistent improvement over surface and sequence-based baselines across three coding environments and three model families. The resulting monitor uses representations from standard forward passes without prompt changes or multi-rollout sampling.
Problem
The paper asks whether internal model representations provide stronger signals of eventual task success than traditional confidence methods in multi-turn agentic workflows.
Method
LTD summarizes residual-stream changes across agent trajectories, while ARP predicts success from representations formed at action decisions.
Results
Internal frameworks consistently outperform token-level and other external calibration baselines across Bash, SQL, and Python tasks and three model families.
Takeaways & Limitations
Internal-state probing provides a passive, zero-overhead reliability safeguard using standard forward-pass representations without prompt modifications or multi-rollout sampling.
Abstract
from arXiv · showhide
As agentic systems getting adopted rapidly in safety critical applications, it is vital to measure the confidence associated with the agentic actions. In comparison to the traditional machine learning systems, agentic workflows have complex failure modes with planning, tool invocation and dynamic environment interactions. In this paper, we investigate whether model's internal representations provide stronger signals of eventual task success in multi-turn agentic setups. We introduce two complementary methods: Latent Trajectory Dynamics (LTD), which summarizes changes in residual-stream representations across an an interaction trajectory, and the Action Representation Probe (ARP), which predicts success from representations formed at action decisions. Across three interactive benchmarks (Bash, SQL, Python) and three model families (Qwen14B, Qwen7B, DeepSeek6.7B), our methods consistently outperform surface level generation and sequence-based calibration baselines providing a zero-overhead reliability monitor that requires neither prompt alterations nor multi-sample rollouts.
1 Introduction
Agentic systems need confidence estimates despite long-horizon planning, tool use, feedback, and absent real-time ground-truth labels. The paper therefore studies mechanistic confidence signals from internal representations and introduces two complementary frameworks for multi-turn agents.
- Safety-critical agents require reliability and failure detection because real-time execution lacks ground-truth labels for immediate evaluation.The motivating domains include software engineering, trading, finance, clinical decision making, and robotics.
- Single-turn confidence methods struggle in agentic workflows because agents interact with tools, external environments, and multi-turn feedback over long-horizon tasks.The paper contrasts these workflows with isolated text generation.
- Mechanistic approaches are investigated to derive granular confidence signals from model behavior in multi-turn agentic setups.Prior mechanistic work in agentic settings is described as mostly single-turn and focused on tool-need decisions or reasoning tasks.
- LTD measures geometric stability and drift in hidden-state execution paths, while ARP predicts task success from action-span residual states.Both frameworks operate directly on the agent’s hidden activations.
- Across Bash, SQL, and Python tasks and three open-weight coding models, internal methods consistently outperform surface baselines under leak-free cross-validation.The paper presents internal state probing as a zero-overhead reliability safeguard.
2 Methodology
The methodology estimates trajectory-level success from internal residual-stream representations. LTD summarizes state transitions across execution phases, while ARP aggregates action-decision representations and calibrates a success probe.
- 2.1 Problem Formulation: Each coding-agent trajectory contains multi-step generations and observations, with a binary label indicating eventual task success.The formulation seeks a trajectory-level confidence estimate.
- 2.2.1 Latent Trajectory Dynamics: LTD retains residual-stream states at observation, reasoning, action, and feedback endpoints to characterize internal changes across a trajectory.Stored generations are teacher-forced to recover the relevant residual-stream states.
- 2.2.1 Latent Trajectory Dynamics: LTD summarizes reasoning, commitment, action, and feedback transitions using means, final values, variability, trends, path efficiency, and unavailable-transition indicators.These features form a 28-dimensional representation followed by an L2-regularized logistic model.
- 2.2.2 Action Representation Probe: ARP mean-pools final-layer residual states from nonterminal action spans to test whether action decisions encode eventual task success.Terminal submit actions are excluded before pooling.
- 2.2.2 Action Representation Probe: ARP standardizes episode representations, projects them onto 64 principal components, and predicts success with a calibrated L2-regularized logistic probe.All preprocessing, fitting, and calibration occur within training folds without access to the outer held-out fold.
3 Experiments
The experiments evaluate confidence estimation on interactive coding tasks with deterministic execution-based verification. They compare calibrated output probabilities, trajectory dynamics, and internal representations across three coding models and environments.
- Experimental Setup: The evaluation uses Qwen2.5-Coder-7B-Instruct, Qwen2.5-Coder-14B-Instruct-AWQ, and DeepSeek-Coder-6.7B-Instruct with greedy decoding.Residual states are obtained by teacher-forcing generated trajectories on the same offline model.
- Experimental Setup: Interactive Python, SQL, and Bash benchmarks provide multi-turn problems with deterministic ground-truth verification through programmatic unit tests and execution harnesses.This avoids LLM-as-a-judge evaluation noise.
- Calibration and Performance Results: The study compares calibrated log probability and HTC with LTD and ARP using AUROC, Brier score, and ECE.AUROC measures discrimination, while ECE and Brier score assess calibration-related error.
- Calibration and Performance Results: Internal methods outperform external baselines on AUROC or Brier score in every model–environment setting, achieving the best AUROC and Brier score in all nine settings.The reported comparison spans three environments and three model families.
4 Related Work
Related confidence-estimation work includes methods adapted from static language-model outputs, trajectory-level uncertainty methods, and mechanistic signals from execution traces or embeddings. The paper’s evaluation table organizes these approaches from surface outputs to internal representations.
- Prior confidence estimation: Early agent-confidence approaches adapt temperature scaling, semantic entropy, and verbalized confidence from static language models.These methods focus on single-turn outputs rather than complete multi-step trajectories.
- Evaluation framing: Table 1 orders calibration paradigms from calibrated log probabilities through sequence dynamics to LTD and ARP internal mechanisms.The table uses five-fold nested cross-validation on frozen protocols.
- Trajectory-aware methods: Later methods model uncertainty propagation across reasoning steps or derive classifiers from trajectories.This line of work addresses interaction structure more directly than static output methods.
- Mechanistic signals: Mechanistic approaches examine signals such as execution traces and embedding-based probes.The paper positions these signals alongside trajectory-based confidence methods.
5 Conclusion
The paper presents internal residual-stream monitoring as a reliable confidence signal for autonomous agents, outperforming surface and sequence-based alternatives without modifying prompts or using multiple rollouts.
- Conclusion: LTD and ARP consistently outperform token-level calibration methods across Bash, SQL, and Python benchmarks and three model families.The methods operate on internal representations produced during standard forward passes.
- Conclusion: Internal residual-stream representations provide a more reliable confidence signal than surface generation text or sequence-level attention statistics.
- Conclusion: The monitoring approach requires neither prompt modifications nor multi-rollout sampling, enabling passive reliability safeguards during deployment.
- Conclusion: The evaluated agents interact through constrained Bash, Python, and SQL action formats while receiving execution feedback across turns.Bash uses one shell command per action, Python supports persistent execution and revision, and SQL uses one statement per action.
C Benchmark Environments & Trajectory Statistics
Evaluation uses three interactive environments from the InterCode benchmark suite and reports trajectory statistics alongside empirical task success rates across environments and model families.
- Benchmark environments: The evaluation covers three distinct interactive agent environments from the InterCode benchmark suite.
- Benchmark environments: InterCode-Bash tests multi-turn Linux shell execution with command feedback and recovery from runtime errors.
- Benchmark environments: InterCode-SQL tests natural-language-to-SQL generation and multi-turn database querying across Spider schemas.
- Trajectory statistics: Table 2 reports trajectory statistics and empirical task success rates across benchmark environments and evaluated model families.
D Data Hygiene & Protocol Freezing
The evaluation removes corrupted task instances, freezes cross-validation boundaries, and extracts leakage-controlled trajectory representations for LTD and ARP.
- Data hygiene: The dataset audit excludes malformed SQL references and Python tasks with test-harness execution exceptions before cross-validation partitioning.These exclusions prevent correct executions from being penalized or spurious outputs from being rewarded.
- Protocol freezing: Five stratified outer folds use fixed random seeds, with every baseline and internal model evaluated on identical fold boundaries.
- LTD representations: LTD summarizes residual-stream trajectories using 28 kinematic features covering directional progression, layer drift, path efficiency, and phase-transition velocity.
- ARP representations: ARP extracts final hidden states from action spans, excludes terminal submission commands, and projects standardized episode vectors onto 64 training-fold principal components.
F Hyperparameter Tuning & Training Grids
All methods use comparable regularized logistic probes with nested cross-validation, Brier-score hyperparameter selection, and probability calibration applied without held-out-fold leakage.
- Training setup: All baselines, LTD, and ARP use L2-penalized logistic regression for comparability across representation spaces.
- Training setup: The optimization uses L-BFGS with a maximum of 2,000 iterations and fixed random seeds.
- Hyperparameter tuning: Hyperparameters are selected by inner four-fold stratified cross-validation optimizing negative Brier score within each outer fold.
- Probability calibration: A monotone Platt calibrator maps probe probabilities to reported confidence while preserving AUROC ranking through a non-negative slope constraint.
- External baselines: Calibrated Logprob aggregates non-terminal token probabilities, while HTC summarizes the same scope with 48 trajectory features.Terminal submit generations are excluded because they primarily signal protocol termination rather than solution correctness.
H Calibration and Evaluation Protocol
The evaluation protocol uses shared task populations and frozen fold assignments, with leakage-controlled preprocessing and out-of-fold predictions. Invalid evaluations and replay-incompatible episodes are excluded before pooled calibration metrics are computed.
- Five stratified outer folds produce exactly one out-of-fold prediction for every included trajectory.Within each outer-training partition, four-fold stratified cross-validation selects hyperparameters by Brier score.
- All methods use identical task populations and frozen fold assignments for each model and environment.Standardization, PCA, and probability calibration are fitted using training data only.
- Invalid correctness evaluations and replay-incompatible episodes are removed before fold construction and method fitting.The shared population therefore includes only trajectories with valid correctness evaluations and faithful offline replay when required.
- Pooled out-of-fold predictions are evaluated with AUROC, Brier score, and expected calibration error using 10 equal-width bins.
I Online Deployment Feasibility & Extraction Mechanics
The paper distinguishes offline replay used to extract reproducible representations during benchmarking from live deployment, where relevant activations are already available during standard generation. ARP can therefore be applied as a lightweight, non-blocking monitor before environment execution.
- Offline extraction: Offline experiments log residual-stream representations through a deterministic teacher-forced pass over recorded trajectories.This separates the reproducible benchmark testbed from real-world online deployment.
- Offline extraction: Single-pass offline replay provides bit-exact, reproducible representations without custom C++/CUDA patches to serving runtimes.Optimized serving systems typically discard intermediate activations after sampling, motivating offline extraction for benchmarking.
- Online deployment: Live deployment requires no secondary forward pass or teacher-forced replay because residual states are materialized during standard autoregressive action generation.The terminal action-token representation is resident before the unembedding projection.
- Online deployment: Fewer than 3.3 × 10^5 operations (< 0.05 ms on a standard GPU) are required to apply ARP when d = 5,120 and k = 64.The computation consists of PCA projection followed by logistic scalar evaluation.
- Online deployment: ARP probing can run as a non-blocking generation hook for instantaneous failure detection and human-in-the-loop intervention before environment execution.