Source-linked AI summary
How to Speculate about Uncertainty in Agentic Coding? A Draft-Model Gate Method
Konstantin Grotov, Valentin Malykh
TL;DR
LLM coding agents can act confidently wrong while their failures are discovered only after costly execution and retries. SU uses a small draft model to score output tokens from black-box agents, separates reasoning and action signals, and calibrates them for pre-execution decisions. Its veto gate reduces execution errors and token costs, but reported cost advantages and broader-domain transfer remain bounded by the studied code-execution setting.
Problem
Black-box coding agents lack exposed internal uncertainty signals, while repeated-sampling estimators are too costly and failures are often discovered only after execution and retry.
Method
SU inverts speculative decoding so a small open-weight draft scores an agent’s generated tokens, extracts phase-aware trajectory features, and calibrates them into a failure signal.
Results
6–8 percentage points lower execution error rate and 14–19% lower token cost were achieved with a pre-execution veto gate, with transfer across benchmarks and agent models.
Takeaways & Limitations
A separate draft model can provide an actionable uncertainty signal for closed-source agents without changing the agent or requiring internal access.
Takeaways & Limitations
Quantitative evidence is limited to code-execution success, and the favorable draft-cost balance may not hold when other objectives require larger or more capable drafts.
Abstract
from arXiv · showhide
LLM agents deployed for software engineering fail expensively: they act confidently wrong, and bad actions are recognized only after costly execution and retry. We present Speculative Uncertainty (SU), a method that recovers a predictive failure signal for a black-box agent from its output tokens alone, with no access to logits, weights, activations, or repeated sampling. Inverting speculative decoding, a small open-weight draft model scores the agent's already-generated trajectory in a single forward pass. From these speculative cross-likelihoods we extract phase-aware features by separating the reasoning and action spans, and calibrate them against a verifiable objective. SU produces a failure-likelihood score that any downstream policy, such as routing, human intervention, or extra test-time compute, can consume directly. To show the signal is actionable, we instantiate one such policy, a pre-execution veto gate, on software engineering agents Qwen3-Coder-480B and closed-source Claude 3.5 Sonnet, cutting execution error rate by 6-8 percentage points and token cost by 14-19% in deployment, transferring to out-of-distribution benchmarks without retraining, and generalizing across agent models.
1 Introduction
SU addresses costly, confidently wrong actions by recovering uncertainty from black-box agent output tokens alone and using it before execution. Its pre-execution gate reduces execution errors and token costs while transferring across benchmarks and agents.
- Production agents can incur execution, context, retry, and latency costs when confident failures are detected only after environmental rejection.
- Closed APIs hide logits, weights, and activations, while repeated-sampling estimators are too costly for long agentic trajectories.
- SU inverts speculative decoding: a small draft model scores already-generated agent tokens in one pass, then separates reasoning and action features for calibration.
- 6–8 percentage points lower per-call execution error rate and 14–19% lower average token cost were achieved by the pre-execution gate on two coding benchmarks.
- The method is usable without white-box access, and its gate transfers zero-shot across benchmarks and agent models.
2 Background and Related Work
Prior work studies calibration, trajectory uncertainty, and tool-call confidence, but often depends on internal model access or repeated generation. SU instead repurposes speculative decoding so a draft evaluates committed target tokens.
- Agentic uncertainty accumulates across interdependent steps, making single-output calibration insufficient for trajectory-level behavior.
- HTC extracts trajectory features from full log-probability sequences and calibrates them, whereas closed agents may not expose those internal probabilities.
- Probe-based methods anticipate erroneous tool calls from target-model internal states, requiring white-box access.
- Speculative decoding normally accelerates inference by pairing a draft with a target model, but SU reverses the role of the draft.
- In SU, the small model evaluates the target’s committed tokens after generation rather than generating candidate tokens.
3 Method
SU scores black-box agent trajectories with a small draft model, extracts phase-aware signals over recent reasoning and action spans, and calibrates them into a pre-execution decision score. A veto gate can execute confident actions or trigger replanning before environmental failure.
- 3.1 Problem Setup: An agent trajectory alternates reasoning and action spans, whose tokens are labeled by phase and evaluated against an external binary objective.
- 3.1 Problem Setup: SU predicts whether the next action passes its verifiable check using a fixed lookback window of the three most recent steps.
- 3.2 Speculative Cross-Likelihoods: Teacher-forced draft scoring produces speculative signals from the observed token stream in one forward pass, without requiring the agent’s logits.
- 3.2 Speculative Cross-Likelihoods: If the draft tracks the agent, high speculative surprisal indicates low speculative-decoding acceptance and serves as a proxy for the agent’s hidden uncertainty.
- 3.3 Phase-Aware Features: Reasoning and action tokens receive separate features because reasoning has higher entropy and can mask failure-relevant deviations in the action span.
- 3.4 Calibrator and Downstream Policy: An ℓ1-regularized logistic calibrator maps the feature vector to a sparse, auditable probability estimate and remains cheap in small-data settings.
- 3.4 Calibrator and Downstream Policy: The resulting failure-likelihood score can support routing, human review, or extra computation, while the evaluated veto gate blocks low-confidence actions and triggers cheaper replanning.
4 Experimental Results
SU predicts execution failures from agent output tokens and supports a pre-execution veto gate. Across coding benchmarks, it reduces execution errors and token costs, transfers across agents, but does not improve task success and remains miscalibrated as a probability.
- Calibration: SU’s score is useful for thresholded gating but is not a calibrated probability, so probability-consuming policies require post-hoc recalibration.Recalibration would preserve AUROC and veto-gate behavior while reducing ECE; the paper leaves this reliability gap unresolved.
- Deployment impact: 6–8 percentage points: the veto gate reduces per-call execution error from 21% to 15% on SWE-Bench Verified and from 14% to 6% on DA-Code.The DA-Code result uses a calibrator trained on SWE-rebench without additional fine-tuning.
- Deployment impact: 14–19%: the gate lowers average tokens per task on both benchmarks by replacing execute-fail-retry loops with cheaper replanning.The reported deployment benefit is reduced compute, API spend, and latency rather than higher benchmark success.
- Deployment impact: Task success does not improve: resolved rate changes from 49% to 44% on SWE-Bench Verified and from 64% to 63% on DA-Code.Replanning is not guaranteed to recover tasks, and false vetoes can block actions that would have succeeded.
- Cross-agent transfer: A Qwen-distilled draft transfers to Claude 3.5 Sonnet at .69 AUROC versus .60 for the untrained baseline, while SFT and TF distillation score .76 and .77 on Qwen.The transfer results support domain alignment as more important than the specific distillation objective.
5 Discussion and Practical Takeaways
SU offers a practical deployment recipe for gating risky agent actions: pair a frontier agent with a distilled draft, calibrate phase-aware features, and intervene before execution. The approach reduces errors and token costs while leaving streaming intervention and broader-domain validation for future work.
- Deployment guidance: A distilled 4B draft, phase-aware linear calibrator, and pre-execution veto gate provide a deployment recipe requiring one small forward pass per step.The pipeline makes no changes to the agent and was evaluated with labeled trajectories from the target agent or a mixed corpus.
- Deployment guidance: 6–8 pp lower error and 14–19% lower token cost are reported for the deployment recipe.These figures summarize the study’s gate-based deployment results.
- Transfer beyond code: SU requires reasoning and action spans plus an external binary oracle, but its transfer beyond software engineering remains unmeasured.The paper identifies SQL, web, and tool-orchestration agents as possible settings without making quantitative claims there.
- From observation to intervention: A streaming veto could monitor action-span entropy token by token and interrupt generation when divergence appears, but this extension is left for future work.The proposed intervention would act before the action block is complete rather than waiting for execution.
6 Conclusion
Speculative Uncertainty anticipates costly failures in production agents by scoring black-box trajectories with a small draft model and gating execution. Its deployment reduces execution errors and token costs, with the design motivated by separate-model scoring and reasoning–action phase structure.
- Conclusion: Production agents can fail confidently and expensively, paying for erroneous actions before their consequences reveal the error.The paper frames SU as an approach to anticipate failure rather than merely discover it after execution.
- Conclusion: SU uses a small open-weight draft and linear calibrator to turn a black-box agent trajectory into a failure-likelihood score that gates execution.The method relies on output tokens rather than the agent’s internal signals.
- Conclusion: 6–8 percentage points lower per-call execution error and 14–19% lower token cost were achieved, with zero-shot transfer and extension to a fully closed-source agent.The conclusion attributes the design’s leverage to separate-model scoring and respect for reasoning–action phase structure.
- Conclusion: The paper expects the approach to transfer to other verifiable-objective settings where agents reason before committing to actions.The stated conclusion is conditional on recurrence of that reasoning–action structure.
Limitations
The evidence is limited by untuned component choices, a single code-execution objective, single-run evaluation, and an objective-dependent assumption about draft-model cost. These boundaries constrain how broadly and confidently the reported results should be generalized.
- Quality of resulting models: SU does not identify an optimum across draft model, alignment, feature construction, and calibrator choices.The paper presents a lightweight framework and operating guidance rather than exhaustive component tuning.
- A single objective and domain: Quantitative evidence is confined to code-execution success, despite the framework’s stated agnosticism to objectives and domains.Additional domains and objectives require future validation.
- Single-run estimates: All reported numbers are single-run point estimates on fixed evaluation sets without seed variance or confidence intervals.The paper cautions that small AUROC differences and the 5 percentage points SWE-Bench Verified task-success change should be interpreted accordingly.
- Draft cost: The favorable cost balance assumes the draft’s per-step forward pass is effectively free relative to the agent and avoided retries.That assumption may fail when another objective requires a larger or more capable draft, so overhead must be reevaluated.
A The Phase Structure in Tool Augmented Agents
Agentic trajectories typically alternate reasoning and action spans whose token dynamics differ substantially. The paper’s phase-aware analysis uses this structure to avoid mixing opposing signals when calibrating failure predictions.
- Trajectory structure: An agentic trajectory is a sequence of emitted tokens in which each step typically contains a reasoning span followed by an executable action span.Reasoning covers planning or analysis, while the action span contains a tool call or code to execute.
- Empirical phase structure: Figure 2 normalizes token position from reasoning start through the reasoning–action transition to action end across 500 SWE-rebench OpenHands trajectories.The plotted set contains 250 successful and 250 failing actions.
- Empirical phase structure: Reasoning is highly exploratory and entropy collapses sharply as generation enters the action span.This phase difference motivates treating reasoning and action separately.
- Methodological implication: Pooling reasoning and action tokens can cancel their opposing failure signals, motivating separate phase features.The paper aggregates per-phase statistics over a short recent-step window to retain local trajectory context.
- Trajectory labeling: The implementation maps assistant free-form text to reasoning and structured tool calls, including serialized arguments, to action spans.These spans are parsed into component labels for trajectory processing.
B.2 Draft-Model Training
Draft training compares no training, realized-token SFT, and teacher-forced distillation while keeping inference strictly black-box. The calibrator uses standardized trajectory features with regularization selected on training data.
- Training regimes: Three draft-training regimes are compared for each draft size: no training, supervised fine-tuning, and teacher-forced distillation.No training uses the released checkpoint; SFT matches realized agent tokens, while teacher-forced distillation aligns full next-token distributions.
- Access constraints: Inference remains strictly black-box, using only the agent’s realized output tokens despite optional offline access to logits for Qwen3-Coder-480B draft training.Claude 3.5 Sonnet cannot be teacher-forced distilled because it exposes no logits, so its draft uses realized-token SFT.
- Training outcome: Teacher-forced distillation yields only marginal gains over realized-token SFT, while the training corpus matters more than the distillation objective.This comparison applies to the open-weight Qwen3-Coder-480B agent.
- Calibration: The calibrator is an ℓ1-regularized logistic regression over 50 standardized features, with statistics computed on the training split only.The feature vector is denoted ϕ(W_i) ∈ R50.
- Calibration: Regularization strength is selected by 5-fold cross-validation on SWE-rebench training data and frozen for all test evaluations, including zero-shot DA-Code transfer.The selection criterion is validation AUROC over the stated λ grid.
B.4 Compute and Latency
SU adds a single teacher-forced draft pass as its only critical-path overhead. For a 4B draft, this costs approximately 100 ms per step on one A100 GPU, much less than agent-model latency.
- Latency: ≈100 ms per step is the 4B draft’s deployment overhead on a single A100 GPU.The pass scores the trajectory-so-far without generation.
- Pipeline cost: Per-step draft latency is the only SU overhead on the agent’s critical path.The draft performs a single teacher-forced forward pass over the trajectory-so-far.
C Feature Taxonomy
The feature taxonomy combines reasoning/action separation with three speculative signals and eight distributional statistics. Span lengths complete the 50-dimensional representation, including boundary and trend measures for phase-specific changes.
- Feature construction: 48 statistical features arise from 2 component types, 3 speculative signals, and 8 aggregate statistics.The components are reasoning and action; the signals are surprisal, gap, and entropy.
- Feature statistics: The eight statistics capture mean, variance, maximum, minimum, skewness, trend, and first/last-10% boundary behavior.Trend is defined as Δ = μ_last − μ_first.
- Feature construction: The final two features are reasoning-span and action-span lengths, completing 2 × 3 × 8 + 2 = 50 features.Boundary and trend statistics let the linear calibrator represent late action-span entropy changes.