PaperlayerView exact revision on arXiv

Source-linked AI summary

Last Step Matters: Early Uncertainty Cannot Predict Failure in Long-Horizon Agents

Zongyue Li, Chengyue Yu, Lei Zang, Chenyi Zhuang, Linjian Mo, Leilei Gan

arXiv:2608.29685v1cs.LG

TL;DR

The paper investigates whether uncertainty signals can predict failures during, rather than only at the end of, long-horizon agent execution. It evaluates mainstream signals on deep-research tasks and finds that final-step confidence is informative, while intermediate prediction remains weak because path switching disrupts the link to final outcomes. The results support using final-step confidence to guide post-completion retries.

  • Problem

    It remains unclear whether uncertainty metrics that detect failures at task completion can reliably predict failures during long-horizon execution, despite the costs of additional inference and tool-use steps.

  • Method

    The study evaluates verbal confidence, token-probability, and consistency-based signals across three deep-research benchmarks and five models throughout agent trajectories.

  • Results

    Final-step verbal confidence achieves mean AUROC 0.85 across 15 model-benchmark combinations, while no evaluated signal exceeds mean AUROC 0.60 at 50% progress.

  • Takeaways & Limitations

    Path switching helps explain weak intermediate prediction, and confidence-guided post-completion retries outperform in-trajectory intervention in the evaluated deep-research setting.

  • Takeaways & Limitations

    The evaluation primarily focuses on long-horizon deep-research tasks and mainstream uncertainty metrics, so broader task types and richer trajectory-aware methods remain future work.

Abstract

from arXiv · show

Early failure prediction is important for long-horizon agents, as it enables timely intervention and can reduce inference and tool-use costs. Uncertainty quantification, such as verbal confidence and perplexity, offers a promising approach to detecting agent failures; however, it has not been explored whether these signals retain their discriminative power during the intermediate stages of long-horizon execution. We evaluate mainstream uncertainty signals on deep-research tasks and find that verbal confidence reliably distinguishes failures at trajectory completion, achieving a mean AUROC of 0.85, whereas all evaluated signals offer limited predictive value earlier in execution, with none exceeding a mean AUROC of 0.60 at 50% trajectory progress. We identify an underlying mechanism explaining this gap: path switching, where agents frequently abandon their current search direction in-trajectory, breaking the link between early signal and final outcome. These findings challenge the assumption that intermediate uncertainty can reliably guide early intervention. They also motivate a practical recommendation for agent harnesses in deep-research settings: use final-step confidence to decide whether to restart, an approach that our experiments find more effective than in-trajectory intervention.

1 Introduction

The paper asks whether uncertainty signals can predict failures before long-horizon agents finish, then evaluates them across deep-research tasks. Final-step confidence is informative, but intermediate signals remain weak because agents switch search paths during execution.

  • Motivation: Long-horizon agents can fail through action loops, incorrect reasoning paths, or hallucinations, making early prediction valuable because each additional step incurs inference and tool-use costs.Early prediction could enable intervention before these costs accumulate.
  • Approach: The study evaluates verbal confidence, token-probability, and consistency-based signals across three deep-research benchmarks and five models ranging from 27B to 1T parameters.The evaluation addresses both completion-time failure detection and prediction during execution.
  • Results: 0.85 mean AUROC: final-step verbal confidence reliably distinguishes failures across 15 model-benchmark combinations.This is the strongest reported completion-time baseline.
  • Results: 0.60 mean AUROC: no evaluated signal exceeds this value at 50% trajectory progress, and all metrics remain weak until 80–90% progress.Predictive ability rises sharply only at trajectory completion.
  • Mechanism: Path switching weakens the link between intermediate confidence and final outcomes because agents frequently abandon their current search direction.After the final switch, verbal confidence becomes substantially more predictive.
  • Implication: The results favor post-completion retries guided by final-step confidence over interventions triggered by intermediate confidence.Confidence-guided post-completion retries outperform in-trajectory intervention in the experiments.

2 Theoretical Background

The paper defines uncertainty at both the action-step and trajectory levels. Step-level uncertainty is computed from the agent’s action distribution and then aggregated across the trajectory into a scalar.

  • Step-level uncertainty: At the step level, uncertainty describes the agent’s action distribution conditioned on the task input, prior environment state, and prior observation.The generic measure may be entropy, perplexity, or transformed verbal confidence.
  • Levels of analysis: The framework separates local uncertainty measurements from the trajectory-level summary used for agent-level analysis.This distinction supports comparing different aggregation choices across long-horizon execution.
  • Trajectory-level uncertainty: At the trajectory level, step-level uncertainty values are aggregated into a scalar for the generated trajectory.The aggregation operator may be the mean, last-step value, tail-weighted mean, or another operator.

3 Experimental Setup

The study evaluates uncertainty signals across deep-research benchmarks, open-weight models, trajectory aggregations, and AUROC-based backward-truncation analysis.

  • Datasets and Models: The evaluation covers BrowseComp, BrowseComp-zh, and HLE, using sampled benchmark subsets described in the experimental setup.BrowseComp has 1,266 questions, BrowseComp-zh uses 289 questions, and HLE uses 157 text-only questions from its 2,500-question benchmark.
  • Datasets and Models: The models span 27B to 1T parameters and are open-weight so token-level probability distributions can be computed.The open-weight restriction supports entropy and related token-probability metrics.
  • Uncertainty Metrics: The study evaluates verbal confidence, token-probability metrics, and consistency-based metrics as uncertainty-signal families.Verbal confidence is elicited on a 0–100 scale at each trajectory step; token metrics include perplexity, maximum token negative log-likelihood, Shannon entropy, and maximum token entropy.
  • Uncertainty Metrics: Six aggregation operators reduce each metric sequence to scalar features, producing 30 features from five base metrics.The operators are mean, running max, running min, last step, last-3 mean, and last-5 mean.
  • Evaluation Protocol: AUROC measures discrimination at each of 11 normalized progress points using only the trajectory prefix available at that point.Trajectories must contain at least 11 steps, and results are averaged across 15 model–benchmark combinations.

4 Discriminative Ability of Uncertainty Metrics

Final-step verbal confidence distinguishes failures effectively, but uncertainty signals remain weak during most of execution and are not improved by aggregation or feature combination.

  • Completion-Time Detection: 0.85 mean AUROC is achieved by final-step verbal confidence across the 15 model–benchmark combinations.This makes final-step verbal confidence a strong completion-time baseline.
  • Early Failure Prediction: At 50% progress, no uncertainty metric exceeds 0.60 mean AUROC, while all metrics remain below 0.70 even at 90%.Verbal confidence performs best throughout, but its mean AUROC rises sharply to 0.85 only at completion.
  • Temporal Aggregation: Running minimum improves mean AUROC from 0.60 to 0.62 at 50% progress, but only modestly.No temporal aggregation method outperforms final-step confidence at completion.
  • Feature Combination: None of the combined models outperforms verbal confidence alone at any trajectory progress point.The combinations use five uncertainty features with regularized logistic regression and LightGBM.

5 Path Switching and Poor Early Predictive Ability

Frequent path switching weakens the connection between early confidence and final correctness, while confidence becomes more informative after the final switch and near completion.

  • Path Switching: Preswitch confidence scores assess search directions later abandoned and therefore provide little indication of eventual success.In the example trajectory, confidence is 70% and 65% before the agent switches direction and later finds a productive lead.
  • Path Switching: 86.8%, 69.7%, and 36.9% of trajectories contain detected switches on BrowseComp, BC-zh, and HLE, respectively.The corresponding averages are 8.5, 4.7, and 1.3 switches per trajectory.
  • Predictive Ability Around Switches: Before the last detected switch, absolute Spearman correlation remains below 0.15; afterward, it rises monotonically.This pattern tests the prediction that switching weakens intermediate confidence–correctness relationships.
  • Post-Switch Confidence Surge: Correct trajectories show a 15-point confidence increase, whereas incorrect trajectories show a 6-point decrease from early-trajectory averages to the final step.The comparison uses final-step confidence minus mean confidence over the first 70% of each trajectory and appears across all 15 combinations.
  • Interpretation: Confidence reflects transient search paths early and aligns with final correctness only after the agent settles near completion.This provides the paper’s path-switching account of the temporal predictive gap.

6 Discussion

Sensitivity analyses show that weak early prediction persists across confidence-elicitation methods, while task environments with more persistent state changes retain more intermediate predictive information.

  • Confidence Elicitation: All three confidence-elicitation methods show weak early performance followed by a sharp increase toward completion.The comparison averages Qwen3.5-122B-A10B results across BrowseComp and BrowseComp-zh.
  • Calibration: At 50% progress, mean confidence is 77.05% versus a 30.77% actual success rate, with ECE 0.4629.At completion, mean confidence is 64.99% and ECE is 0.3422.
  • Calibration: Miscalibration does not change AUROC conclusions because AUROC depends on confidence ranking rather than absolute values.Monotonic recalibration that preserves ranking leaves AUROC unchanged, though fixed-threshold precision and recall can change.
  • Task-Type Sensitivity: The coding-task comparison is consistent with path dependence from persistent state changes and less frequent switching.The authors report alignment between reduced path switching on TerminalBench and higher intermediate AUROC.
  • Task-Type Sensitivity: Intermediate confidence is more predictive on coding-agent benchmarks than on deep-research benchmarks.Across two Qwen3.5 models on TerminalBench, intermediate AUROC ranges from 0.655 to 0.781 and exceeds 0.85 at completion.

7 Implications for Agent Harnesses

Intermediate uncertainty offers limited guidance for intervention during deep-research trajectories, while final-step confidence supports more effective restart decisions. Experiments find post-completion retries outperform in-trajectory intervention at comparable budgets.

  • Implications for Agent Harnesses: Intermediate uncertainty signals have poor predictive ability for final outcomes, making early intervention difficult to guide reliably.At 50% progress, both tested confidence thresholds provide limited failure-detection performance.
  • Failure Detection During Execution: At 50% progress, τ=0.8 yields precision 0.68 and recall 0.62, while τ=0.9 yields recall 0.96 and precision 0.64.The τ=0.9 precision is only slightly above the average failure rate of 0.62.
  • Failure Detection at Completion: At 100% progress, τ=0.8 achieves precision 0.95 and recall 0.46, while τ=0.9 achieves precision 0.88 and recall 0.76.These results indicate that confidence signals are primarily useful for guiding intervention at task completion.
  • Intervention Strategies: Post-completion intervention Pareto-dominates in-trajectory intervention across both evaluated models and all three benchmarks.With comparable step budgets, post-completion intervention achieves higher accuracy.
  • Recommendations: The recommended harness strategy is to use final-step confidence to identify failures and restart low-confidence trajectories with prior-answer awareness.Intermediate intervention may incur false-positive costs and disrupt path switching that could otherwise support recovery.

8 Related Work

Prior work studies trajectory uncertainty and uncertainty-guided intervention, but does not directly establish when intermediate signals become predictive during execution. This paper addresses that gap through systematic analysis in long-horizon deep-research tasks.

  • Related Work: Recent uncertainty research covers trajectory-level quantification and uncertainty-guided in-trajectory intervention or post-completion retries.Examples include SAUP, UProp, BrowseConf, ParallelMuse, and AUQ.
  • Uncertainty Quantification: SAUP aggregates step-level uncertainty with situation-dependent weights, while UProp separates current-decision uncertainty from uncertainty propagated by preceding decisions.Both incorporate intermediate uncertainty into trajectory-level estimates.
  • Open Gap: Prior methods implicitly assume intermediate steps contribute useful information to final prediction without directly measuring predictive ability throughout trajectories.Agentic Confidence Calibration reports that final-step token confidence can outperform full-trajectory averaging but does not evaluate early prediction during execution.
  • This Work: This work systematically analyzes whether and when uncertainty metrics become predictive of failure in long-horizon deep-research tasks.

9 Conclusion

Across evaluated deep-research benchmarks, final-step verbal confidence detects failures reliably, whereas uncertainty signals remain weak at intermediate stages. Path switching helps explain this gap, and post-completion confidence-guided retries outperform in-trajectory intervention.

  • Conclusion: Verbal confidence achieves mean AUROC 0.85 at task completion, while no evaluated metric exceeds mean AUROC 0.60 at 50% progress.Neither additional features nor feature combinations close this gap.
  • Conclusion: Path switching weakens the relationship between intermediate uncertainty and final outcomes, helping explain why discriminative signal strengthens near completion.
  • Conclusion: Confidence-guided post-completion retries outperform in-trajectory intervention in the reported intervention experiments.

Limitations

The evaluation emphasizes mainstream uncertainty metrics and long-horizon deep-research tasks, with additional coding-task comparison. Several metric families also require completed rollouts or answers, limiting their suitability for partial-trajectory prediction.

  • Scope: The evaluation primarily focuses on long-horizon deep-research tasks, with TerminalBench 2.0 providing an additional coding-task comparison.Broader validation across mathematical reasoning, coding, dialogue, shopping, and desktop control remains future work.
  • Metrics: The study evaluates verbal confidence, token-probability metrics, and consistency-based metrics as mainstream uncertainty-quantification approaches.
  • Confidence Elicitation: The three confidence elicitation methods differ in where and how confidence is obtained, yet exhibit the same qualitative trend.The methods include in-tool-call, in-content, and two-stage confidence elicitation.
  • Evaluation Pattern: Across evaluated settings, uncertainty metrics have limited intermediate predictive ability and become more informative near trajectory completion.
  • Task-Type Comparison: Path switching is less frequent in coding tasks than in deep-research tasks, consistent with stronger intermediate uncertainty prediction in coding.

F Controlling for Task Difficulty

Stratifying tasks by difficulty shows that path switching remains common and confidence becomes more predictive after the last switch within every difficulty group. Early predictive ability also varies across models, but the comparison does not establish a systematic relationship with scale or task performance.

  • Controlling for task difficulty: 95.0% of hard, 86.4% of medium, and 64.1% of easy trajectories contain at least one path switch.Path switching is more frequent on harder tasks.
  • Controlling for task difficulty: AUROC increases after the last switch from 0.490 to 0.742 for hard tasks, from 0.539 to 0.821 for medium tasks, and from 0.567 to 0.766 for easy tasks.The comparison is between confidence immediately before and five steps after the last switch.
  • Controlling for task difficulty: Within-group improvement indicates that task difficulty alone does not explain the relationship between path switching and predictive ability.
  • Model variation: Qwen3.5-27B reaches AUROC values of 0.533 at 20% progress and 0.583 at 50%, while Kimi K2.6 reaches 0.645 and 0.671 at those stages.These are the weakest and highest early-predictive-ability values among the compared models at the reported stages.
  • Model variation: The model comparison does not establish a systematic relationship between early predictive ability and model scale or task performance.Models also differ in architecture, activated parameter count, and training procedure.
  • Trajectory-aware monitoring: At 50% progress, the external monitor reaches 0.680 AUROC versus 0.560 for verbal confidence, but at completion verbal confidence reaches 0.933 versus 0.806.The monitor outperforms verbal confidence throughout the first 90% of progress, while completion reverses the comparison.
Loading 2608.29685v1…