PaperlayerView exact revision on arXiv

Source-linked AI summary

From Uncertainty to Action: Learning to Steer LLM Agents

Hanwen Li, Jinhao Duan, Guanhua Zhu, Junchi Lu, Bo Shen, Chenxi Yuan, Kaidi Xu

arXiv:2610.09115v1cs.AIcs.CLcs.LG

TL;DR

Uncertainty estimates may help identify failing LLM-agent trajectories, but whether they can guide whether, where, and how to steer remains unclear. The paper measures counterfactual steering outcomes at every step and trains VoS to choose where to steer while a harm-budgeted trigger limits intervention on successful trajectories. VoS improves on unmodified execution across all 12 settings and outperforms the strongest prior uncertainty-triggered method in 11.

  • Problem

    Whether uncertainty can guide whether, where, and how to steer an agent remains unclear, although multi-step execution offers opportunities to correct errors before they propagate.

    Source: §1 · p. 1

  • Method

    The paper evaluates steering at every non-terminal step with four mechanisms to build SOT, then trains VoS to estimate steering value and uses a harm-budgeted trigger to decide whether to intervene.

    Source: §1 · pp. 2–3

  • Results

    7.8 points on average: VoS improves on unmodified execution in all 12 settings and beats the strongest of five prior uncertainty-triggered methods in 11 by 2.9 points on average.

    Source: §1 · p. 3

  • Takeaways & Limitations

    Measured steering outcomes provide a better training target than failure, binary, or process-reward labels, while no single uncertainty estimator suffices across settings.

    Source: §1 · p. 3, §6 · p. 10

    SOT is expensive to build, and token-distribution requirements limit the study to moderate-sized open-weight agents.

    Source: §6 · p. 10

Abstract

from arXiv · show

Steering an LLM agent means deciding whether to correct it, at which step, and with which mechanism. Uncertainty is often used to decide when to correct an agent, but whether it can guide these decisions remains unclear. We steer agent trajectories separately at every non-terminal step with each of four mechanisms and run each continuation to completion. The resulting stepwise outcome table (SOT) holds about 82,000 counterfactual continuations of 1,864 trajectories from three benchmarks and two agents. It shows that uncertainty can identify failing trajectories, but that no single signal reliably locates the step at which steering helps. We therefore propose VoS (Value of Steering), a trajectory-level monitor, offline or online, that learns from SOT the value of steering at each step and decides where to steer by it. A harm-budgeted trigger decides whether to steer, limiting the fraction of successful trajectories that VoS disturbs. VoS improves on unmodified execution in all 12 settings of benchmark, agent, and offline or online use, by 7.8 points on average, and outperforms the strongest of five existing uncertainty-triggered methods in 11, by 2.9 points on average. Ablations show that training on measured outcomes and a tight harm budget are both essential.

1 INTRODUCTION

Uncertainty can help identify failing agent trajectories, but it does not reliably show whether, where, or how steering will improve an outcome. The paper measures counterfactual steering outcomes at every step and uses them to train VoS, a monitor that selects where to steer and limits harm to successful trajectories.

1 INTRODUCTION

  • Effective steering requires choosing whether to intervene, at which step, and with which mechanism; predicted failure risk alone cannot determine whether a correction improves the outcome.A likely failure may not be recoverable, and steering a successful trajectory can introduce errors.Source: §1 · pp. 1–2
  • Uncertainty helps identify failing trajectories, but no single signal reliably locates helpful steps or identifies which mechanism to use.A signal’s failure-detection ranking does not necessarily match its ranking of within-trajectory steering gains.Source: §1 · p. 2
  • VoS uses a harm-budgeted trigger to decide whether to steer and a value function learned from measured outcomes to choose where to steer, offline or online.The trigger limits the fraction of successful trajectories that are steered.Source: §1 · pp. 2–3
  • About 82,000 counterfactual continuations across 1,864 trajectories form SOT, measuring recovery on failed trajectories and harm on successful ones.Every non-terminal step is tested with each of four steering mechanisms.Source: §1 · p. 2
  • Across three benchmarks with two agents, offline and online, VoS improves on unmodified execution in all 12 settings by 7.8 points on average, and outperforms the strongest of five prior uncertainty-triggered methods in 11 by 2.9 points on average.Ablations show measured outcomes are a better training target than failure, binary, or process-reward labels, and a tight harm budget beats steering more trajectories.Source: §1 · p. 3

2 STEPWISE COUNTERFACTUAL EVALUATION OF STEERING

The section defines stepwise counterfactual evaluation: replay each trajectory from a recorded prefix, apply a steering mechanism, and measure the resulting task-score change. It also specifies the policies, evaluation criteria, benchmarks, agents, and uncertainty estimators used in the study.

2.1 SETTING AND NOTATION

  • An agent’s reference trajectory is a sequence of reasoning, action, and observation steps; each step has token-distribution uncertainty scores, and the benchmark assigns the trajectory a task score and success or failure label.The step-level scores are oriented so higher values indicate greater uncertainty.Source: §2.1 · p. 3

2.2 STEERING MECHANISMS

  • Four mechanisms insert a message before the next action: self-reflection, critic feedback, stronger-model assistance, and memory-based guidance.Only stronger-model assistance calls an external model.Source: §2.2 · p. 3

2 STEPWISE COUNTERFACTUAL EVALUATION OF STEERING

  • The best failure detectors are not the best within-trajectory locators, and no estimator consistently orders steps by the outcomes of memory-based guidance.Figure 2 reports random-step score changes and compares failure detection with step localization for AppWorld and Gemma.Source: Figure 2 · p. 4

2.3 MEASURING STEERING OUTCOMES

  • SOT records the measured task-score change for every non-terminal step, trajectory, and mechanism by replaying the prefix and running the continuation to completion.Successful reference trajectories can only have zero or negative outcomes, so they measure potential harm from steering.Source: §2.3 · pp. 3–4

2.4 STEERING POLICIES AND EVALUATION

  • A policy steers each trajectory at most once; offline policies see the complete reference trajectory and roll back, while online policies decide from the current prefix.Evaluation counts scores across successful and failed trajectories and tracks the fraction of successful trajectories steered as the false-trigger rate.Source: §2.4 · p. 4

2.5 BENCHMARKS, AGENTS, AND UNCERTAINTY ESTIMATORS

  • The study uses AppWorld, ALFWorld, and WebShop with open-weight Gemma and Qwen agents, computing 14 uncertainty estimators from step token distributions.The estimators include probability-, entropy-, information-geometric-, mutual-information-, and self-evaluation-based signals.Source: §2.5 · pp. 4–5

3 WHAT UNCERTAINTY TELLS A STEERING POLICY

The section probes uncertainty as a signal for whether to steer, where to steer, and which mechanism to use. It finds trajectory-level failure information, but weak and mechanism-dependent guidance about helpful steps.

3.1 WHETHER: UNCERTAINTY IDENTIFIES FAILING TRAJECTORIES

  • Trajectory statistics detect failures (AUROC up to 0.73 Gemma, 0.71 Qwen), but random steering lowers successful-trajectory scores by up to 0.14 and 0.32, respectively.A monitor must control how often it steers successful trajectories.Source: §3.1 · p. 5

3.2 WHERE: THE SAME SIGNAL PLAYS DIFFERENT ROLES

  • Detection and localization rankings are unrelated or opposed (−0.12 Gemma; −0.41 Qwen); P(True) nevertheless has within-trajectory AUROC above 0.5 for every mechanism on both agents.P(True) is among Gemma’s weakest failure detectors, and its detection direction is reversed on Qwen (0.44).Source: §3.2 · p. 5
  • For memory guidance, even P(True)’s median per-trajectory rank correlation with measured gain is only +0.15 on both agents, and every estimator is negative on at least a quarter of trajectories.P(True)’s top-ranked step is still the best step more often than a random step in all eight agent-mechanism combinations.Source: §3.2 · p. 5

3.3 WHICH: MECHANISMS DISAGREE ON WHERE TO STEER

  • Mechanisms agree more on trajectory recoverability (0.17 to 0.66) than step ordering (0.02 to 0.19); token entropy has opposite outcome associations across two Gemma mechanisms.The reversal occurs for memory-based guidance versus stronger-model assistance.Source: §3.3 · pp. 5–6

4 STEERING MONITORS

VoS separates where to steer, using a learned estimate of steering value, from whether to steer, using a calibrated harm budget. Across the reported comparisons, it leads most settings, while the useful uncertainty signals vary and repeated steering adds modestly.

4.1 STEERING VALUE AND HARM-BUDGETED TRIGGERING

  • The value function estimates each mechanism’s expected effect on final task score; offline VoS selects the highest-value step above threshold, while online VoS acts at the first qualifying step.It is fit by regression on measured outcomes at every step and mechanism, using features that distinguish trajectories and steps; separate functions are fitted per mechanism.Source: §4.1 · pp. 6–7
  • A harm budget α aims to recover failed trajectories while steering at most an α fraction of successful training trajectories, but this calibration is not guaranteed on test trajectories.The threshold is calibrated from the quantile of successful trajectories’ largest out-of-fold value estimates; α is selected on training trajectories.Source: §4.1 · p. 7

4.2 MAIN RESULTS

  • VoS has the highest score in 11 of 12 settings, and raising its budget to 100% lowers its gain in nine settings.Always steering helps Gemma on AppWorld but costs Qwen 9.9 points; VoS gains more on Gemma and does not lose in any setting.Source: §4.2 · p. 7

4.3 WHAT THE MONITOR RELIES ON

  • Across the twelve settings, the all-signal monitor is never more than 1.2 points below the best, while no single signal family is reliable across settings.Self-evaluation is the best single family in nine settings but not the other three; token probabilities trail the best monitor by 2.7 points on average.Source: §4.3 · p. 8, Figure 4 · p. 8
  • Failure labels, binary steering-help labels, hindsight step correctness, and value-of-continuing targets do not match measured advantage; failure labels recover least and step correctness can select the wrong step.The first judged error lies in the last third of recoverable trajectories, while the most helpful steps are in the middle.Source: §4.3 · p. 8, Table 2 · p. 8

4.4 WHAT STEERING COSTS

  • On AppWorld with Gemma, VoS stays above prior methods across the plotted cost range and, at less than a third of their cost, exceeds their best score at any budget.The comparison plots task goal completion against cost for budgets from 5% to 30%, with unmodified execution as the reference.Source: §4.4 · p. 8, Figure 5 · p. 8
  • In real AppWorld runs, one steering event yields at least 70% of the best gain of every method that gains, and VoS leads at every event count.The figure compares task goal completion against the number of allowed steering events, with unmodified execution as the reference.Source: §4.4 · p. 9, Figure 6 · p. 9

5 RELATED WORK

Prior work studies uncertainty estimation, steering mechanisms, and intervention timing, but existing counterfactual comparisons cover only triggered states. This paper positions stepwise re-execution as a way to compare intervention outcomes across a trajectory.

5 RELATED WORK

  • LLM uncertainty estimators use token probabilities, sampled-generation consistency, internal representations, or model confidence, and are often evaluated by error detection; agent methods also trigger reflection, tool calls, deferral, or help.This work evaluates 14 estimators from four families by their contribution to decisions about whether, where, and how to steer.Source: §5 · p. 9
  • Steering work includes self-reflection, tool-based critique, and past experience, while alternative responses include deferring queries to stronger models or withholding predictions.Self-correction without reliable feedback can degrade results.Source: §5 · p. 9
  • DIAL and Zhang et al. compare counterfactual outcomes only at trigger-selected states, whereas SOT measures outcomes at every step by re-executing the agent.Process reward models and failure attribution locate errors, which need not be the steps where steering helps most.Source: §5 · p. 9

6 CONCLUSION

The conclusion finds that uncertainty identifies failing trajectories but does not reliably locate helpful steering steps; VoS learns stepwise steering value from measured outcomes. The approach requires counterfactual outcomes for each benchmark, agent, and mechanism.

6 CONCLUSION

  • VoS improves on unmodified execution in all 12 settings by learning a value function over signals from SOT and pairing it with a harm-budgeted trigger.The trigger limits steering of successful trajectories; the learned value function helps decide where to steer.Source: §6 · p. 10
  • Current uncertainty estimators identify failing trajectories but do not reliably locate where steering helps or which mechanism to use.The conclusion calls for estimators aimed at the value of steering at a step, rather than output correctness, to reduce counterfactual supervision needs.Source: §6 · p. 10
  • Building SOT is expensive because it reruns agents from every step with every mechanism, and the estimators require token distributions.The study therefore covers only moderate-size open-weight agents; black-box verbalized confidence is much less effective.Source: §6 · p. 10

A EXPERIMENTAL SETUP

The setup covers benchmark and agent settings and describes separate training and evaluation splits for AppWorld, ALFWorld, and WebShop.

A EXPERIMENTAL SETUP

  • Table 3 lists the benchmarks and agent settings covered by the experiments.The supplied passage does not specify the individual agent settings.Source: A EXPERIMENTAL SETUP · p. 17, Table 3 · p. 17
  • AppWorld and ALFWorld use ten random, failure-stratified splits of evaluated tasks into training and evaluation halves.The table covers evaluated tasks only.Source: A EXPERIMENTAL SETUP · p. 17
  • WebShop policies train on 300 training goals and evaluate on 100 held-out goals, with deviations over ten draws of 250 training goals.Source: A EXPERIMENTAL SETUP · p. 17
  • Memory notes are synthesized from failed reference trajectories in each benchmark’s training split, never from evaluated tasks.For AppWorld and ALFWorld, these tasks have no outcomes and are disjoint from both halves.Source: A EXPERIMENTAL SETUP · p. 17

B STEERING MECHANISMS

The four steering mechanisms are presented through AppWorld prompt examples, including the prompts that produce guidance and the messages agents receive.

B STEERING MECHANISMS

  • The reflection mechanism asks the agent to distinguish confirmed requirements from assumptions or gaps, then act on the first gap before coding.The prompt also requires a brief reflection before code at every subsequent turn.Source: B STEERING MECHANISMS · p. 17
  • The critic checks a trajectory against live app state in a read-only environment and, if incorrect, provides a corrected next step.Source: B STEERING MECHANISMS · p. 17
  • Stronger-model assistance asks a second model call to turn reasoning into code, then instructs the agent to execute the deferred action exactly.Source: B STEERING MECHANISMS · p. 18
  • The advisor checks an agent’s recent steps against failure patterns and supplies a diagnosis, next action, and up to two further actions.The agent is instructed to follow the guidance as its next action.Source: B STEERING MECHANISMS · p. 18

C DETAILS OF THE MONITOR

The monitor combines uncertainty signals into features, learns steering value from measured outcomes, and uses a calibrated threshold to select whether and where to steer.

C DETAILS OF THE MONITOR

  • The monitor uses 14 estimators across probability, entropy, pointwise mutual information, and self-evaluation families.Missing step values are replaced with the estimator’s mean over that trajectory.Source: C DETAILS OF THE MONITOR · p. 18
  • Offline features describe signals relative to the full trajectory, while online features describe them relative to the observed prefix.Features include signal values and changes; offline features also include within-trajectory ranks and summary statistics, while online features include running statistics and percentiles.Source: C DETAILS OF THE MONITOR · p. 18
  • The offline value function uses gradient-boosted regression trees, while the online version uses ridge regression.Hyperparameters were chosen on AppWorld and then fixed across splits, agents, and benchmarks; ALFWorld and WebShop were not tuned.Source: C DETAILS OF THE MONITOR · pp. 18–19, Table 4 · p. 19
  • The harm-budget threshold is calibrated from successful training trajectories’ maximum out-of-fold value estimates, with the budget selected by training task score.Offline steering uses the highest-estimate step if it reaches the threshold; online steering uses the first qualifying step from t = 3 onward.Source: C DETAILS OF THE MONITOR · p. 19
  • A separate value function for each mechanism is trained on failed and successful trajectories using the measured steering outcome as its target.Each step is weighted by the inverse trajectory length so short and long trajectories contribute equally in total to the loss.Source: C DETAILS OF THE MONITOR · p. 19

D METHODS FROM PRIOR WORK

The prior-work baselines retain their original triggers or predictors where possible, while steering mechanisms are adapted when the originals do not fit these single-agent runs. DIAL’s original optimizer is also replaced because it uses environment scoring unavailable to the other methods.

Calibration

AUQ

  • AUQ retains its low-confidence reflection trigger, but omits its uncertainty-aware memory and consistency-weighted best-of-N reflection.The memory doubled output tokens in these runs and did not prevent any task from getting stuck in a loop; the authors say it acted as a prompt to think one step further.Source: D METHODS FROM PRIOR WORK · p. 19

DIAL

  • DIAL retains its ℓ1-regularized logistic gate trained to predict whether steering helps, but uses memory-based guidance instead of its environment-scoring optimizer.The original optimizer would require hidden unit-test scoring on AppWorld, information unavailable to the other methods.Source: D METHODS FROM PRIOR WORK · p. 19

Barbi et al.

DS-MCM

  • DS-MCM retains its trigger on deviations in reasoning entropy from successful-step levels and uses memory-based guidance to steer.Source: D METHODS FROM PRIOR WORK · p. 19

ReDAct

E ADDITIONAL RESULTS

The additional results compare policies under matched steering mechanisms, compare mechanism-specific monitors with a routed monitor, and report mechanism and signal costs. A further table subtracts gains attributable to bare rewinding.

Appendix map

  • The appendix tables cover memory-based policy comparisons, mechanism-monitor comparisons, and mechanism-call costs used in the main-text analyses.Source: E ADDITIONAL RESULTS · p. 20

Matched mechanisms

  • Table 5 compares policies when all use memory-based guidance, so differences reflect when and where they steer rather than which mechanism they use.AUQ and ReDAct use memory-based guidance instead of reflection and a stronger model, respectively.Source: Table 5 · p. 20

Monitor comparison

  • Table 6 reports improvement over unmodified execution for Gemma monitors built on individual mechanisms and a routed monitor choosing the highest estimated-value mechanism at each step.Source: Table 6 · p. 20

Mechanism costs

  • Table 7 reports AppWorld mechanism-call costs per steering event in tokens, calls, agent-step equivalents, and share of total event cost.Source: Table 7 · p. 20

Resampling control

  • Table 8 subtracts the change from a bare rewind at the same step, crediting policies only for what their steering message adds.A gray reference monitor is trained on rewinds alone to learn where rerunning helps.Source: Table 8 · p. 21

F ADDITIONAL FIGURE DETAILS

The additional figures and table provide detailed comparisons using the same outcomes and task splits, including token costs and agreement between mechanisms. They also show Qwen counterparts to earlier figures.

Additional figures

Figure 7

  • Figure 7 counts steering cost in tokens from the steered continuation and mechanism call, minus the online tokens of the replaced remainder.Source: Figure 7 · p. 21

Table 9

  • Table 9 reports uncertainty-signal token costs per agent step on AppWorld, distinguishing rereading context from cached context.It also notes that probabilities and entropies come from the pass that generated the action.Source: Table 9 · p. 22

Figure 8

Figure 9

  • Figure 9 compares mechanism agreement on failed trajectories for Gemma and Qwen, with best-outcome correlations across trajectories below the diagonal and stepwise correlations above.The above-diagonal correlations are averaged over trajectories.Source: Figure 9 · p. 22
Loading 2610.09115v1…