PaperlayerView exact revision on arXiv

Source-linked AI summary

Can an Agent Learn to Predict Its Own Failure? Outcome-Grounded Confidence Calibration in LLM Agents

Ashutosh Hathidara

arXiv:2610.18614v1cs.AIcs.LG

TL;DR

Long-horizon agents need confidence about whether an episode will ultimately succeed, but existing RL calibration targets current-answer or step correctness. This paper uses GRPO rollouts to score each step’s verbalized confidence against episode outcomes. The resulting signal predicts outcomes across environments and regimes, with stronger evidence for distinguishing tasks than individual attempts.

  • Problem

    Existing RL calibration targets single answers or local step correctness rather than predicting whether a long-horizon agent rollout will ultimately succeed.

    Source: §1 · p. 1

  • Method

    The method uses GRPO task rollouts as outcome labels and scores each step’s verbalized confidence against its realized episode outcome with a proper scoring rule.

    Source: §1 · p. 1, §4.1 · p. 3, §4.2 · p. 4

  • Results

    Confidence becomes predictive across environments and training regimes; online, the top 10% of early steps succeed at 0.56–0.66 versus 0.49 untrained, with ECE 0.05–0.12.

    Source: §1 · p. 1, §7.4 · p. 8

  • Takeaways & Limitations

    The strongest live-agent gain is between tasks: confidence helps identify which tasks the agent will fail, while broader attempt-level foresight remains unresolved.

    Source: §8 · p. 8, Limitations · p. 9

    Live WebShop results are inconclusive because held-out success is low for both arms, and broader attempt-level foresight and additional model families remain open.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Controlling an acting LLM agent means deciding, at each step of a long-horizon task, whether to trust its next action, abstain, or hand off to a human. That decision needs a per-step confidence signal predicting whether the current rollout will eventually succeed. The agent's own self-reported verbalized confidence is the natural candidate, but existing reinforcement learning methods calibrate it against the correctness of a decision at the current step. We propose a method to calibrate it against the eventual outcome of a multi-step episode: under GRPO, the group of rollouts for a task supplies success labels, and a proper scoring rule scores each step's confidence against the realized outcome. In offline RL on recorded trajectories of a Qwen2.5-7B agent in ALFWorld, TextWorld, and WebShop environments, the AUROC of step confidence against episode outcome rises from near chance to 0.68-0.85, and the top-10% of steps the agent is most confident about succeed ~4.3x more often than average. In online RL on the live ALFWorld agent, four calibration rewards improve early-step AUROC, expected calibration error goes from 0.42 to below 0.12, and selective execution improves over both the untrained agent and a format-only RL control, without degrading task success. We also introduce an evaluation protocol that separates episode depth, task difficulty, and attempt-level foresight, which shows where the gain lies: the trained agent recognizes from its first step which tasks it will fail, while telling a succeeding attempt from a failing one at the same task is harder, with some early-window signal emerging on TextWorld.

1 Introduction

Long-horizon agents need confidence about eventual episode success to guide runtime decisions, but existing calibration targets local correctness. The paper trains step-wise confidence against episode outcomes and reports that this signal becomes predictive across environments and training regimes.

1 Introduction

  • Long-horizon agents need a per-step estimate of eventual success to decide whether to act, verify, or escalate to a human.Source: §1 · p. 1
  • Existing confidence training targets single-answer or step-level correctness, not whether a long-horizon rollout will ultimately succeed.Source: §1 · p. 1
  • The paper trains step-wise confidence against eventual task outcomes using task rollouts as labels and a proper scoring rule under GRPO.Source: §1 · p. 1
  • Outcome-grounded calibration makes a weak agent’s confidence predictive of eventual outcomes across environments and training regimes.Source: §1 · p. 1
  • The stated contributions are outcome-grounded GRPO training, an episodic-confidence evaluation protocol, and an empirical analysis of what training teaches the agent.Source: §1 · pp. 1–2

2 Related Work

Prior confidence work primarily evaluates synchronous answer correctness, whereas this paper targets eventual agent success over long-horizon tasks. Related approaches differ in confidence signals and reward designs, while selective prediction supplies the governance framing.

Measuring and eliciting confidence

  • Confidence elicitation methods commonly target synchronous single-answer correctness, including verbalized probabilities, self-evaluation, and black-box elicitation.Source: §2 · p. 2

Reinforcement learning for confidence

  • Prior calibration RL optimizes synchronous outputs; the paper instead targets eventual success across a long-horizon task.Source: §2 · p. 2

Positioning against concurrent agentic methods

  • Concurrent agentic methods calibrate current-answer or step correctness, while this work trains explicit verbalized confidence against eventual rollout success.Source: §2 · p. 2

Selective prediction

  • The paper frames confidence-based action gating as selective prediction, evaluated with risk-coverage curves and trained using proper scoring rules.Source: §2 · p. 2

3 Problem Setup and Evaluation Protocol

The protocol defines confidence as predicting eventual episode success from the history so far, then separates pooled ranking from task-difficulty, attempt-foresight, and depth effects. It also specifies calibration and selective-execution metrics for evaluating the signal.

3 Problem Setup and Evaluation Protocol

  • The protocol separates pooled confidence evaluation from estimands that isolate task difficulty, attempt-level foresight, and episode depth.Source: §3 · p. 2

3.1 Predictive Confidence Target

  • Confidence is the probability that the episode ultimately succeeds given the observed history, rather than the probability that the current step is correct.Source: §3.1 · p. 2

3.2 Estimands

  • Four AUROC estimands compare steps across different sets: all steps, an early window, step 0, and between-task or within-task comparisons.Source: §3.2 · pp. 2–3

3.3 Calibration & Selective Execution

  • Calibration uses ECE and Brier score, while selective execution measures success among high-confidence steps or tasks.Source: §3.3 · p. 3

4 Methodology

The method uses GRPO rollouts to assign outcome labels to every step, then combines outcome-relative and confidence-reward signals. A proper scoring rule supplies the calibration reward, with optional confidence-spread regularization and offline or online training routes.

4 Methodology

  • Each task rollout’s outcome labels every step, allowing the method to score confidence against eventual success and incorporate it into GRPO training.Source: §4 · p. 3, Figure 1 · p. 3

4.1 Group-Relative Training with Outcome

  • GRPO advantages use group-relative outcomes and per-step rewards, with the group’s success rate and reward mean defining the comparison.Source: §4.1 · p. 3

4.2 Per-Step Reward

  • The per-step reward combines calibration, confidence spread, and format terms; the default calibration reward is the Brier score, while spread regularization favors interior confidence values.Source: §4.2 · p. 4

4.3 Training Strategies

  • Offline training uses recorded untrained-agent trajectories, shared outcome labels, and an action-match reward; the policy does not act on its own outputs.Source: §4.3 · p. 4
  • Online training updates the live agent on its own rollouts, sampling six rollouts per task across 120 iterations.Source: §4.3 · p. 4

5 Experimental Setup

Experiments use Qwen2.5 agents across ALFWorld, TextWorld, and WebShop, with offline and online calibration comparisons. The setup specifies training data, reward variants, evaluation choices, and baselines.

Experimental Setup

  • Experiments use Qwen2.5-7B-Instruct across ALFWorld, TextWorld, and WebShop, with a Qwen2.5-14B-Instruct model also tested.Models use LoRA at rank 32 and α = 64; prompts request an admissible action and confidence that the episode will succeed.Source: §5 · p. 4
  • Training data comprise 1,068 ALFWorld rows from 200 games, 943 TextWorld rows from 200 games, and 1,066 WebShop rows from 372 games.Each environment is trained independently for 150 steps.Source: §5 · p. 4
  • The format-only baseline is compared with four calibration variants: Brier+φ, balanced Brier, Brier+dispersion, and balanced+dispersion.Each calibration experiment includes a spread or dispersion term at wφ = 0.10.Source: §5 · p. 4
  • Table 1 compares base instruction-tuned and RL-trained Qwen2.5-7B-Instruct models on held-out single-trajectory recordings using pooled step-level AUROC and top-10% confident-sample success lift over base success; Table 2 compares WebShop calibration reward forms using pooled AUROC.Table 2’s “Format” measure is the fraction of samples that correctly predict the tagged confidence tokens.Source: Table 1 · p. 5, Table 2 · p. 5
  • The evaluation fixes within-task AUROC at k ≤2 as the estimand and uses validation argmax over checkpoints {40, 80, 120} for three specified arms.The checkpoint rule applies to the balanced, dispersion, and balanced+dispersion arms.Source: §5 · p. 5
  • Comparisons include untrained and format-only models, offline entropy and sequence log-probability signals, and depth-based prediction baselines.The depth-only and depth-plus-length baselines capture predictions from episode position and output length alone.Source: §5 · p. 5

6 Results

Outcome-grounded calibration improves confidence as a predictor of eventual success in offline and online training. Online ALFWorld results show better early-step discrimination and selective execution without reduced task success, while analysis of the signal identifies where those gains occur.

6.1 Offline RL Training

  • Offline pooled AUROC rises from 0.517, 0.578, and 0.462 to 0.681, 0.722, and 0.850; the most-confident decile succeeds about 4.3× the base rate.The offline samples are single recorded trajectories, so these pooled estimates cannot separate confidence from task difficulty or episode depth.Source: §6.1 · p. 5
  • After offline calibration, verbalized confidence exceeds token entropy in every environment, while depth and output length alone match or exceed confidence AUROC on ALFWorld and TextWorld.For TextWorld and WebShop, verbalized confidence versus token entropy is 0.722 vs. 0.399 and 0.850 vs. 0.430, respectively; residualized confidence retains AUROC 0.626, 0.677, and 0.811.Source: §6.1 · pp. 5–6
  • Six calibration reward forms yield overlapping AUROC of 0.820–0.882, while task reward alone reaches 0.559 against 0.502 untrained; estimator choice also changes results.Under matched ranking-plus-spread reward, RLOO attains 0.801 and 0.876 versus GRPO’s 0.613 and 0.676; PPO fails to learn parsable response format across two seeds.Source: §6.1 · p. 6

6.2 Online RL Training

  • Among early steps, top-10% success rises from 0.493 untrained to 0.56–0.66 across calibration arms, while task success holds at 0.189–0.203 versus 0.201 untrained.The format-only control moves neither early top-10% nor top-25% success; top-25% success rises from 0.363 to 0.415–0.446 across calibration arms.Source: §6.2 · pp. 6–7, Table 4 · p. 7
  • Live ALFWorld pooled early-window AUROC rises by +0.072 (Brier+φ), +0.076 (balanced Brier), +0.101 (Brier+dispersion), and +0.048 (balanced+dispersion).The format-only control gains +0.019 [-.010,.051].Source: §6.2 · p. 6, Table 4 · p. 7
  • In live TextWorld, episode success rises from 0.146 to 0.262, with within-task AUROC gains of +0.265 at step 0 and +0.070 over steps 0–2; 14B AUROC rises from 0.697 to 0.729.The TextWorld within-task gains decay to zero over the full episode; the 14B evaluation is described as minimal and fluctuating.Source: §6.2 · p. 7

7 Analysis of the Trained Signal

The online gains primarily improve between-task ranking: training brings forward recognition of which tasks will succeed or fail, rather than reliably distinguishing attempts within the same task. Depth and white-box baselines clarify the scope of that signal and its use for triage.

7.1 Decomposition

  • Between-task early-window AUROC rises from 0.825 to 0.848–0.897, while within-task AUROC shows no significant gain and no experiment meets the pre-registered within-task criterion.Only Brier+dispersion’s between-task gain, +0.072 [.029,.115], is significant; all task-level AUROC delta intervals include zero.Source: §7.1 · p. 8

7.2 Foresight at Step 0

  • At step 0, pooled AUROC rises from 0.562 to 0.642–0.700, but within-task AUROC stays at chance, indicating ranking of tasks rather than grading the first action.Confidence is emitted after the sampled first action but before environment feedback; the untrained model emits exactly 0.50 at step 0 under greedy decoding on a 64-task probe.Source: §7.2 · p. 8

7.3 Depth and White-Box Baselines

  • The depth-only baseline reaches 0.499–0.500 AUROC in the early window, despite matching or beating trained experiments on pooled all-step AUROC.The early-window and depth-matched estimands are therefore essential for evaluating a controller; token entropy is the strongest offline competitor, while online experiments lack a white-box competitor.Source: §7.3 · p. 8

7.4 Confidence as a Triage Signal

  • Early-step selective execution succeeds at 0.56–0.66 for the most-confident 10%, versus 0.49 untrained and a 0.20 base rate, with ECE 0.05–0.12.The highest-confidence steps do not reliably have higher success, and the most-confident bin is overconfident; the paper recommends relying on a band of high confidence rather than only the very highest values.Source: §7.4 · p. 8

8 Conclusion

The paper trains language agents to calibrate verbalized confidence against eventual task outcomes under GRPO. On a live multi-turn agent, this provides a usable early-triage signal without hurting task success; the gain is strongest between tasks, not between attempts.

8 Conclusion

  • Training rewards verbalized confidence against eventual task outcomes under GRPO.The conclusion frames this as a method for calibrating a language agent’s confidence.Source: §8 · p. 8
  • On a live multi-turn agent, confidence becomes a usable early-triage signal without hurting task success.The decomposition finds a between-task gain: the agent recognizes earlier which tasks it will fail, rather than which attempts will fail.Source: §8 · p. 8

Limitations

The strongest evidence is for a between-task effect in live ALFWorld; broader attempt-level foresight remains unsettled.

Limitations

  • Attempt-level foresight is not established broadly: within-task estimands move only in TextWorld’s early window, not yet on ALFWorld.The positive live multi-turn results cover ALFWorld and TextWorld at 7B; live WebShop results are inconclusive because held-out success is low for both arms and too few tasks are informative.Source: Limitations · p. 9

Ethical considerations

Online calibration experiments report positive early-window AUROC contrasts against a format-only control, while the appendix defines estimands that distinguish task difficulty, attempt quality, and depth. The section also reports the untrained baseline and an ethical-compliance statement.

Ethical considerations

  • The experiments were conducted under the ACL Ethics Policy and relevant research-integrity guidelines; the authors report no known unaddressed ethical risks.Source: Ethical considerations · p. 9

A Estimand Definitions

  • The estimands use the same AUROC statistic but vary comparison sets and score transforms to separate task difficulty, attempt quality, and depth.Pooled AUROC leaves these factors uncontrolled; between-task AUROC retains task-difficulty signal, while within-task comparisons cancel task difficulty.Source: A Estimand Definitions · pp. 11–12

B Untrained Estimand Ladder

  • Within-task early-window AUROC for the untrained agent is 0.677 [.628,.721], versus 0.763 [.739,.786] over all steps on the 45-task training pool.The appendix notes that only tasks containing both outcomes are estimable, leaving wide intervals when the window narrows.Source: A Estimand Definitions · p. 12, B Untrained Estimand Ladder · p. 13

C Detailed Results on Online RL Training

  • The depth-only baseline’s within-task all-step AUROC is non-monotone across checkpoints: 0.651 untrained, 0.677 at 40, 0.745 at 80, and 0.673 at 120.Source: C Detailed Results on Online RL Training · p. 13
  • Brier+dispersion exceeds format-only by +0.127 [.062,.189] in within-task early-window AUROC (5 tasks) and +0.082 [.050,.114] in pooled early-window AUROC.Brier+φ exceeds format-only by +0.053 [.016,.090] in pooled early-window AUROC.Source: C Detailed Results on Online RL Training · p. 13

D Detailed results on Other Environments

Live TextWorld results show early-window within-task gains, but their source is not isolated; live WebShop evidence is inconclusive. At 14B, ALFWorld pooled AUROC is directionally higher after training, while task-level triage changes remain uncertain.

D Detailed results on Other Environments

  • In live TextWorld, within-task AUROC changes by +0.017 [-.050,.082] by five steps and −0.029 [-.087,.029] over all steps; all-step AUROC changes from 0.697 to 0.668.The early-window transfer gain is reported without attributing it to de-censoring or calibration in isolation.Source: D Detailed results on Other Environments · pp. 14–15, Table 9 · p. 14
  • On live WebShop, held-out success is 0.9% untrained and 1.6% trained, leaving too little variation to identify a within-task early-window gain meaningfully.Pooled discrimination is 0.825 [.785,.889] untrained and 0.877 [.784,.951] trained, with overlapping intervals.Source: D Detailed results on Other Environments · p. 15

E Across Model Size

  • At 14B on ALFWorld, pooled matched-emission AUROC is 0.697 [.615,.781] untrained and 0.729 [.635,.818] trained, with a paired delta of +0.032 [-.016,.073].The within-task early-window estimand is not identifiable reliably with only six untrained and seven trained held-out tasks containing both outcomes.Source: E Across Model Size · p. 15

F Calibration and Selective Execution

  • Task-level triage success is 0.362 untrained and 0.349–0.354 for the listed trained arms; task-level triage and AUROC deltas all have confidence intervals including 0.Full-coverage step-level success is 0.186–0.203 in the early window (k ≤2).Source: F Calibration and Selective Execution · p. 15

G Details on Offline RL Training

The appendix reports additional offline results and ablations, then documents reward configurations, compute, reproducibility, and artifact-use details.

G Details on Offline RL Training

  • ALFWorld AUROC rose from 0.517 to 0.681, TextWorld from 0.578 to 0.722, and WebShop from 0.462 to 0.850; top-10% success lifts reached 4.33×, 4.35×, and 4.35×.Source: G Details on Offline RL Training · p. 15
  • Depth and content-token count alone yielded AUROC 0.769/0.849/0.858 on untrained rows and 0.787/0.855/0.799 on trained rows across ALFWorld, TextWorld, and WebShop.Source: G Details on Offline RL Training · p. 15
  • On a separate held-out split, WebShop ranking plus spread reached AUROC 0.795 and 5.0× top-10% success, while TextWorld ranking reached 0.775 and 5.4×; these values are not comparable to Table 1.Source: G Details on Offline RL Training · p. 16

H Ablations

  • WebShop Brier AUROC plateaued near 0.86 by a learning rate of 5 × 10^-6; greedy confidence AUROC was 0.863 versus 0.846 at T = 0.8, and spread weight 0.10 raised high-confidence-bin success from 0.33 to 0.37.Source: H Ablations · p. 16

I Hyperparameters, Prompt & Rewards

  • Across seeds, Brier (0.823–0.864) and spherical (0.852–0.882) reward forms overlapped; task-reward-only runs scored 0.559 and 0.555, while three PPO runs scored 0.500.The PPO runs were associated with a cold-start value head and collapsing confidence-tag emission.Source: I Hyperparameters, Prompt & Rewards · p. 18

J Computational Budget and Reproducibility

  • The study reports approximately 200 GPU-hours of total training compute, with experiments run on a single node with 8× NVIDIA H200 GPUs.Online experiments used single training seeds; reported intervals are 95% task-level cluster bootstraps, with disjoint train, validation, and test seeds.Source: J Computational Budget and… · p. 18

K Scientific Artifacts and Licenses

L Human Subjects / M Use of AI Assistants

Loading 2610.18614v1…