PaperlayerView exact revision on arXiv

Source-linked AI summary

When Harnesses Lose the Signal: Causal Evaluation of Recovery in LLM Agents

Shuyao Xiao, Shengling Wang, Xuan Chen, Ke Chao, Ming Cui, Feifei Qian, Chaoyang Mei, Fanlin Meng, Ziming Yu, Junxi Yin

arXiv:2610.00372v1cs.AI

TL;DR

Average success alone does not show which trajectories recovery rescues or harms, or how recovery’s value changes with timing. The paper compares paired runs from the same state and uses their outcomes to train CIR, a selective refresh policy. On held-out ALFWorld tasks, CIR improves success over never refreshing while leaving evaluated clean trajectories untouched.

  • Problem

    Average task success does not show which trajectories recovery changes, even though refresh can rescue a failing trajectory or derail one that would otherwise succeed.

    Source: §1 · p. 1

  • Method

    The paper compares paired continuations from the same state with and without recovery, then trains CIR to estimate refresh value from information available before intervention.

    Source: §1 · p. 2, §4.1 · p. 4, §4.2 · p. 5

  • Results

    CIR raises success from 70.33% with never refreshing to 73.33% across 300 episodes, a 3.00-percentage-point gain.

    Source: §5.4 · p. 8

  • Takeaways & Limitations

    Paired causal evidence can support more selective recovery without retraining the underlying agent.

    Source: §1 · p. 2, §6 · p. 9

    Experiments use controlled benchmark environments and do not involve human subjects or private user data.

    Source: ETHICS STATEMENT · p. 10

Abstract

from arXiv · show

Large language model agents rely on external harnesses to pass information between the model and its environment and to recover from execution errors. Yet recovery is usually judged only by average task success. This hides an important tension. The same operation can rescue a failing trajectory or disrupt one that would otherwise succeed. We frame recovery as a causal decision problem. Starting from the same execution state, we compare what happens with and without recovery, separate rescue from harm, and study how the value of recovery changes over time. We then introduce the Causal Intervention Router (CIR), a lightweight policy that uses information available before recovery to decide when intervention is worthwhile. On long-horizon ALFWorld tasks with Qwen3-14B, CIR raises success from 70.33% to 73.33%, a gain of 3.00 percentage points. It leaves all evaluated trajectories with correct observations untouched. Additional controls show that the benefit of recovery cannot be explained solely by the new observation returned by the environment. These results provide a practical way to evaluate recovery and apply it selectively.

1 INTRODUCTION

Recovery can rescue a failing trajectory or harm one that would otherwise succeed, so average success alone misses which trajectories change. The paper evaluates recovery through paired continuations and uses that evidence to route refresh selectively.

1 INTRODUCTION

  • Average success does not reveal which trajectories recovery changes: refresh can rescue a failure or derail a trajectory that would otherwise succeed.The same average gain may reflect different balances of rescues and harms.Source: §1 · p. 1
  • The evaluation addresses two challenges: an ordinary run reveals only one outcome, and recovery’s value can change as actions and feedback accumulate.Comparing both choices from the same state at different times makes these effects measurable.Source: §1 · p. 1
  • Paired runs from the same execution state compare refresh with no recovery, separating rescue, harm, and unchanged outcomes.The framework repeats comparisons at different times and across refresh variants.Source: §1 · p. 2, Figure 1 · p. 2
  • The paired outcomes serve both to evaluate recovery and to train CIR, which uses only information available at decision time.CIR is designed to limit harm to trajectories with correct observations.Source: §1 · p. 2
  • CIR raises Qwen3-14B success from 70.33% to 73.33%, a 3.00-percentage-point gain, while leaving all evaluated clean trajectories untouched.The gain reaches 9.33 points when the agent receives a more severely outdated observation.Source: §1 · p. 2

2 RELATED WORK

Prior recovery methods improve agent execution, while aggregate evaluations and failure-diagnosis approaches leave open which trajectories recovery changes and when it should be applied. This work distinguishes itself by measuring recovery’s causal value across states and times, then using the paired evidence to train a decision rule.

2 RELATED WORK

  • Recovery methods use reflection, feedback, or search, but aggregate evaluations do not identify changed trajectories or successful ones disrupted by recovery.Examples include Reflexion, Self-Refine, CRITIC, and LATS.Source: §2 · pp. 2–3
  • Failure diagnosis and counterfactual repair methods localize causes or repair failures, but do not directly answer whether recovery from the same state improves or damages outcomes.The related work includes AgentBoard, MAST, and Causal Agent Replay (Shah, 2026).Source: §2 · p. 3
  • This work keeps the recovery operation fixed while measuring its causal value across states and times, using the same evidence to train a deployment-time decision rule.This distinguishes its focus from failure localization and repair generation.Source: §2 · p. 3

3 PAIRED COUNTERFACTUAL EVALUATION OF RECOVERY

The evaluation changes the observation while keeping the underlying task fixed, then branches execution from the same state with and without recovery. It measures rescue and harm separately because an average effect alone does not show how recovery changes individual outcomes.

3 PAIRED COUNTERFACTUAL EVALUATION OF RECOVERY

  • Paired counterfactual evaluation compares recovery with continued execution from the same task state, isolating the recovery decision and revealing rescue and harm.Controlled observation changes make the error explicit.Source: §3 · p. 3, §3.3 · p. 4

3.1 PERTURBATION AND RECOVERY SETTING

  • The setting varies observation condition and recovery delay: observations are clean, stale, or missing, and recovery may be refresh or replanning.Stale observations use information from an earlier state; missing observations remove their content.Source: §3.1 · p. 3

3.2 COUNTERFACTUAL COMPARISON

  • Each task branches at the same state and history into a no-recovery continuation and one applying the selected operation.The paired design holds the earlier trajectory fixed for comparison.Source: §3.2 · p. 3

3.3 RESCUE AND HARM

  • Recovery rescues when it turns failure into success and harms when it turns success into failure; the paired outcomes also include preserved success and unchanged failure.The average recovery effect is the rescue rate minus the harm rate, so the net alone cannot distinguish rare changes from frequent opposing changes.Source: §3.3 · p. 4

4 FROM EVALUATION TO SELECTIVE RECOVERY

CIR uses pre-recovery information to decide whether and when to refresh, balancing expected improvement under erroneous observations against harm to clean trajectories. Its paired-outcome models distinguish detecting an error from predicting whether refresh will help.

4 FROM EVALUATION TO SELECTIVE RECOVERY

  • CIR estimates recovery value from pre-decision information, refreshing only when expected benefit justifies risk and at most once per trajectory.It combines error detection with separate predictions of success under refresh and continued execution.Source: §4 · p. 4

4.1 DECISION OBJECTIVE

  • The objective rewards improvement under erroneous observations relative to never recovering, while limiting clean trajectories turned from success into failure.At runtime, CIR estimates recovery value before either possible outcome is known.Source: §4.1 · p. 4

4.2 ESTIMATING RECOVERY VALUE

  • CIR uses separate models for observation error and success outcomes; its rescue and harm scores identify promising states but are not probabilities of rescue or harm.The models are supervised by observation condition and paired no-recovery and refresh outcomes.Source: §4.2 · p. 5

4.3 DYNAMIC RECOVERY DECISIONS

  • CIR reconsiders refresh at multiple candidate times, acting at the first time all conditions are met and otherwise continuing without recovery.Thresholds are selected on development data to maximize net gain under erroneous observations while keeping clean-trajectory harm within the specified limit.Source: §4.3 · pp. 5–6

5 EXPERIMENTS

On held-out ALFWorld tasks, CIR improves success over never refreshing while leaving clean-observation episodes untouched. Paired experiments show that refresh effects depend on observation state and timing, and that returned observation content is not necessary for most shared rescues.

5.1 EXPERIMENTAL SETUP

  • In ALFWorld, paired runs began from the same state and model input, differing only in recovery; perturbations included one- or two-step stale or missing observations, with delays d=0,1,2,4.The evaluation included 75 prefix-feasible tasks, each tested under four observation conditions, for 300 episodes; these tasks did not overlap with the 93 fitting tasks.Source: §5.1 · p. 6

5.2 RECOVERY DEPENDS ON STATE AND TIMING

  • At delay 0, refresh lowers success by 6.45 points under clean observations but raises it by 7.53 points under two-step stale observations, a 13.98-point difference.The stale-minus-clean difference remains positive at delays 1, 2, and 4; the factual-success cohort contains 93 tasks.Source: §5.2 · p. 7, Figure 2 · p. 7
  • For the complete primary cohort, refresh changes success by −3.77 points under clean observations and +6.60 points under stale observations, a 10.38-point difference.In the independent cohort, the corresponding effects are 0.00 and +4.00 points, a 4.00-point difference; Figure 2 summarizes the complete cohorts.Source: §5.2 · pp. 7–8, Figure 2 · p. 7
  • In the complete primary cohort, refresh produces 2 rescues and 6 harms under clean observations, versus 17 rescues and 10 harms under stale observations.The paired outcome counts show how different rescue and harm patterns can underlie average effects.Source: §5.2 · p. 8

5.3 WHY DOES REFRESH HELP?

  • At delay 4 under two-step stale observations, full refresh, content-ablated refresh, and replanning change success by +13.43, +10.45, and +1.49 points, respectively.Full refresh rescues ten tasks, and content-ablated refresh rescues eight of those, showing returned content is not required for those shared rescues.Source: §5.3 · p. 8, Figure 3 · p. 8

5.4 CIR LEARNS WHEN TO RECOVER

  • 3.00 percentage points: CIR raises success from 70.33% to 73.33% over never refreshing on 300 held-out episodes, with 11 rescues and 2 harms.It intervenes in 16.3% of episodes.Source: §5.4 · p. 8, Figure 4 · p. 9
  • 9.33 points: under two-step stale observations, CIR raises success from 60.00% to 69.33%; it never intervenes on clean episodes.The gain under stale observations comes from 9 rescues and 2 harms; gains for one-step stale and missing observations are each 1.33 points.Source: §5.4 · p. 8, Table 1 · p. 9
  • CIR has the highest point estimate against two anomaly baselines, but its margins are 1.33 and 1.00 points with confidence intervals spanning zero.The rate-matched anomaly baseline intervenes in 15.7% of episodes and gains 2.00 points, versus CIR's 16.3% intervention rate and 3.00-point gain.Source: §5.4 · p. 9

6 CONCLUSION

Paired causal evidence supports selective recovery: CIR improves success by 3.00 percentage points without retraining the agent. The experiments are limited to controlled benchmark environments and exclude human subjects and private user data.

6 CONCLUSION

  • CIR improves success by 3.00 percentage points across 300 episodes from 75 held-out tasks, without retraining the underlying agent.Its largest gain occurs when observations are more severely outdated, and it attains the highest observed success rate among the compared policies.Source: §6 · p. 9
  • The results support using paired causal evidence to apply recovery selectively in LLM-agent harnesses.The paired evaluation separates trajectories recovery rescues from those it harms and shows that the value of refresh depends on observation, timing, and trajectory.Source: §6 · p. 9
  • Generative AI tools were used only to polish manuscript language, and the authors reviewed edits and take responsibility for the final content.They report no use of these tools for research ideas, methodology, experiments, analysis, or scientific conclusions.Source: AI USE STATEMENT · p. 10
  • The experiments use controlled benchmark environments and involve neither human subjects nor private user data.The authors identify no additional ethical risks beyond those commonly associated with LLM-agent research.Source: ETHICS STATEMENT · p. 10

A TASK-LEVEL OVERLAP BETWEEN RECOVERY OPERATIONS

The two refresh variants agree on most task outcomes, including many cases where neither changes the result. Both yield positive paired gains over no recovery, while full refresh has a modest point-estimate advantage over content-ablated refresh.

A TASK-LEVEL OVERLAP BETWEEN RECOVERY OPERATIONS

  • 59 of 65 task outcomes agree across the refresh variants, including 38 stable successes and 13 stable failures.The six disagreements comprise three rescues and three differences in harm outcomes.Source: A TASK-LEVEL OVERLAP BETWEEN RECOVERY… · p. 13
  • Eight shared rescues show that returned content is not necessary for those observed benefits.The analysis compares full and content-ablated refresh outcomes against the same no-recovery outcome.Source: A TASK-LEVEL OVERLAP BETWEEN RECOVERY… · p. 13
  • Paired net gains are nine tasks for full refresh versus no recovery, seven for content-ablated refresh versus no recovery, and two for full versus content-ablated refresh.Excluding two tasks that had already terminated changes the analysis denominator from 67 to 65 but not these gains.Source: A TASK-LEVEL OVERLAP BETWEEN RECOVERY… · p. 13
  • Full refresh has a 3.08-pp point estimate over content-ablated refresh, while both variants produce positive gains relative to no recovery.The joint analysis uses the 65 tasks on which both interventions can be executed, with matching task identity and exact token prefixes before recovery.Source: A TASK-LEVEL OVERLAP BETWEEN RECOVERY… · pp. 13–14

B ADDITIONAL RECOVERY RESULTS

At delay d = 1, paired outcomes in the factual-success cohort show that stale-state refresh includes both rescues and harms, while clean-state trajectories are mostly successful but include harms.

B ADDITIONAL RECOVERY RESULTS

Related papers

Loading 2610.00372v1…