When to Intervene? An Empirical Study of Errors and Self-Recovery in LLM Agents

1 Introduction

• Existing intervention approaches do not explicitly characterize self-recovery or its role in subsequent execution, and its relationship to later error handling and task completion remains insufficiently understood.Intervention may interrupt self-recovery before it has an opportunity to occur.Source: §1 · p. 1
• ReInBench contains 1,259 agent traces and 1,403 error events, including 725 self-recoveries across computer-use and Web tasks.It records error occurrence, recognition when observable, recovery outcomes, and execution context.Source: §1 · pp. 1–2
• The study examines recovery frequency and speed, differences across Execution LLMs, links to later error handling and task completion, and recovery prediction.The research questions ask how often and quickly agents recover, whether recovery relates to later errors and task completion, and whether to intervene or wait.Source: §1 · p. 1
• ReCoF combines Recovery Pattern Experts trained on historical contexts and outcomes with Conditional Decision Fusion that adjusts expert contributions to recovery capability and error difficulty.Its goal is to predict whether an error will self-recover and inform whether additional intervention is needed.Source: §1 · p. 2
• ReCoF achieves an average relative improvement of 9.02% in Macro-F1 over the strongest baseline on ReInBench.It predicts whether an error will self-recover to inform whether additional intervention is needed.Source: §1 · p. 2

2 Background

• Figure 1 frames agent self-recovery alongside error detection.The agent’s execution loop consists of actions, observations, and recorded reasoning over time.Source: Figure 1 · p. 2, §2.1 · p. 2

2.1 LLM Agents

• An LLM agent uses an LLM for reasoning and planning, while tools or interfaces execute actions toward a user-specified goal; its time-ordered steps form an agent trace.The trace records actions, environment observations, and reasoning when available.Source: §2.1 · p. 2

2.2 Errors and Self-Recovery

• Errors are incorrect states that may cause failure or deviation, while self-recovery resolves a local error through the agent’s own subsequent reasoning and actions, without external assistance.Recovery concerns the local error, not necessarily completion of the whole task; intervention supplies guidance, whereas waiting lets the agent continue.Source: §2.2 · pp. 2–3

3 ReInBench

• 1,259 agent traces and 1,403 error events form ReInBench; 725 errors self-recover, including 553 with observable recognition and 172 without.The dataset covers computer-use and Web tasks, including browsing, document and spreadsheet editing, and cross-application workflows.Source: §3 · p. 3

3.1 Data Collection

• After filtering, the annotated collection includes 986 OSWorld traces across 358 tasks and 273 Online-Mind2Web traces across 273 tasks.The sources include computer-use agents built on Claude 3.7 Sonnet, Claude Sonnet 4.5, and Kimi K2.5, and SeeAct traces generated with GPT-5-2025-08-07.Source: §3.1 · p. 3

3.2 Annotation

• Each event records its onset, observable recognition, and recovery point; self-recovery requires observed evidence that the error or task-relevant obstruction is removed.Recovery can occur without an observable recognition point, and a recurrence after verified local recovery begins a new event.Source: §3.2 · p. 3
• LLM-assisted annotations undergo rule-based checks and expert verification; every model-generated annotation is manually checked, with disagreements reannotated through full-trace review until consensus.The process preserves initial outputs, verification results, and adjudication records for traceability.Source: §3.2 · p. 4

3.3 Experimental Settings

• RQ1 measures recovery proportions and timing, including onset-to-recovery intervals and, when recognition is observable, separate recognition and correction intervals.Timing analyses use recovered events with identifiable boundaries; events without observable recognition contribute only to end-to-end timing.Source: §3.3 · p. 4
• RQ2 compares later recovery outcomes and durations after prior recovery, then relates trace-level recovery proportions and recovery extent to task completion.The analysis separates same-type from different-type event pairs and groups traces as having no, partial, or complete recovery.Source: §3.3 · p. 4
• For RQ3, ReCoF predicts at error onset whether the error will self-recover, combining historical-context Recovery Pattern Experts with an Assessment LLM verdict through Conditional Decision Fusion.Inputs include the task goal and up to five recent execution steps; future execution and recovery annotations are excluded.Source: §3.3 · pp. 4–5
• RQ3 uses task-disjoint train, validation, and test partitions, compares ReCoF with monitoring baselines, and reports precision, recall, F1, Macro-F1, and accuracy.The study also tests ablations removing Recovery Pattern Experts or Conditional Interaction, and calibrates a separate decision threshold for each Execution LLM.Source: §3.3 · p. 5

4.1 Self-Recovery Performance (RQ1)

• Recovery durations concentrate within a few steps but have long tails, while Execution LLMs differ in self-recovery frequency and task coverage.Median end-to-end durations are two steps for OSWorld and three for Web tasks; across the broader shared-task set, each LLM has a median observed recovery duration of two steps.Source: §4.1 · p. 6, Figure 4 · p. 6, Figure 5 · p. 7

4.3 Intervention Timing (RQ3)

• ReCoF uses Recovery Pattern Experts trained on historical execution contexts and outcomes, then conditionally weights their predictions using an Assessment LLM’s contextual verdict and the Execution LLM’s identity.The weighting accounts for differences in recovery capability across models and the difficulty of the current error.Source: §4.3 · pp. 6–7
• ReCoF exceeds the strongest baseline in mean Macro-F1 by 4.78, 1.30, and 11.30 percentage points for Claude 3.7 Sonnet, Claude Sonnet 4.5, and Kimi K2.5, respectively.Its mean Macro-F1 scores are 73.53%, 74.18%, and 72.99%, respectively.Source: Table 2 · p. 7, §4.3 · p. 8

4.2 Subsequent Execution (RQ2)

• 76.0% of later errors self-recover in same-type pairs, compared with 56.8% across all error-event pairs.For same-type pairs, 79.4% of later recoveries take no more steps than the earlier recovery, versus 63.5% overall; evidence for different-type pairs is more limited.Source: §4.2 · p. 6, Figure 6 · p. 7
• Task completion reaches 60.0% when all annotated errors recover, compared with 20.6% when none recover and 18.0% when only some recover.Successful traces also have a higher mean proportion of recovered errors than unsuccessful traces, 71.9% versus 37.8%.Source: §4.2 · p. 6, Figure 7 · p. 7

4.3 Intervention Timing (RQ3)

• Threshold offsets of ±0.01 change mean Macro-F1 by at most approximately 1.43 percentage points, while larger offsets reduce performance.Five or ten task folds also yield competitive results across the three Execution LLMs.Source: Figure 8 · p. 7, §4.3 · p. 8
• Removing either Recovery Pattern Experts or conditional interaction reduces Macro-F1, indicating that both prediction sources and their combination contribute to ReCoF’s results.The ablations report distinct reductions across the three Execution LLMs.Source: §4.3 · p. 8

4.4 Threats to Validity

• ReInBench covers a limited set of Execution LLMs, task environments, and error types, and RQ3 evaluates self-recovery prediction without executing interventions.The authors call for broader coverage and experiments examining how intervention timing and methods jointly affect task completion.Source: §4.4 · p. 8

5.1 Self-Recovery in LLM Agents

• Prior work evaluates trained self-reflection and executable recovery cases, whereas ReInBench analyzes naturally occurring self-recovery during original execution [14] [3].It records error occurrence, observable recognition, and recovery, enabling study of frequency, speed, and variation across Execution LLMs.Source: §5.1 · p. 8
• Experience-reuse studies address later trials [12] or memory management [17], while this analysis tracks recovery experience within the same trace.It compares recovery outcomes and durations for subsequent same-type and different-type errors, and relates local recovery to task completion.Source: §5.1 · p. 8

5.2 Errors and Intervention

• Monitoring methods target decisive errors or task failure [21], while ReCoF predicts whether a particular local error will self-recover.Source: §5.2 · p. 8
• Recovery systems provide corrective guidance or retrieve recovery lessons [10], while observation-refresh work estimates the benefit of a specific recovery operation.Those intervention decisions concern the expected benefit of assistance, rather than directly predicting local self-recovery.Source: §5.2 · p. 8
• ReCoF combines learned recovery patterns with contextual assessment to predict whether a local error will self-recover and inform intervention decisions.The target is a particular error’s recovery through continued execution, rather than decisive errors or eventual task failure.Source: §5.2 · p. 8

• The study constructs ReInBench to analyze self-recovery across computer-use and Web tasks.Source: §6 · pp. 8–9
• 9.02% average relative improvement in Macro-F1 over the strongest baseline on ReInBench is achieved by ReCoF.Source: §6 · p. 9
• Agents self-recover from a substantial share of errors, often within a few steps, with performance varying across Execution LLMs.Self-recovery also supports handling subsequent same-type errors and overall task completion.Source: §6 · p. 9
• The authors identify broader dataset coverage and experiments on intervention timing and methods as future work.Source: §6 · p. 9

• ReCoF predicts whether an error will self-recover to inform whether additional intervention is needed.Source: A Detailed Description of ReCoF · p. 10
• Recovery Pattern Experts learn relationships between historical execution contexts and recovery outcomes.Source: A Detailed Description of ReCoF · p. 10
• Conditional Decision Fusion adjusts expert contributions using the Assessment LLM’s contextual verdict and the Execution LLM’s identity.Source: A Detailed Description of ReCoF · p. 10

• ReCoF receives recent execution context for the error, using a window of H = 5 steps.The context includes observations such as available feedback and screenshots; the final included step has the observation after the error action.Source: §A.1 · p. 10
• Prediction uses information available at error occurrence, excluding retrospective error descriptions and execution after that step.A common observation boundary makes historical examples and prediction-time inputs comparable.Source: §A.1 · p. 10
• The prediction target is whether the local error fails to self-recover within the observed trace, not whether the entire task is completed.Non-recovery is the positive class, with historical recovery outcomes providing supervision.Source: §A.1 · p. 10

A.2 Recovery Pattern Experts

• Text, visual, and joint image–text views connect task interpretation and application state with recovery outcomes.The views encode task and execution text, screenshots, and their interleaved context.Source: §A.2 · p. 10
• Temporal summaries capture recent changes alongside the execution context, including reasoning and screenshot differences across windows.Additional descriptors summarize action types and parameters, repeated actions, and textual cues; missing features use zero vectors with availability indicators.Source: §A.2 · p. 10
• The expert bank combines multiple classifier families with model-specific predictors and partial pooling across Execution LLMs.The partial-pooling kernel adjusts the weight assigned to examples from other Execution LLMs through η.Source: §A.2 · p. 10
• A robust RBF predictor is selected by the lower decile of bootstrap validation AUC, while the ensemble votes across selected predictor families.The ensemble selects a predictor and threshold by Macro-F1 in 200 label-stratified validation subsets.Source: §A.2 · p. 10

A.3 Conditional Decision Fusion

• Conditional Decision Fusion addresses cases where similar execution patterns can have different recovery outcomes under different task requirements.It uses contextual assessment to adjust expert contributions to the current error context.Source: §A.3 · pp. 10–11
• The Assessment LLM judges whether an error will self-recover from the task goal, five-step execution text, and available screenshots.Its instruction distinguishes correcting the target error from completing the task; its binary verdict marks predicted non-recovery as 1 and self-recovery as 0.Source: §A.3 · p. 11
• The fusion representation combines expert scores with the Execution LLM identity and contextual verdict.The fusion vector has 2D + M + 1 dimensions, where the Execution LLM identity is one-hot encoded.Source: §A.3 · p. 11
• The contextual verdict changes each expert’s contribution, while Execution LLM identity adds a model-specific adjustment.The same expert score can receive different contributions under predicted self-recovery and predicted non-recovery; model-specific experts and thresholds provide additional adaptation.Source: §A.3 · p. 11

A.4 Fusion Training and Decision Calibration

• Three task-grouped validation folds keep events from each task together, and final performance is evaluated on a task-disjoint test partition.Each fusion model is fitted on the other two folds, producing out-of-fold predictions for its held-out tasks.Source: §A.4 · p. 11
• Model–class group weights balance the regularized logistic objective, limiting domination by the largest group and limiting score-combination complexity.The objective uses C = 0.01.Source: §A.4 · p. 11
• For each Execution LLM, the final threshold is the median of 200 bootstrap thresholds that maximize Macro-F1 on validation predictions.Candidate thresholds include midpoints between distinct validation scores, two boundary candidates, and 0.5; ties favor the candidate closest to 0.5, then the smaller value.Source: §A.4 · p. 11
• Averaging fold models reduces dependence on one validation partition, while model-specific thresholds accommodate calibration differences.The fold models are retained without a final refit.Source: §A.4 · p. 11
• At error occurrence, ReCoF predicts whether an error is likely to self-recover or remain unresolved to inform intervention decisions.The predictions provide a basis for deciding when to allow continued execution and when to prioritize external intervention.Source: §A.4 · p. 11