Source-linked AI summary
Accurate Failure Prediction in Agents Does Not Imply Effective Failure Prevention
Rakshith Vasudev, Melisa Russak, Dan Bikel, Waseem Alshikh
TL;DR
Proactive LLM-critic intervention is assumed to improve reliability, but deployment outcomes are not determined by offline failure-prediction accuracy alone. The paper formalizes a disruption–recovery tradeoff and proposes a 50-task pilot to predict whether intervention will help or harm. Across regimes, intervention degrades high-success tasks while producing modest gains in low-success settings, making pre-deployment identification of unsafe intervention the framework’s primary value.
Problem
The paper addresses limited evidence about whether proactive LLM-critic interventions improve reliability at deployment time, despite their use for costly agent failures.
Method
The paper compares baseline and intervention outcomes across benchmarks and uses disruption–recovery estimates from a small pilot to determine whether deployment is advisable.
Results
Intervention consistently degrades performance in high-success regimes but can produce modest, reliable gains in low-success regimes, with outcomes driven more by agent response to correction than critic accuracy.
Takeaways & Limitations
Estimate the target agent’s disruption–recovery profile in its target domain before enabling proactive intervention, using the pilot to identify when not to deploy.
Takeaways & Limitations
The framework is validated within distribution, while cross-distribution transfer of pilot estimates is untested and likely unreliable; behavior may also differ across frameworks and domains.
Abstract
from arXiv · showhide
Proactive interventions by LLM critic models are often assumed to improve reliability, yet their effects at deployment time are poorly understood. We show that a binary LLM critic with strong offline accuracy (AUROC 0.94) can nevertheless cause severe performance degradation, inducing a 26 percentage point (pp) collapse on one model while affecting another by near zero pp. This variability demonstrates that LLM critic accuracy alone is insufficient to determine whether intervention is safe. We identify a disruption-recovery tradeoff: interventions may recover failing trajectories but also disrupt trajectories that would have succeeded. Based on this insight, we propose a pre-deployment test that uses a small pilot of 50 tasks to estimate whether intervention is likely to help or harm, without requiring full deployment. Across benchmarks, the test correctly anticipates outcomes: intervention degrades performance on high-success tasks (0 to -26 pp), while yielding a modest improvement on the high-failure ALFWorld benchmark (+2.8 pp, p=0.014). The primary value of our framework is therefore identifying when not to intervene, preventing severe regressions before deployment.
1. Introduction
Execution-time intervention is motivated by costly agent failures but can either recover failing trajectories or disrupt successful ones. The paper formalizes this disruption–recovery tradeoff and argues that deployment should depend on the target agent’s response to intervention, not critic accuracy alone.
- LLM agents face costly failures on complex, multi-step tasks, motivating binary LLM critics that predict failure and intervene during execution.
- The intervention framework compares baseline and critic-assisted outcomes on the same tasks, distinguishing recoveries from baseline failures and disruptions of baseline successes.Baseline failures are F = A + C, while baseline successes are S = B + D.
- Intervention improves performance when baseline failure rate p exceeds the disruption–recovery threshold d/(r + d).The change in success rate is modeled as ∆Success = p · r − (1 − p) · d.
- The same critic policy can mildly help one model while catastrophically harming another because agents differ in how they absorb mid-trajectory corrections.
- Strong critic prediction does not guarantee gains: an AUROC of 0.94 can still accompany large regressions when disruption dominates recovery.The paper reports avoidable performance regressions of up to 26 percentage points and attributes harm to the base agent’s disruption–recovery ratio rather than critic performance.
- The paper proposes evaluating the disruption–recovery condition before deployment and identifies post-hoc selection as a more effective alternative when mid-execution disruption dominates.
2. Related Work
Prior work uses intermediate signals to guide reasoning, feedback, retries, or trajectory selection. This paper instead focuses on deciding whether binary critic-based intervention should be deployed, isolating execution-time disruption as a complementary failure mode.
- Process reward models predict intermediate-step correctness and commonly guide search or rerank trajectories using step-level supervision.
- Best-of-N decoding and self-consistency evaluate completed trajectories post-hoc without altering execution, preserving reasoning context while producing reliable gains.
- Self-correction frameworks augment agents with feedback or retries, but analyses report that intrinsic self-correction can be unreliable and degrade performance.
- This paper is orthogonal to these methods: it provides a framework for deciding whether to deploy binary LLM-critic intervention at all.It examines when accurate external detection signals fail or succeed to improve outcomes once applied during execution.
3. Experimental Setup
The experiments use a binary LLM critic, calibrated intervention policies, and simple ROLLBACK or APPEND mechanisms across benchmarks with distinct baseline success regimes. Evaluation separates task instances across training, validation, and test splits.
- The critic is trained on 7,636 trajectory steps labeled by final task outcome, using Qwen3-0.6B adapted with LoRA rank 16.
- Task-level splitting assigns 158 tasks to training, 19 to validation, and 21 to testing without overlap.
- The critic achieves test AUROC 0.936 and F1 0.963 across 1,372 held-out samples.
- Temperature scaling fits one validation temperature per backbone, triggering intervention when calibrated failure probability exceeds τ = 0.6.
- ROLLBACK undoes the latest action and retries, whereas APPEND preserves the action and adds a warning; the study limits episodes to 15 actions and 3 interventions.
- The deliberately minimal mechanisms establish a lower bound on disruption rate d for evaluating more sophisticated interventions.
- The evaluation uses three benchmarks spanning high-success HotPotQA, medium-success GAIA, and high-failure ALFWorld regimes.
4. Main Results
Intervention outcomes depend on both task regime and model behavior: interventions consistently harm high-success settings but can produce modest gains when baseline failure is high. The disruption–recovery framework and a 50-task pilot correctly predict these outcome directions.
- Framework and evaluation: The disruption–recovery framework predicts that intervention becomes beneficial when baseline failure exceeds d/(r + d).The evaluation tests this prediction across high-success HotPotQA, medium-success GAIA, and low-success ALFWorld.
- High-success regime: HotPotQA: In HotPotQA, no intervention improves performance, with effects ranging from neutral-to-mildly negative to catastrophic depending on the model.Qwen-3-8B underperforms baseline by roughly 2–3 pp in its best setting; GLM-4.7 is 0 to approximately 4 pp worse, while MiniMax-M2.1 is extremely sensitive.
- Medium-success regime: GAIA: In GAIA, no intervention condition outperforms baseline, with moderate degradations for Qwen-3-8B and GLM-4.7 and catastrophic losses exceeding 30% for MiniMax-M2.1.The direction mirrors HotPotQA despite wider confidence intervals from the smaller task set.
- Implications: Strong critic discrimination alone is insufficient: intervention can degrade performance below the disruption–recovery threshold, with sensitivity varying by more than an order of magnitude across models.The dominant limitation is the agent’s ability to absorb mid-trajectory corrections; ignoring d/r risks avoidable regressions.
- Low-success regime: ALFWorld: On ALFWorld, a 50-task pilot estimates failure at approximately 89%, recovery r ≈12%, disruption d ≈56%, and a threshold p⋆≈82%, predicting a positive net effect.The observed failure rate exceeds the estimated threshold: 89% > 82%.
- Low-success regime: ALFWorld: ALFWorld confirms the prediction: uncalibrated APPEND gains +2.8 pp on 202 tasks (p = 0.014), while ROLLBACK achieves the largest absolute improvement of +4.7 pp.No intervention causes degradation in this regime.
- Mechanism-level interpretation: Across benchmarks, non-negative gains occur when recovered episodes exceed disrupted episodes, whereas harmful settings violate d/r < 1.This accounting explains the sign of intervention effects across benchmarks, models, and mechanisms.
- Calibration and model sensitivity: Calibration is regime-dependent, and model scale does not reliably predict robustness to intervention.On ALFWorld, QA-fitted temperatures suppress recovery opportunities, while MiniMax-M2.1 has a 7.3:1 disruption-to-recovery ratio despite its large scale; Qwen-3-8B and GLM-4.7 are more robust.
5. Ablations
Ablations show that neither critic scale, threshold choice, feedback content, nor learned triggering removes the intervention mechanism’s disruption cost. Oracle comparisons further indicate that post-hoc selection offers more headroom than mid-execution intervention.
- Critic scaling: The 14B critic does not improve failure prediction over the 0.6B baseline in this data regime.The 0.6B critic achieves AUROC 0.936, while the best 14B configuration reaches 0.927; higher-capacity configurations can overfit.
- Threshold sensitivity: The best threshold still underperforms the no-intervention baseline by approximately 3 pp on Qwen-3-8B / HotPotQA.Threshold sweeps change intervention frequency but do not alter the underlying disruption–recovery tradeoff.
- Feedback content: Feedback changes outcomes in model-specific ways: removing warnings helps Qwen-3-8B but harms GLM, while contextual feedback worsens MiniMax-M2.1.Feedback sophistication is therefore not a monotone improvement knob.
- Learned versus heuristic policies: In high-success settings, learned critic policies remain below baseline, while matched-rate and late-intervention heuristics perform comparably to the best learned variants.These comparisons attribute most harm to intervention itself rather than poor triggering.
- Oracle comparison: Oracle mid-execution intervention yields 3–8 pp gains, whereas oracle post-hoc Best-of-2 selection yields 11–17 pp.Post-hoc selection avoids the disruption introduced during execution.
- Summary: Across ablations, threshold tuning, feedback design, and critic sophistication cannot overcome model-dependent disruption costs.Simple early-intervention avoidance can match the best learned-critic behavior in high-success regimes.
6. Early-Step Intervention as a Dominant Failure Mode
Early interventions are the dominant source of degradation because they disrupt trajectories that would have succeeded immediately. These disruptions can trigger strategy changes, repeated interventions, and unrecovered failure.
- Failure concentration: Nearly all harm events arise from interventions at steps 0–1, when the baseline agent would have succeeded immediately.On HotPotQA, every observed ROLLBACK and APPEND regression falls into this category.
- Cascade mechanism: A step-0 rollback can make an agent abandon a correct answer, switch strategies, trigger repeated interventions, and exhaust its intervention budget.Agents frequently fail to recover even with substantial remaining step budget.
7. Practical Guidelines
Deployment should be preceded by a domain-specific pilot estimating disruption and recovery for the target agent. Intervention is advisable only when baseline failure exceeds the resulting threshold with a safety margin.
- Pre-deployment assessment: Estimate the target agent’s disruption–recovery profile in the target domain before enabling proactive intervention.The key threshold is p⋆ = d/(r + d).
- Pilot before deployment: A pilot of 50–100 matched tasks estimates baseline failure p, recovery r, and disruption d before deployment.Deploy only when p exceeds p⋆ = d/(r + d) by a safety margin.
- Avoid early intervention: Imposing a minimum step of 2 reduces failures caused by early interventions in high-success settings.Most regressions arise at steps 0–1, where trajectories are often already correct.
- Selection versus intervention: When d/r > 1, prefer post-hoc selection because intervention requires baseline failure above 0.5 and has a low oracle ceiling.Oracle Best-of-2 selection provides substantially larger headroom without mid-execution disruption.
8. Limitations
The paper’s evidence is bounded by critic-scale, mechanism, benchmark, statistical-power, and pilot-transfer limitations. These constraints make the deployment diagnostic domain-specific rather than universal.
- Critic model scale: Critic-scale conclusions are limited because experiments center on a 0.6B critic and only one 14B setup family.Future work should test whether frontier-scale critics can directly reduce disruption rates.
- Intervention mechanisms: The study evaluates only ROLLBACK and APPEND, so more targeted intervention mechanisms may reduce disruption.The framework requires any new mechanism to achieve d/(r + d) below the target failure rate.
- Benchmark and agent coverage: Results cover three benchmarks and three agent backbones within smolagents, so disruption–recovery behavior may differ elsewhere.The authors recommend treating r and d estimates as domain-specific rather than universal.
- Statistical power: The ALFWorld gain of +2.8 pp is directional because it is close to the ±4 pp detection limit at 80% power.Negative results in high-success regimes are described as statistically clear.
- Pilot transfer: Pilot transfer across distributions is untested and likely unreliable given strong model- and domain-dependence.The pilot is validated within-distribution, with ALFWorld predicting ALFWorld outcomes.
9. Conclusion
Execution-time intervention is governed by a disruption–recovery tradeoff, not critic accuracy alone. It often harms high-success agents, offers only modest gains in low-success regimes, and should be piloted before deployment.
- Intervention outcomes are driven less by failure-prediction accuracy than by how each agent responds to mid-trajectory correction.
- Expected improvement requires p > d/(r + d), where r and d are recovery and disruption rates.The criterion explains variation across agents and domains.
- A small pilot estimating recovery and disruption rates should precede deployment, rather than intervention being enabled by default.The guidelines also advise avoiding early-step intervention and using post-hoc selection when disruption outweighs recovery.
- Improved critic accuracy has limited upside unless agents can incorporate corrections without destabilizing behavior, leaving meaningful risk outside high-failure regimes.
Impact Statement
The paper’s deployment value is identifying intervention failure modes before production. Its analyses connect regressions to early disruptions and cascades, while matched-pair and bootstrap procedures support the reported comparisons.
- High-accuracy critics can still degrade deployed performance by up to 26 percentage points, motivating pre-production evaluation.
- Statistical analysis: Paired task-level bootstrap testing uses 10,000 resamples, while Holm–Bonferroni correction controls family-wise α = 0.05 across conditions.
- Measurement: Recovery occurs when baseline fails but intervention succeeds, whereas disruption occurs when baseline succeeds but intervention fails.Rates are computed on matched baseline and intervention task pairs.
- Failure modes: Early-step disruption concentrates intervention harm, including cases where correct answers are replaced by wrong answers, format errors, or no output.Representative examples show correct baseline trajectories derailed after one or more triggers.
- Failure modes: No-answer failures rise from 3.3% at baseline to 46.1% under intervention on MiniMax / HotPotQA as cascades exhaust the step budget.When disruption exceeds recovery, repeated interventions compound harm.
G. Model Sensitivity Analysis
Model sensitivity to intervention varies substantially and is not explained by scale alone. Calibration and recovery differences shape disruption–recovery ratios, while post-hoc selection offers more headroom than mid-execution control.
- Observed sensitivity: −26 pp for MiniMax-M2.1 contrasts with 0.0 pp to −3.7 pp for GLM-4.7 on HotPotQA.The same intervention can therefore be catastrophic for one model and near-neutral for another.
- Mechanisms: Calibration can reduce intervention frequency without improving prediction accuracy, and its effect differs across models.GLM becomes less trigger-happy, whereas MiniMax receives nearly every intervention the critic proposes.
- Mechanisms: 7.3:1 disruption-to-recovery for MiniMax versus 1.5:1 for GLM explains their sharply different responses.For every task MiniMax recovers, 7.3 are disrupted.
- Mechanisms: Intervention cascades raise the no-answer rate from 3.3% to 46.1% on MiniMax / HotPotQA, accounting for a substantial share of performance collapse.Cascades trap the agent in rollback loops until the step budget is exhausted.
- Alternative strategies: A 6–11 pp disruption tax separates oracle selection from oracle intervention, showing that mid-execution control has intrinsic cost beyond prediction error.
- Alternative strategies: Oracle selection has +11 pp average headroom, whereas critic-based selection remains preliminary and requires larger-scale validation.The cited selection study had limited statistical power.
J. Boundary Condition Analysis: Non-Agentic Settings
The framework’s assumptions matter: without multi-step recovery, intervention should not help even when failure is common. Single-shot SWE-bench results are consistent with neutral-to-negative effects, though their estimates are imprecise.
- Boundary setting: Single-shot patch generation removes iterative debugging and tool use, violating the framework’s multi-step recovery assumption.The boundary experiment uses SWE-bench Lite with 30 instances × 3 seeds.
- Boundary prediction: As r → 0, the improvement threshold p⋆ = d/(r + d) → 1, predicting neutral-to-negative intervention effects regardless of baseline failure rate.
- Boundary results: Qwen shows 0.0 pp under APPEND and −2.2 pp under ROLLBACK despite p = 90%, consistent with the absence of recovery dynamics.MiniMax achieves 0% across all conditions, reflecting insufficient base capability for this task format.
- Interpretation: Retry is not recovery in single-shot settings because independent attempts provide no information the model can use to adapt its strategy.
- Statistical caveat: With 90 observations per condition, the study detects effects of at least 12 pp at 80% power, so Qwen’s small effects are consistent with neutrality rather than precise estimates.