Source-linked AI summary
Accurate Failure Prediction in Agents Does Not Imply Effective Failure Prevention
Rakshith Vasudev, Melisa Russak, Dan Bikel, Waseem Alshikh
TL;DR
Because deployment-time effects of proactive LLM-critic interventions are poorly understood, this paper develops a disruption–recovery framework and pilot-based estimate of whether intervention will help or harm. It finds that intervention degrades high-success regimes but can yield modest gains in low-success regimes, with outcomes depending on agent response rather than critic accuracy alone.
Problem
Whether proactive LLM-critic interventions reliably benefit deployed agents remains unclear, despite their use to mitigate failures in costly multi-step tasks.
Source: §1 · p. 1Method
The paper models intervention through recovery and disruption rates and uses a pilot to estimate whether deployment is likely to help or harm.
Source: §9 · p. 8Results
Intervention degrades high-success regimes but can yield modest, reliable gains in low-success regimes, with gains on the order of 1–3 pp.
Source: §9 · p. 8Takeaways & Limitations
The framework’s practical value is identifying when not to intervene, while gains in low-success regimes remain bounded.
Source: §9 · p. 8Pilot estimates are validated within-distribution; cross-distribution transfer is untested and likely unreliable given model- and domain-dependence.
Source: §8 · p. 8
Abstract
from arXiv · showhide
Proactive interventions by LLM critic models are often assumed to improve reliability, yet their effects at deployment time are poorly understood. We show that a binary LLM critic with strong offline accuracy (AUROC 0.94) can nevertheless cause severe performance degradation, inducing a 26 percentage point (pp) collapse on one model while affecting another by near zero pp. This variability demonstrates that LLM critic accuracy alone is insufficient to determine whether intervention is safe. We identify a disruption-recovery tradeoff: interventions may recover failing trajectories but also disrupt trajectories that would have succeeded. Based on this insight, we propose a pre-deployment test that uses a small pilot of 50 tasks to estimate whether intervention is likely to help or harm, without requiring full deployment. Across benchmarks, the test correctly anticipates outcomes: intervention degrades performance on high-success tasks (0 to -26 pp), while yielding a modest improvement on the high-failure ALFWorld benchmark (+2.8 pp, p=0.014). The primary value of our framework is therefore identifying when not to intervene, preventing severe regressions before deployment.
1. Introduction
LLM agents face costly failures, and execution-time critics aim to redirect predicted failures but can substantially degrade outcomes. The section frames intervention safety as an agent-dependent tradeoff between recovery and disruption, not critic accuracy alone.
Introduction
- Intervention helps only when failure likelihood p exceeds d/(r + d), with recoveries and disruptions determining the threshold.Recoveries are baseline failures that intervention turns into successes; disruptions are baseline successes turned into failures.Source: §1 · p. 1
- The same critic policy can help one agent yet harm another because disruption and recovery depend more on base-agent properties than critic accuracy.Across models, benchmarks, and intervention mechanisms, differences in disruption–recovery profiles explain observed gains and regressions.Source: §1 · pp. 1–2
- A critic with AUROC 0.94 can still fail to improve performance and cause regressions up to 26 percentage points; harm tracks the base agent’s disruption–recovery ratio.The reported regressions occur despite strong predictive signal.Source: §1 · p. 2
- Oracle analysis limits gains from perfect failure prediction to 4 to 8 percentage point gains, while scaling the critic from 0.6B to 14B does not improve prediction quality in this data regime.The bottleneck is whether agents absorb mid-trajectory corrections or are destabilized by them.Source: §1 · p. 2
- When disruption dominates recovery, mid-execution control has a low ceiling, and post-hoc selection is a more effective alternative.Intervention schemes that ignore this tradeoff risk doing more harm than good, regardless of critic scale or accuracy.Source: §1 · p. 2
2. Related Work
The section positions the paper among methods that use signals or feedback to improve agent trajectories, while distinguishing its focus: deciding whether execution-time intervention should be deployed at all.
2. Related Work
- Process reward models predict intermediate-step correctness and guide search or rerank trajectories, typically using step-level supervision (Lightman et al., 2024; Setlur et al., 2025).Source: §2 · pp. 2–3
- Best-of-N decoding (Stiennon et al., 2020), self-consistency (Wang et al., 2023; Kang et al., 2025), and Prospector select completed trajectories post-hoc without altering execution.These methods preserve reasoning context and have shown reliable gains, contrasting with the large gap between intervention and selection ceilings (Marín, 2025).Source: §2 · p. 3
- Self-correction frameworks add feedback or retries while preserving context, but intrinsic self-correction is often unreliable and can degrade performance (Huang et al., 2024; Wu et al., 2024).Examples include Reflexion (Shinn et al., 2023), Self-Refine (Madaan et al., 2023), and Reflect, Retry, Reward.Source: §2 · p. 3
- The framework is orthogonal to new intervention mechanisms, asking whether any binary LLM critic-based intervention should be deployed when external detection succeeds or fails to improve execution-time outcomes.It examines when accurate external detection signals do not translate into better outcomes once intervention is applied.Source: §2 · p. 3
3. Experimental Setup
The experiments test a Qwen3-0.6B critic trained to predict task outcomes, using held-out tasks, calibrated and uncalibrated predictions, and two simple intervention mechanisms. They span benchmarks with high, medium, and low baseline success rates.
LLM critic model and training data
- The Qwen3-0.6B critic was LoRA-adapted (rank 16) on 7,636 smolagents trajectory steps from HotPotQA and GAIA, with task-outcome labels.Data were split by unique tasks into 158 training, 19 validation, and 21 test tasks with no overlap; ALFWorld was unseen during training.Source: §3 · p. 3
LLM critic performance
- On 1,372 held-out samples, the critic achieved AUROC 0.936 and F1 0.963 across evaluated agent backbones.Performance varied by backbone but remained consistently high; Table 1 reports results aggregated by backbone.Source: §3 · p. 3, Table 1 · p. 3
Calibration
- Temperature Scaling reduced ECE by 44% for Qwen-3-8B and 58% for GLM-4.7, had negligible effect for MiniMax-M2.1, and used a failure threshold τ = 0.6 by default.A single temperature per backbone was fitted on validation data; downstream experiments report calibrated and uncalibrated settings. Table 2 summarizes calibration results.Source: §3 · p. 3, Table 2 · p. 3
Intervention mechanisms
- ROLLBACK undoes the latest action and permits a retry when predicted failure exceeds τ, while APPEND executes the action and adds a warning message.These minimal mechanisms serve as natural practitioner baselines; within the theoretical framework, they establish a lower bound on disruption rate d relative to recoverable benefit r.Source: §3 · pp. 3–4
Experimental design
- A 2 × 2 design crossed ROLLBACK versus APPEND with calibrated versus uncalibrated critic outputs, under budgets of 15 actions and 3 interventions per episode.The design separates calibration effects from intervention effects.Source: §3 · p. 4
Benchmarks
- Baseline success rates were 51–64% on HotPotQA [Yang et al., 2018], 19–47% on GAIA [Mialon et al., 2024], and 5.8–14.7% on ALFWorld.These benchmarks represent high-, medium-, and low-success regimes, respectively; Table 3 summarizes their statistics and characteristics.Source: §3 · p. 4, Table 3 · p. 4
4. Main Results
Intervention outcomes depend on both task regime and model: the disruption–recovery framework correctly predicts their direction, with degradation in high-success settings and gains possible in low-success settings. Strong critic discrimination alone does not ensure safe intervention.
High-success regime: HotPotQA
- On HotPotQA, Qwen-3-8B gains were unmeasurable, GLM-4.7 effects ranged from 0 to ≈4 pp negative, and MiniMax-M2.1 variants reduced success by 25–30 pp.Baseline success ranged from 57–70%; MiniMax’s losses were statistically significant and persisted across calibration and mechanism choices.Source: §4 · p. 4
Medium-success regime: GAIA
- On GAIA, no intervention outperformed baseline: Qwen-3-8B and GLM-4.7 fell roughly −4 to −13 pp, while MiniMax-M2.1 losses exceeded 30%.Despite wider confidence intervals from fewer tasks, the direction mirrored HotPotQA.Source: §4 · p. 4
Low-success regime: ALFWorld
- With an 89% failure rate above the estimated 82% threshold, ALFWorld interventions were predicted to help; full evaluation found APPEND +2.8 pp (p = 0.014) and ROLLBACK +4.7 pp.A 50-task pilot estimated recovery r ≈12% and disruption d ≈56%; the critic transferred zero-shot with frozen weights and temperature fitted on ALFWorld validation data.Source: §4 · pp. 4–5, Table 4 · p. 5
Mechanism-level interpretation
- Across ALFWorld cases with non-negative gains, recovered episodes exceeded disrupted ones (d/r < 1), while all harmful HotPotQA and GAIA settings violated that inequality.This accounting explains the sign of effects across benchmarks, models, and mechanisms.Source: §4 · p. 5
Calibration
- On ALFWorld, QA-fitted temperatures suppressed intervention too aggressively, so uncalibrated critics outperformed calibrated ones, reversing the high-success pattern.Calibration must match underlying recovery dynamics rather than being assumed universally beneficial.Source: §4 · p. 5
Model sensitivity
- Intervention robustness did not track model scale: MiniMax-M2.1’s disruption-to-recovery ratio was 7.3:1, versus 3% intervention-rate reduction and 12% recovery, compared with GLM-4.7’s 71% reduction and 25% recovery.MiniMax suffered catastrophic losses on HotPotQA and GAIA, while Qwen-3-8B and GLM-4.7 experienced more moderate harm in unfavorable regimes and modest gains in favorable ones; the limitation is agents’ ability to absorb corrections.Source: §4 · p. 5
Summary
- Across benchmarks, the disruption–recovery framework correctly predicts intervention direction: degradation in high-success settings and gains in low-success settings.The results are summarized in Table 4.Source: §4 · p. 5, Table 4 · p. 5
5. Ablations
Across ablations, high-success intervention harms persist despite changes to critic scale, threshold, feedback, or policy, with outcomes shaped by disruption and model-specific responses. Even perfect failure prediction yields only modest mid-execution gains, while post-hoc selection avoids disruption.
5.1. Scaling the LLM critic
- 0.905 vs. 0.936 overall AUROC for 14B v1 versus the 0.6B critic, while the best 14B variant reaches 0.927 and v3 falls to 0.752.With approximately 4,000 training examples from two benchmarks, the passage identifies training-data diversity—not model capacity—as the binding constraint.Source: Table 5 · p. 6, §5.1 · p. 6
5.2. Threshold sensitivity
- τ = 0.7 is the best tested threshold on Qwen-3-8B / HotPotQA, but remains approximately 3 pp below the 57% no-intervention baseline.The response is nonmonotonic: low thresholds over-intervene, while very high thresholds under-intervene but still trigger on occasional false positives.Source: Table 6 · p. 6, §5.2 · p. 6
5.3. Role of feedback content
- Removing visible feedback helps Qwen-3-8B but harms GLM, while contextual LLM feedback slightly improves Qwen’s APPEND intervention without beating baseline and substantially worsens MiniMax-M2.1.Feedback sophistication is not a monotone improvement: it can raise disruption as readily as recovery.Source: §5.3 · p. 6, Table 7 · p. 6, Table 8 · p. 6
5.4. Learned vs. heuristic intervention policies
- On Qwen-3-8B / HotPotQA, learned-critic interventions remain below baseline and are broadly comparable to matched-rate heuristics; late intervention performs similarly to the best critic variants.Avoiding early intervention captures most of the learned critic’s practical benefit in this high-success regime.Source: Table 9 · p. 7, §5.4 · p. 7
5.6. Summary
- Threshold tuning cannot fix negative net effects, feedback and timing interact with the agent, and simple late-intervention heuristics match learned critics in high-success settings.The summary attributes the bottleneck to intervention disruption rather than sophisticated failure prediction.Source: §5.6 · p. 7
5.5. Oracle ceiling: limits of intervention.
- 3–8 pp gains from oracle mid-execution intervention trail 11–17 pp from oracle post-hoc Best-of-2 selection.Post-hoc selection operates on completed trajectories and avoids mid-execution disruption.Source: Table 10 · p. 7, §5.5 · p. 7
6. Early-Step Intervention as a Dominant Failure Mode
Nearly all intervention-related harm arose from steps 0–1, when the baseline agent would have succeeded immediately. Early interventions can destabilize correct trajectories, motivating a minimum-step constraint such as min step≥2.
6. Early-Step Intervention as a Dominant Failure Mode
- On HotPotQA, all observed regressions for both ROLLBACK and APPEND followed steps 0–1 interventions that disrupted trajectories the baseline would have solved immediately.The analysis attributes degradation to disrupting already-correct trajectories rather than errors in long-horizon reasoning.Source: §6 · p. 7
- A step-0 rollback can make a sensitive model abandon a correct answer, trigger repeated interventions, exhaust the budget, and prevent recovery despite remaining steps.This cascade motivates a minimum-step constraint, such as min step≥2, to prevent early-step intervention.Source: §6 · p. 7
7. Practical Guidelines
Estimate the target agent’s disruption–recovery profile in the target domain before enabling proactive intervention; the key threshold is p⋆=d/(r+d), below which intervention is expected to be net harmful.
7. Practical Guidelines
- Deploy only when baseline failure probability p exceeds p⋆=d/(r+d) by a safety margin.Estimate p, recovery rate r, and disruption rate d by comparing outcomes with intervention enabled versus disabled under matched conditions.Source: §7 · pp. 7–8
- Run a small 50–100-task pilot with the chosen intervention mechanism before full deployment; the study uses calibrated ROLLBACK.Source: §7 · p. 7
- Most regressions in high-success settings arise from interventions at steps 0–1; imposing min step≥2 reduces failures and recovers some avoidable harm.Source: §7 · p. 8
- When d/r > 1, p⋆>0.5 and intervention can help only if failure is likelier than success; prefer post-hoc multi-trajectory selection when compute permits.Oracle intervention has only a single-digit-point ceiling, while oracle Best-of-2 selection has substantially larger headroom without disruption.Source: §7 · p. 8
8. Limitations
The limitations concern critic scale, simple intervention mechanisms, narrow benchmark and framework coverage, limited power for the positive ALFWorld result, and untested cross-distribution pilot transfer.
Critic model scale
- The 0.6B critic reached AUROC 0.936 versus 0.927 best at 14B, while perfect prediction offers only 4–8 pp gain, suggesting scale is not the dominant bottleneck.Whether much larger or frontier-scale critics can directly reduce disruption rates remains untested.Source: §8 · p. 8
Intervention mechanism complexity
- ROLLBACK and APPEND are intentionally simple; more targeted corrections may reduce disruption, and viability requires d/(r + d) below the target failure rate.The studied mechanisms set a lower bound on d for comparison with more complex designs.Source: §8 · p. 8
Benchmark and agent coverage
- Results cover three benchmarks and three agent backbones within smolagents, so the authors recommend domain-specific pilots rather than treating r and d estimates as universal.Disruption–recovery behavior may differ in other frameworks or task domains.Source: §8 · p. 8
Statistical power on positive results
- +2.8 pp (p = 0.014) on ALFWorld is directional evidence, near the ±4 pp detection limit at 80% power for 202 tasks × 3 seeds; high-success negative results are statistically clear.The ALFWorld gain is close to the detection limit, so it is not treated as a precise estimate.Source: §8 · p. 8
Pilot transfer
- Pilot transfer is validated only within-distribution, and cross-distribution estimates are untested and likely unreliable given model- and domain-dependence.The tested example uses an ALFWorld pilot to predict ALFWorld outcomes; QA-to-coding transfer is untested.Source: §8 · p. 8
9. Conclusion
Execution-time intervention is governed by a disruption–recovery tradeoff, and outcomes depend more on how agents respond to corrections than on failure-prediction accuracy alone. The conclusion recommends testing intervention with a small pilot rather than deploying it by default.
9. Conclusion
- 1–3 pp gains remain bounded: intervention degrades high-success regimes, while low-success regimes show positive effects without significant harm and pilots predict the shift.Full-scale evaluation confirms the positive effects in favorable regimes; an oracle LLM critic analysis indicates a low ceiling for critic-based intervention.Source: §9 · p. 8
- Expected improvement requires p > d/(r + d), where r and d are recovery and disruption rates; this criterion explains variation across domains and agents.The framework captures the tradeoff between recovering failing trajectories and disrupting trajectories that might otherwise succeed.Source: §9 · p. 8
- Evaluate intervention with a small pilot estimating r and d, avoid early-step intervention, and prefer post-hoc selection when disruption outweighs recovery (d/r > 1).These are the deployment guidelines drawn from the disruption–recovery framework.Source: §9 · p. 8
- Critic-based intervention is a model-dependent control problem, not merely a prediction task; better critic accuracy has limited upside unless agents incorporate corrections without destabilizing behavior.Meaningful risk remains outside high-failure regimes.Source: §9 · p. 8
Impact Statement … F. Intervention Cascades and No-Answer Failures
Execution-time interventions can harm successful trajectories as well as recover failing ones: cascades may exhaust an agent’s step budget, while pilot-based diagnostics can help identify when intervention is likely to help or harm.
Impact Statement
- High-accuracy critics can still degrade performance by up to 26 percentage points, and a 50-task pilot can help identify when intervention may help or harm.The framework can also help practitioners redirect compute toward alternatives such as post-hoc selection; it adds diagnostic tools, not new agent capabilities.Source: Impact Statement · p. 9
A. Per-Seed Results; A.1. HotPotQA; C. Statistical Testing and Power Analysis
- Appendix tables report per-seed accuracy, with means matching Table 4; significance testing uses paired task-level bootstrapping and Holm–Bonferroni correction.Tables 11–13 cover Qwen-3-8B, GLM-4.7, and MiniMax-M2.1 on HotPotQA.Source: A. Per-Seed Results · p. 11, Table 11 · p. 11, Table 12 · p. 11
B. Qualitative Trajectory Examples; B.1. Disruption Examples (Baseline Success →Intervention Failure)
- Disruptions included second-guessing a correct answer, eroding confidence, and derailing a correct reasoning path into a format error.The examples were selected to illustrate these mechanisms; among seed-42 cases, the authors observed 11 disruptions and 7 recoveries.Source: §B.1 · pp. 11–12, B. Qualitative Trajectory Examples · p. 11
B.2. Recovery Examples (Baseline Failure →Intervention Success)
- Intervention-triggered research corrected a wrong intuition about solo albums and retrieved the correct answer about the host of For One Night Only.Both examples began with a baseline failure and ended with a correct intervention answer.Source: §B.2 · p. 12
B.3. Cascade Example; B.4. Summary Statistics
- Repeated interventions after a correct initial answer exhausted the step budget and produced no output on the Roseanne/Scream 2 question.The accompanying summary statistics cover Qwen-3-8B on HotPotQA, seed 42, across 100 tasks.Source: §B.3 · p. 12, §B.4 · p. 12
D. Recovery and Disruption Accounting
- Recovery means a baseline failure becomes an intervention success, while disruption means a baseline success becomes an intervention failure; counting raw events preserves the same ordering and similar thresholds.Rates are computed per task using matched baseline and intervention runs from identical initial conditions.Source: D. Recovery and Disruption Accounting · p. 13
E. Early-Step Disruption Analysis
- Calibration helps mainly by suppressing early triggers, a minimum-step rule recovers 2–3 pp, and intervention harm concentrates in high-success regimes.For GLM-4.7 on ALFWorld, the 85.3% baseline failure rate exceeds the 74% threshold, and all conditions show positive or neutral effects.Source: E. Early-Step Disruption Analysis · pp. 13–14, Table 18 · p. 14
F. Intervention Cascades and No-Answer Failures
- On MiniMax / HotPotQA, the no-answer rate rises from 3.3% at baseline to 46.1% under intervention; cascades amplify harm when disruption exceeds recovery.When recovery exceeds disruption, cascades instead compound benefit.Source: F. Intervention Cascades and No-Answer… · p. 14
G. Model Sensitivity Analysis · H. Oracle Bounds and the Disruption Tax · I. Exploratory Analysis: LLM critic-Based Selection
Intervention effects vary sharply by model, and the oracle comparison reveals a disruption tax from mid-execution control. LLM critic-based trajectory selection is preliminary and requires larger-scale validation.
G. Model Sensitivity Analysis
- −26 pp on HotPotQA for MiniMax-M2.1 contrasts with GLM-4.7 effects of 0.0 pp to −3.7 pp.The same LLM critic and threshold can cause moderate or catastrophic harm depending on how calibration interacts with model-specific characteristics.Source: G. Model Sensitivity Analysis · pp. 14–15
- Calibration reduced LLM critic trigger rates differently across models; it helped GLM by making the critic less trigger-happy, while MiniMax received nearly every proposed intervention.Successful intervention requires controlling intervention rate, not just prediction accuracy.Source: G. Model Sensitivity Analysis · pp. 14–15
- MiniMax’s disruption-to-recovery ratio is 7.3:1, versus 1.5:1 for GLM; for every task MiniMax recovers, 7.3 are disrupted.The analysis attributes this difference to model-specific intervention rates and recovery rates.Source: G. Model Sensitivity Analysis · p. 14
- The no-answer rate on MiniMax/HotPotQA rose from 3.3% at baseline to 46.1% under intervention, a 14× increase.Rollback loops exhausted the agent’s step budget before it called a final answer, accounting for a substantial fraction of MiniMax’s performance collapse.Source: G. Model Sensitivity Analysis · p. 14
H. Oracle Bounds and the Disruption Tax
- 6–11 pp separates oracle selection from oracle intervention, quantifying an intrinsic disruption tax that prediction accuracy alone cannot eliminate.Oracle intervention acts only on trajectories that fail under baseline; oracle selection chooses the better of two completed trajectories (Best-of-2).Source: H. Oracle Bounds and the Disruption Tax · p. 15
I. Exploratory Analysis: LLM critic-Based Selection
- For contested tasks with one successful and one failed seed, the exploratory analysis selected the trajectory with the lower LLM critic score, but the results are preliminary.The analysis was intended to motivate future work rather than support confirmatory claims.Source: §I · p. 15
- +11 pp average oracle headroom motivates selection-based approaches; answer-only LLM judgment reached 51.7% ≈ random, and critic-based selection needs larger-scale validation.A properly powered study would need n ≥50 contested tasks for 80% power.Source: §I · p. 15
J. Boundary Condition Analysis: Non-Agentic Settings · K. Prompts
In single-shot code generation, where recovery is unavailable, interventions are neutral to marginally harmful as predicted by the disruption–recovery framework. The remaining prompts specify the feedback used and report calibration and intervention-analysis findings.
J. Boundary Condition Analysis: Non-Agentic Settings
- As r →0, the threshold p⋆ = d/(r + d) →1, predicting neutral-to-negative intervention effects regardless of baseline failure rate.The framework assumes sequential actions, mid-trajectory intervention, and possible recovery (r > 0); the test uses single-shot patch generation without iterative debugging or tool use.Source: J. Boundary Condition Analysis:… · p. 15
- In single-shot settings, attempts are statistically independent and provide no information for adaptation, unlike agentic rollback, which preserves partial progress.All observed regressions arose from interfering with trajectories already correct at steps 0–1.Source: J. Boundary Condition Analysis:… · p. 16, Table 22 · p. 16
- Neutral-to-negative effects where recovery assumptions are violated support the framework’s testable predictions within and at the boundary of its intended regime.The framework is scoped to multi-step agentic execution.Source: J. Boundary Condition Analysis:… · p. 16
- With 90 observations per condition, the study had 80% power to detect effects ≥12 pp, so Qwen’s observed effects are within the noise floor.The authors interpret them as consistent with a neutral effect, not as precise estimates.Source: J. Boundary Condition Analysis:… · p. 16
K.1. LLM Critic Feedback Prompt
- When the critic triggers under APPEND, the agent receives: “The LLM critic model predicts this action may lead to task failure. Please reconsider your approach.”Source: §K.1 · p. 16
K.2. Contextual LLM Feedback Prompt
- The contextual feedback template tells the agent to stop, notes predicted failure probability, and requests a different approach; GLM calibration reduces interventions by 71%, while MiniMax’s temperature of 1.05 provides almost no reduction.The template includes the probability as {failure_probability:.0%}; GLM’s fitted temperature is 8.81.Source: §K.2 · pp. 16–17, Table 24 · p. 17
- GLM recovers twice as often as MiniMax, post-hoc selection has more headroom than mid-execution intervention, and contested-task power is 23% for detecting a 15 pp difference.The contested-task analysis has n = 11 tasks.Source: Table 25 · p. 17, Table 26 · p. 17, Table 27 · p. 17
J. Boundary Condition Analysis: Non-Agentic Settings
- On SWE-bench Lite, Qwen’s effects were 0.0 pp (APPEND) and −2.2 pp (ROLLBACK), while MiniMax achieved 0% across conditions.The framework predicts intervention cannot help without recovery dynamics, even with Qwen’s high failure rate (p = 90%); the table caption describes the overall results as neutral to marginally negative.Source: J. Boundary Condition Analysis:… · p. 16, Table 28 · p. 17
Related papers
This paper cites
- SWE-bench: Can Language Models Resolve Real-World GitHub Issues?2023
- Self-Consistency Improves Chain of Thought Reasoning in Language Models2022
- Let's Verify Step by Step2023
- Self-Refine: Iterative Refinement with Self-Feedback2023
- HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering2018
- On Calibration of Modern Neural Networks2017