PaperlayerView exact revision on arXiv

Source-linked AI summary

ReflexGrad: Within-Episode Failure Recovery in LLM Agents via Progress-Gated Dual-Process Routing

Ankush Kadu, Aswanth Krishnan

arXiv:2511.14584v4cs.LGcs.AI

TL;DR

LLM agents can exhaust their step budgets after committing to a wrong approach, although the trajectory contains information that could help them escape. ReflexGrad combines fast policy refinement with stall-triggered causal replanning; on ALFWorld it reports 88.1% on GPT-5 and 75.4% on Qwen-3-8B without demonstrations, beating compute-matched 1-shot baselines.

  • Problem

    Agents can repeat a wrong approach until the step budget runs out, while existing methods do not act on the needed recovery information within a single episode.

    Source: §1 · p. 1

  • Method

    ReflexGrad routes between fast textual policy refinement and stall-triggered causal replanning, using a deterministic priority merge to keep policy updates coherent.

    Source: §1 · p. 1

  • Results

    On ALFWorld’s 134 tasks without demonstrations, ReflexGrad reaches 88.1% on GPT-5 and 75.4% on Qwen-3-8B, beating compute-matched 1-shot baselines at p<0.05.

    Source: §1 · p. 1, §10 · p. 9

  • Takeaways & Limitations

    ReflexGrad is most applicable when episodes show repeated low-progress streaks and a per-step progress signal is available.

    Source: §10 · p. 9

    When the corrective receptacle is unmapped, neither process recovers in time; this occurs in the Heat/Examine regime and contributes to the 5.2pp gap to 1-shot ReflAct.

    Source: §3.9 · p. 5

Abstract

from arXiv · show

We present ReflexGrad, a dual-process architecture for within-episode failure recovery in LLM agents without demonstrations. When agents commit to a wrong approach early and exhaust the step budget, the post-failure trajectory contains the information to escape -- but no published architecture acts on it within a single episode. ReflexGrad routes between a fast process (TextGrad-style continuous refinement every $k{=}3$ steps) and a slow process (Reflexion-style causal diagnosis when $m{=}5$ consecutive low-progress scores fire a routing gate). A deterministic priority merge keeps the natural-language policy coherent, and each slow activation emits three observable artifacts: a reproducible trigger, a causal diagnostic, and a verified fix. On ALFWorld 134 tasks, $n{=}10$ seeds, no demonstrations, ReflexGrad lifts Qwen-3-8B from $35.1\%$ to $75.4\%$ ($+40.3$pp), beating compute-matched 1-shot LATS by $+2.7$pp ($p{\approx}0.01$), ToT by $+5.7$pp ($p{<}10^{-4}$), and Self-Refine by $+6.7$pp ($p{<}10^{-5}$); on GPT-5 the lift is $46.3{\to}88.1\%$ ($+41.8$pp). The $1.5$pp cross-model difference is within seed noise ($p{\approx}0.13$), suggesting that the routing mechanism, rather than model scale, is the primary source of the gain. Code, prompts, per-seed logs, and sensitivity sweeps are released.

1. Introduction

ReflexGrad addresses within-episode recovery when agents repeat a wrong approach until their step budget runs out. It combines fast policy refinement with stall-triggered causal replanning and reports demo-free gains on ALFWorld.

1. Introduction

  • Agents can exhaust their step budgets after committing early to a wrong approach, despite the post-failure trajectory containing information needed to escape.The reported evaluation includes cross-model ablation, compute-matched baselines, sensitivity analysis, step-budget scaling, and cross-domain probes.Source: §1 · pp. 1–2
  • Existing approaches either defer strategic correction until the next trial, refine locally within the wrong strategy, or search actions without updating the policy.The paper identifies the missing mechanism as routing from tactical refinement to strategic correction within one episode.Source: §1 · p. 1
  • A fast process refines the policy every k=3 steps, while a slow process performs causal replanning after m=5 consecutive low-progress scores; a priority merge keeps updates coherent.Slow activations produce a reproducible trigger, causal diagnostic, and verified fix.Source: §1 · p. 1
  • 88.1% on GPT-5 and 75.4% on Qwen-3-8B on ALFWorld’s 134 tasks, with no demonstrations, beating compute-matched 1-shot LATS, ToT, and Self-Refine.The evaluation uses n=10 seeds.Source: §1 · p. 1
  • The 5.2pp residual gap to 1-shot ReflAct localizes to Heat and Examine tasks requiring world knowledge transferred by a demonstration; the authors do not claim parity.The introduction also reports a 1.5pp cross-model gain difference within seed noise (p ≈0.13).Source: §1 · p. 2

2. Background

The background contrasts methods that correct strategy across trials with methods that refine policy locally within a session. It frames demo-free, within-episode learning as the gap between them.

2. Background

  • Reflexion uses verbal self-critique followed by a full retry, but its correction takes effect only on the next trial and it requires demonstrations to bootstrap.The published method uses two few-shot demonstrations and reaches 97% on ALFWorld over 12 trials.Source: §2 · p. 2
  • TextGrad refines a natural-language policy from immediate output within a single session without demonstrations, but its local gradients can sharpen a wrong strategy.Its gradients adjust the next action rather than the strategy itself.Source: §2 · p. 2
  • The stated gap is a method that routes between both temporal scales within one episode and merges their policy updates without conflict.Reflexion provides strategic correction across trials, while TextGrad provides tactical correction inside the wrong strategy.Source: §2 · p. 2

3. Method

ReflexGrad combines local textual refinement with low-progress-triggered causal replanning, using a deterministic priority merge to keep policy updates coherent. Its method also bounds policy growth, partially substitutes for demonstrations, and specifies conditions under which slow replanning may help.

3.2. Routing rule

  • SLOW activates only after m consecutive scores fall below threshold; cooldown then suppresses both processes while the plan executes.The rule is designed to avoid activation from one noisy low score, and its sensitivity sweep’s worst setting reaches 84.3%.Source: §3.2 · p. 2

3.3. Fast process: per-step textual refinement

  • FAST computes a textual gradient every k=3 steps from recent trajectory context, providing local tactical correction without strategic reorientation.The gradient is a natural-language string proposing policy refinements.Source: §3.3 · pp. 2–3

3. Method

  • The architecture routes between fast textual refinement and stall-triggered causal replanning, with a priority merge maintaining a coherent natural-language policy.Slow activations produce a trigger, causal diagnostic, and plan as a proposed fix.Source: Figure 1 · p. 3

3.4. Slow process: stall-triggered causal replanning

  • SLOW uses causal reasoning over a longer trajectory to diagnose a root cause and generate a short corrective plan that persists through cooldown.The plan contains 1–3 high-level subgoals and replaces the next gradient update.Source: §3.4 · p. 3

3.5. Policy merge

  • The deterministic merge gives an active plan priority over a gradient, which takes priority over the base policy.The design discards lower-priority signals rather than averaging potentially contradictory natural-language instructions.Source: §3.5 · p. 3

3.6. Routing distribution

  • On ALFWorld, the routing rule sends approximately 85% of steps to FAST and 15% to SLOW; the fast share remains within 80–90% on TextWorld and OSWorld.The split is an outcome of the rule, not a tuned hyperparameter.Source: §3.6 · pp. 3, 5

3.7. Policy drift and bounded growth

  • Policy growth is bounded by a 10-step working-memory window and superseding updates, while slow plans replace accumulated gradient drift.Across 134 tasks on both models, policy size grows from approximately 150 tokens at step 1 to 380 at step 15, with a maximum observed size of 520; the architecture only partially substitutes for demonstrations when needed world knowledge is not exposed by the environment.Source: §3.7 · p. 5, §3.8 · p. 5

3.9. When the slow process helps: a conceptual analysis

  • The analysis proposes that slow replanning helps when local refinement has stalled but a global policy gap remains, while unmapped world knowledge can exceed the episode budget.The escape condition is sufficient but not necessary and cannot be proven without a metric on the policy space; observed synergy is +12.0pp on GPT-5 versus +6.8pp on Qwen-3-8B, while the Heat/Examine regime accounts for a 5.2pp gap to 1-shot ReflAct.Source: §3.9 · p. 5

4. Evaluation Protocol

The evaluation protocol addresses concerns that single-benchmark, single-model, single-seed results leave robustness and confounds unresolved. It pairs numeric claims with tests designed to isolate their confounds.

4. Evaluation Protocol

  • Single-number reports on a benchmark subset, model, and seed leave open whether gains are robust, model-dependent, demonstration-driven, or compute-driven.The paper’s Table 1 lists the confounds and the test addressing each.Source: §4 · p. 5
  • Every numeric claim is paired with an isolation test, and numbers without such a test are excluded.The stated example is a single accuracy on one model and one seed.Source: §4 · p. 5

5. Experimental Setup

The experiments use ALFWorld’s standard 134-task set for primary evaluation, with cross-model ablations and compute-matched comparisons designed to isolate common evaluation confounds.

5. Experimental Setup

  • Primary experiments use ALFWorld’s standard 134 tasks across six categories, enabling direct head-to-head comparison with prior methods.The paper prioritizes per-category, sensitivity, scaling, and failure analyses on this benchmark; cross-domain probes are reported separately.Source: §5 · p. 5
  • Table 1 maps evaluation confounds to the tests and sections that address them.The paper reports cross-model ablations, compute-matched baselines, sensitivity analysis, step-budget scaling, and cross-domain probes.Source: Table 1 · p. 6
  • GPT-5 supplies the primary results and Qwen-3-8B the cross-model ablation; ReflexGrad uses no demonstrations, while baselines use their standard 1-shot setup.Both models use default deterministic decoding, and reproducibility is supported by a released list of ten seeds.Source: §5 · p. 6
  • Headline results use 15 steps with fixed routing settings, and ten seeds report means and sample standard deviations.The gradient window is k=3, slow trigger m=5, low-progress threshold θlow=4, cooldown c=5, and working memory is 10.Source: §5 · p. 6
  • Compute-matched Self-Refine, Tree of Thoughts, and LATS baselines are re-implemented on Qwen-3-8B in their published 1-shot regime.The paper reports Welch’s t-tests for comparisons and notes that the 1.5pp cross-model gain difference is within seed noise (p≈0.13).Source: §5 · p. 6

6. Results

ReflexGrad improves ALFWorld performance across both evaluated models, with component synergy, compute-matched gains, and robustness across tested settings. Results also show category-specific limits and bounded evidence beyond ALFWorld.

6.1. Cross-model ablation

  • +41.8pp on GPT-5 and +40.3pp on Qwen-3-8B are the architectural gains over zero-shot, with the 1.5pp difference not statistically separated.Each model-specific gain is separated from its zero-shot baseline at p<10^-6; the cross-model difference has p≈0.13 and is interpreted as substantive equivalence, not identity.Source: §6.1 · p. 6, §6 · p. 6
  • +12.0pp synergy on GPT-5 and +6.8pp on Qwen-3-8B show super-additive gains from combining the fast and slow processes.TextGrad-only contributes +23.1pp on GPT-5 and +26.1pp on Qwen-3-8B; lower synergy on Qwen-3-8B reflects less remaining recoverable error.Source: §6.1 · p. 6

6.2. Compute-matched comparison

  • +2.7pp over 1-shot LATS at about 30% lower compute, +5.7pp over 1-shot Tree of Thoughts, and +6.7pp over 1-shot Self-Refine favor demo-free ReflexGrad.The comparison is conservative: baselines use their favorable 1-shot setup, while ReflexGrad uses no demonstrations; it does not match ReflAct’s 80.6%.Source: §6.2 · p. 7, Table 3 · p. 7

6.3. Per-category breakdown

  • Qwen-3-8B reaches 90.9% on Pick & Place, 88.5% on Clean & Place, 81.8% on Pick Two, and 77.3% on Cool & Place.Heat and Examine are lower because the architecture does not supply the tacit verb-receptacle knowledge a demonstration transfers directly.Source: §6.3 · p. 7

6.4. Sensitivity to routing thresholds

  • 84.3% is the worst observed result across the routing-threshold sweep, and the tested settings vary by at most 3.8pp.Defaults were fixed through separate exploratory piloting rather than selected by the sweep.Source: Table 4 · p. 7, §6.4 · p. 7

6.5. Scaling with per-episode step budget

  • +20.1pp from 5 to 10 steps and +14.2pp from 10 to 20 steps show diminishing returns, with only +2.2pp from 15 to 20.The roughly 90% ceiling at 20 steps reflects residual failures dominated by missing world knowledge rather than insufficient recovery steps.Source: §6.5 · p. 7, Table 5 · p. 7

6.6. Cost efficiency

  • ReflexGrad has the lowest cost per percentage point gained despite using more total compute.The reported explanation is that super-additive synergy concentrates extra calls on recoverable failures.Source: §6.6 · p. 7, Table 6 · p. 8

6.7. Cross-domain evidence (limited)

  • On TextWorld, ReflexGrad reaches 89% ± 2.2 versus zero-shot 56% ± 3.1, while OSWorld rises from 14/20 to 16/20.These preliminary probes are applicability evidence; ALFWorld is the only fully validated benchmark, and OSWorld is not a benchmark-wide statistical claim.Source: §6.7 · p. 7

7. Failure Analysis

The failure analysis groups nearly all 33 observed Qwen3-8B failures into world-knowledge, receptacle-navigation, and evaluator-hallucination cases.

7. Failure Analysis

  • 21/33 failures involve missing world knowledge, 8/33 receptacle navigation, and 4/33 evaluator hallucination.The first two are described as recoverable in principle with longer budgets or retrieval; sustained evaluator false positives account for the remaining cases.Source: §7 · p. 8

8. Limitations

The paper limits its strongest claims to demo-free, within-episode evaluation and identifies boundaries in benchmark breadth, reproducibility, and future diagnostics.

8. Limitations

  • The architecture establishes a demo-free, within-episode regime distinct from methods that use demonstrations to transfer tacit world knowledge.The two regimes are described as complementary rather than directly comparable on a single accuracy axis.Source: §8 · p. 8
  • TextWorld and OSWorld results are applicability evidence, not statistical claims; ALFWorld is the only fully validated benchmark.WebShop and Mind2Web are identified as natural next benchmarks for fuller statistical comparison.Source: §8 · p. 8
  • GPT-5 is closed, while Qwen-3-8B is reproducible from the released checkpoint and is the falsifiable contribution.The paper also names predictive diagnostics and additional interactive benchmarks as future work.Source: §8 · p. 8

9. Related Work

ReflexGrad is positioned among methods for reflection, prompt optimization, search, and adaptive planning by its within-episode, progress-gated routing. The authors describe its full combination of design properties as unoccupied by prior or concurrent work, while noting nearby approaches and comparisons.

Reflection and self-critique

  • Reflexion and ReflAct critique across trials, while ReflexGrad uses Reflexion-style diagnosis within one episode; Self-Refine lacks a persistent policy update and ExpeL learns across episodes.CRITIC uses tool-interactive critiquing, while ReflexGrad operates without cross-episode learning.Source: §9 · p. 8

Prompt optimization

  • TextGrad, DSPy, and OPRO optimize natural-language prompts through textual gradients; ReflexGrad applies this mechanism every k=3 steps within an episode.The paper characterizes this as a fast tactical loop.Source: §9 · p. 8

Inference-time search and adaptive planning

  • ToT and LATS explore reasoning paths without updating the policy, while AdaPlanner switches on binary execution outcomes rather than consecutive low progress scores.Voyager builds a skill library across episodes but does not perform within-episode recovery.Source: §9 · p. 8

Dual-process and failure recovery

  • Concurrent works address dual-process routing, failure recovery, post-hoc repair, trained routing, or progress-gated slow processes, but the authors identify no work at ReflexGrad’s full design intersection.The stated intersection is training-free, within-episode, progress-gated, dual-process, single-model, and demo-free.Source: §9 · pp. 8–9

Experience retrieval

  • ExpRAG reports 83.6% on ALFWorld via trajectory retrieval, approaching the GPT-5 result and motivating comparison with dual-process routing at comparable compute.The paper leaves that comparison to future work.Source: §9 · pp. 8–9

10. Conclusion

ReflexGrad demonstrates within-episode failure recovery without demonstrations, reaching 88.1% on GPT-5 and 75.4% on Qwen-3-8B on ALFWorld 134 tasks. The accompanying sections document baseline implementations, per-seed counts, worked recovery traces, and prompt templates.

10. Conclusion

  • 88.1% on GPT-5 and 75.4% on Qwen-3-8B were achieved on ALFWorld 134 tasks without demonstrations, beating compute-matched 1-shot baselines at p<0.05.The conclusion identifies a progress-gated router combining per-step refinement with stall-triggered causal replanning.Source: §10 · p. 9

B. Implementation details for compute-matched baselines

  • The compute-matched baselines use ALFWorld adaptations of Self-Refine, ToT, and LATS, with one benchmark demonstration and published call budgets or defaults.Implementations were sanity-checked on each method’s original benchmark; their ALFWorld numbers are author-implemented.Source: B. Implementation details for… · p. 10

C. Per-seed counts

  • Across representative seeds, GPT-5 ReflexGrad averages 118 solved tasks (88.1%), while Qwen-3-8B averages 101 (75.4%).The runs cover 134 ALFWorld tasks and use no demonstrations.Source: C. Per-seed counts · p. 10

D. Failure mode trace

  • After four low-progress scores, the slow process diagnoses the stove–microwave mismatch; the agent follows its corrective plan and completes the heating task.The trace reports subsequent progress scores of {6, 7, 8, 9} before task completion.Source: D. Failure mode trace · p. 11

E. Full LLM prompts (reproducibility)

  • The prompt appendix documents seven-call slow-process activations, including causal diagnosis and plan generation, plus evaluator and cooldown specifications.Cooldown is deterministic and suppresses the fast process for five steps.Source: E. Full LLM prompts (reproducibility) · pp. 11–12

F. Extended worked traces

  • Worked traces show fast-process-only success in six steps, but also a slow-process diagnosis that cannot be executed before the step budget expires.The latter failure involved an unmapped desk-lamp location and is described as a receptacle-navigation failure.Source: F. Extended worked traces · p. 12

G. Statistical analysis details

The appendix reports statistical comparisons, sensitivity and implementation details, benchmark probes, design choices, and limits on the evaluator’s role. Together, these details qualify the evidence for ReflexGrad’s gains and describe where its routing approach may apply.

G. Statistical analysis details

  • The 1.5pp difference between GPT-5’s +41.8pp and Qwen-3-8B’s +40.3pp gains is not separated (p ≈ 0.13), supporting substantive equivalence rather than identity.Reported standard deviations are sample standard deviations with n=10 seeds, using n−1 (Bessel’s correction).Source: G. Statistical analysis details · p. 12
  • ReflexGrad’s 75.4 ± 2.2 versus LATS’s 72.7 ± 2.0 is a 2.7pp gap (p ≈ 0.010); its gaps over ToT and Self-Refine are 5.7pp and 6.7pp.The comparisons use Welch’s two-sample t-test and are separated at p<0.05.Source: G. Statistical analysis details · p. 12

H. Hyperparameter sensitivity full sweeps

I. Compute and reproducibility

  • Code, prompts, per-task logs, and per-seed counts are released; episodic memory resets at episode boundaries, with failure-recovery triples retained as released artifacts.The memory subsystem has no learned retrieval network or cross-episode episodic persistence.Source: §I · p. 13, J. Memory subsystem overview · p. 13

L. OSWorld per-category breakdown

M. Comparative method matrix

  • Only ReflexGrad fills the within-episode, strategic-recovery, and demo-free combination in the method matrix.The matrix distinguishes these dimensions from cross-trial learning, compute model, and primary supervision signal.Source: M. Comparative method matrix · p. 14, Table 10 · p. 15

N. Hyperparameter rationale and design alternatives

P. Evaluator analysis: is E a hidden demonstration?

  • Evaluator pretraining may include ALFWorld semantics, so the demo-free claim is not unconditional; the reported architectural lift is measured on top of what the evaluator provides.The similar gains across GPT-5 and Qwen-3-8B suggest the lift is not driven primarily by frontier-model evaluator knowledge.Source: P. Evaluator analysis: is E a hidden… · pp. 16–17

Q. Reproducibility checklist

The checklist records resources and settings needed to reproduce the experiments, then outlines conditions under which ReflexGrad-style routing may or may not help. It also gives expected gains and costs from the reported decomposition.

Q. Reproducibility checklist

  • The code, evaluator, compute-matched baselines, task identifiers, and fixed 10-seed list are released; ALFWorld evaluation uses the standard out-of-distribution split without training data.The released repository includes per-seed integer counts and the experiment’s code and prompts.Source: Q. Reproducibility checklist · p. 17
  • Three quantified Qwen-3-8B failure modes are world knowledge (21/33), receptacle navigation (8/33), and evaluator hallucination (4/33).The checklist points to concrete traces in the appendices.Source: Q. Reproducibility checklist · p. 17

R. Practitioner decision checklist

  • The proposed fit requires repeated low-progress streaks, a natural-language policy, a cheap reliable progress signal, and strategic alternatives that could rescue stalled behavior.The checklist specifies that at least 10% of failed episodes should contain three or more consecutive stalled steps with minor retries.Source: R. Practitioner decision checklist · pp. 17–18
  • Routing may be a poor fit for 1–2-step tasks, unreliable progress signals around ηfp ≳ 0.10, or tasks where strategic recovery and tactical refinement are the same operation.It is also unlikely to help when recovery requires missing world knowledge that cannot be inferred from trajectories or pretraining.Source: R. Practitioner decision checklist · p. 18
  • On suitable text-game benchmarks, fast refinement adds about 23–26pp; slow diagnosis adds a further 7–12pp at about five extra LLM calls per activation.At the empirical 15% activation rate, the slow process adds about 0.75 calls per episode.Source: R. Practitioner decision checklist · p. 18
  • The combined system’s cost-per-percentage-point is about 1.4 calls/pp, favorable relative to either component alone.The checklist frames the expected synergy gain as likely at least 5pp when its stated conditions hold.Source: R. Practitioner decision checklist · pp. 17–18

Related papers

Loading 2511.14584v4…