PaperlayerView exact revision on arXiv

Source-linked AI summary

My FAULT: Self-Diagnosis as Credit Assignment in Self-Evolving Agentic Reinforcement Learning

Yihua Zhu, Qianying Liu, Weixu Qiao, Xuan Ren, Weiwei Xu, Wenbo Li, Wei Wang, Ruijia Chen, Xinmiao Luan, Yin Luo, Hao Huang, Xiang Zheng, Hidetoshi Shimodaira

arXiv:2610.01161v1cs.CL

TL;DR

Terminal-reward agentic RL provides no outcome-driven signal for same-outcome rollout groups and little guidance about which decisions caused failure, while natural-language diagnoses need verification and quantitative valuation. FAULT verifies self-diagnosed errors, learns their relative costs from outcomes, and redistributes bounded credit across steps as its policy, diagnoser, and prices co-evolve. Across two model scales, it leads on longer-horizon ALFWorld and WebShop while remaining competitive on shorter-horizon Search-based QA; on ALFWorld, signal coverage reaches 95%, versus 41% for GRPO and 72% for GiGPO.

  • Problem

    Terminal-reward RL can provide no outcome-driven signal for same-outcome rollout groups and does not identify which intermediate decisions caused failure, while natural-language diagnoses need verification and quantitative valuation.

    Source: §1 · p. 2

  • Method

    FAULT verifies self-diagnosed errors, learns their relative costs from terminal outcomes, and redistributes a bounded penalty budget across diagnosed steps while the policy and diagnoser co-evolve.

    Source: §1 · p. 2, §2.4 · p. 6

  • Results

    Across two model scales, FAULT leads on ALFWorld success rate and WebShop task score while remaining competitive on Search-based QA; ALFWorld signal coverage is 95%, versus 41% for GRPO and 72% for GiGPO.

    Source: §5 · pp. 11–12

  • Takeaways & Limitations

    By applying learned error costs even when rollouts share a terminal outcome, FAULT can distinguish trajectories and concentrate learning signal on steps needing correction.

    Source: §1 · p. 2

    The initial self-diagnoser’s dominant-error step accuracy is 0.48 under a ±2-step tolerance, below the 0.60 threshold, and its off-table rate is 0.31, above the 0.10 limit.

    Source: §3.5 · p. 11

Abstract

from arXiv · show

Agentic reinforcement learning (RL) has emerged as a powerful approach for training large language model agents on multi-step tasks, yet reliance on terminal outcome rewards creates two credit-assignment problems, particularly in long-horizon tasks. First, same-outcome rollout groups provide no learning signal from terminal rewards. Second, terminal rewards provide only trajectory-wide feedback, making it difficult to identify which decisions caused a failure. Recent work supplements terminal rewards with finer-grained information from trajectory analysis, such as natural-language reflections on intermediate decisions and errors. However, natural-language diagnoses are difficult to use directly for credit assignment: their error claims may be unreliable, and they do not quantify how much each error should affect learning. We propose Self-Diagnosis-guided Terminal Credit Redistribution (FAULT), which turns diagnosed errors into explicit step-level credit anchored by terminal outcomes. FAULT checks diagnostic evidence and learns relative error costs from task outcomes. During training, the policy and self-diagnoser co-evolve, while error costs are updated online from recent outcomes. On ALFWorld, FAULT recovers learning signals from same-outcome groups, reaching 95% signal coverage versus 41% for GRPO and 72% for GiGPO, while better localizing credit to specific error steps. Across two model scales, FAULT delivers strong. improvements on the long-horizon ALFWorld and WebShop tasks while remaining competitive on short-horizon Search-based QA.

1 Introduction

Terminal-outcome rewards leave long-horizon agentic RL with no learning signal from same-outcome rollout groups and little guidance about which decisions caused failure. FAULT verifies self-diagnosed errors and prices them from outcomes to redistribute credit across steps, improving signal coverage, error localization, and results on longer-horizon tasks.

Problem

  • Group-relative RL loses outcome-driven signal when every rollout has the same terminal outcome, and its rollout-level advantage does not identify which steps caused failure.Same-outcome groups accounted for 59–64% of preliminary ALFWorld training groups.Source: §1 · p. 2
  • Natural-language trajectory analyses identify intermediate errors but remain descriptive, and their claims need verification before they can provide quantitative step-level credit.Useful credit also requires calibrating the relative importance of different errors from outcomes.Source: §1 · p. 2

Approach

  • FAULT verifies evidence for self-diagnosed errors, learns relative error costs from terminal outcomes, and redistributes a bounded penalty budget across diagnosed steps.It separates error localization from valuation, allowing costs to distinguish trajectories even when their terminal outcomes match.Source: §1 · p. 2

Results

  • Across two model scales, FAULT leads on ALFWorld success rate and WebShop task score while remaining competitive on shorter-horizon Search-based QA.Figure 1 also reports that FAULT outperforms raw-backbone GRPO and GiGPO on all three tasks.Source: Figure 1 · p. 3, §1 · p. 3
  • Decisive-step MRR is 0.494 for FAULT, versus 0.303 for GiGPO and 0.266 for random ranking.A blind LLM judge identifies the decisive error step in each failed trajectory; methods rank steps by their penalties.Source: Figure 1 · p. 3, §1 · p. 3
  • On ALFWorld, usable signal coverage is 95% for FAULT, 41% for GRPO, and 72% for GiGPO.FAULT uses every all-fail group and most all-success groups, whereas GiGPO uses no all-fail groups.Source: §1 · p. 2, Figure 1 · p. 3

2 Method

FAULT initializes a shared actor and diagnoser, then uses verified diagnoses and outcome-fitted error prices to redistribute bounded penalties across steps during RL. The actor, diagnoser, and prices co-evolve through shared updates and online refitting.

2 Method

  • FAULT combines a shared actor and diagnoser with terminal-outcome error prices to assign explicit step-level credit.The method begins with diagnosis SFT and initial price learning, then diagnoses on-policy rollouts, prices their errors, and redistributes a bounded penalty across diagnosed steps.Source: §2 · p. 3

2.1 Problem Formulation

  • Terminal outcomes are the guaranteed-correct judgment, so FAULT preserves outcomes and redistributes only their credit across steps.The task formulation samples groups of rollouts from a frozen behavior policy and gives each trajectory a terminal reward.Source: §2.1 · p. 3

2.2 Cold Start: Verifiable Diagnosis and Diagnosis SFT

  • A one-time cold start fixes an error taxonomy, trains the shared actor-diagnoser on verified teacher diagnoses, and initializes error prices; no teacher is used afterward.Diagnosis records include an error class, step, and quote, and reports are checked against the cited trajectory step.Source: §2.2 · pp. 4–5

2.3 Terminal-Anchored Error Pricing

  • FAULT estimates each error class’s relative cost by regressing terminal outcomes on verified error rates, comparing successes and failures within each task.Prices are periodically refitted on a sliding window of recent RL rollouts, with bounded optimization, a small-sample correction, and shrinkage toward an anchor.Source: §2.3 · p. 5

2.4 Self-Evolving RL with Step-Level Credit Redistribution

  • For each rollout group, fixed error prices set a bounded trajectory penalty that is normalized across diagnosed steps using their localization weights.The cap keeps successful trajectories’ episode scores above failed trajectories’; using error rates rather than counts avoids rewarding shorter trajectories solely for accumulating fewer errors.Source: §2.4 · p. 5
  • The actor and diagnoser share weights, but diagnosis tokens receive no policy gradient; verified rollout records enter a sliding window for online price refits.FAULT combines task-wide episode comparisons with comparisons among samples from similar pre-response contexts, applying the same step penalty in both channels.Source: §2.4 · p. 6

3 Experiments

The experiments evaluate FAULT on three benchmarks at two model scales, then analyze benchmark performance, learning-signal coverage, step-level credit, trajectory length, ablations, and diagnostic quality. FAULT leads on longer-horizon ALFWorld and WebShop, while remaining competitive on shorter-horizon Search-based QA.

3.1 Experimental Setting

  • The evaluation covers ALFWorld, WebShop, and seven Search-based QA datasets, using Qwen3 1.7B and 4B backbones, with task-specific evaluation metrics and training setups.Baselines include raw-backbone GRPO and GiGPO, SEED, and diagnosis-SFT-initialized GRPO and GiGPO at 4B.Source: §3.1 · pp. 6–7

3.2 Main Results

  • Across both scales, FAULT leads ALFWorld success and WebShop task score; gains over raw-initialized GRPO/GiGPO span 17.1–41.7, 8.7–21.0, and 3.0–5.1 points on ALFWorld, WebShop, and Search-based QA averages.At 1.7B it ties SEED at 38.9 on Search-based QA; at 4B it scores 43.8, 0.6 points below SEED.Source: §3.2 · p. 7, Table 1 · p. 8

3.3.1 Learning Signal in Same-Outcome Groups

  • Usable signal appears in 41.4% of GRPO, 72.5% of GiGPO, and 95.1% of FAULT groups; FAULT’s pooled signal-retention Index is 0.821 versus about 0.42 for both baselines.The Index compares budget-gap and advantage-gap contrast proxies, not a common-scale estimate of optimizer credit; FAULT’s same-outcome contribution is 0.449 versus 0.063 for GiGPO.Source: §3.3.1 · pp. 7–8, Figure 3 · p. 8

3.3.2 Step-Level Localization of Credit

  • FAULT ranks the judge’s decisive error step at MRR 0.494, versus 0.303 for GiGPO and 0.266 for random ranking, and concentrates credit more strongly on same-outcome groups.Mean normalized entropy is 0.29 for FAULT, 0.72 for GiGPO, and 0.91 for GRPO; 28% of FAULT’s penalized trajectories put the whole penalty on one step.Source: §3.3.2 · pp. 9–10, Figure 4 · p. 9

3.3.3 Training Trajectory Length

  • Across updates 151–160, FAULT averages 12.0 active steps per ALFWorld episode, compared with 13.8 for SEED and about 22 for GRPO near the end.Failures can run to the 30-step limit, and revisits to searched locations recede faster under FAULT than under GRPO.Source: Figure 5 · p. 9, §3.3.3 · p. 10

3.4 Ablation Studies

  • On ALFWorld, full FAULT reaches 91.0% success, exceeding fixed prices/no cap (89.6%), constant λ (79.1%), fixed judge (82.8%), and earlier GRPO-based diagnostic shaping (77.6%).The recorded variants favor the full design but do not isolate localization, pricing, or the judge one factor at a time; some runs are not matched controls.Source: §3.4 · p. 10, Table 2 · p. 10

3.5 Diagnostic Quality Before and During RL

  • During RL, L1 error-slot recall rises from 0.66 to 0.71 and off-table hallucination falls from 0.32 to 0.16, while L2 macro-F1 decreases slightly by 0.015.The assessment uses single-seed training rollouts and a fixed teacher reference; initial held-out step localization and off-table predictions miss their stated thresholds.Source: §3.5 · pp. 10–11, Figure 6 · p. 11

4 Related Work

Prior agentic RL methods derive credit from terminal outcomes, while process-based methods seek finer-grained feedback. FAULT combines verified diagnoses with outcome-fitted error costs to assign bounded step-level credit.

Outcome-based agentic RL

  • Outcome-based agentic RL uses terminal rewards, but same-outcome groups provide no outcome-driven gradient.GRPO broadcasts each rollout’s group-normalized outcome advantage to all its steps.Source: §4 · p. 11

Process-based agentic RL

  • Process-based methods add step rewards or trajectory analyses, but language diagnoses remain unchecked and their weights are not learned from outcomes.Learned step rewards, rollouts, hindsight, and natural-language analysis are among the approaches discussed.Source: §4 · p. 11
  • FAULT retains terminal rewards while using verified diagnoses, outcome-fitted prices, and bounded step credit.This approach addresses the stated gaps in checking diagnostic claims and learning their outcome-related weight.Source: §4 · p. 11

5 Conclusion

FAULT assigns step-level credit using checked diagnoses and relative error costs learned from terminal outcomes. It improves ALFWorld and WebShop results across two model scales while remaining competitive on shorter-horizon Search-based QA.

5 Conclusion

  • FAULT reaches 95% usable signal coverage on ALFWorld versus 41% for GRPO and 72% for GiGPO, with decisive-step MRR of 0.494 versus GiGPO’s 0.303.At both model scales, FAULT leads in ALFWorld success rate and WebShop task score while remaining competitive on shorter-horizon Search-based QA.Source: §5 · pp. 11–12

AI use statement

  • Generative AI assisted method and experiment design, implementation, proofs, writing, and several method components, with authors reviewing AI-assisted content and testing code.AI also generated cold-start teacher diagnoses, Search-based QA cold-start trajectories, and decisive-step labels for localization analysis.Source: AI use statement · p. 12

Ethics statement

  • The study uses established public benchmarks and did not recruit human participants or collect private user data.Its stated goal is to make credit assignment more transparent by linking step-level signals to diagnosed errors and task outcomes.Source: Ethics statement · p. 12

Reproducibility statement

  • The appendices specify the method, estimator, penalty schedule, training loop, benchmarks, baselines, protocols, initialization, and hyperparameters.The diagnosis prompt is in Appendix C.5, other prompts are released with the code, and the listed benchmarks and base models are public.Source: Reproducibility statement · p. 12

A Notation

  • Notation conventions distinguish vectors, sets, indicators, clipping, and standard deviation, while Table 3 lists symbols and where they are introduced.Superscripts mark versions, roles, or stages rather than powers.Source: A Notation · p. 18, Table 3 · p. 18

B Method Details

This section details FAULT’s error-diagnosis and credit-redistribution procedure, including advantage-centering, token updates, and its four-stage training loop. It also describes the expanded experimental setup, benchmarks, and baselines.

B.1 Cold Start: Taxonomy, Diagnosis SFT and Initial Prices

  • Cold start freezes an L1/L2 error taxonomy, fine-tunes a diagnosis model on structured teacher reports, and separately fits initial prices from verified diagnoses and terminal outcomes.The teacher diagnoses trajectories once during cold start; diagnosis targets contain no prices, scores, or advantages.Source: §B.1 · p. 19

B.2 Online Self-Diagnosis, Verification and Aggregation

  • Before each actor update, the unchanged rollout policy samples multiple JSON diagnoses per trajectory; entries are verified against taxonomy classes, executed steps, and verbatim quotes.Free-form labels outside the frozen taxonomy are excluded from scoring.Source: §B.2 · p. 19
  • Trajectories pass the pricing gate only with parsable diagnoses and majority-correct outcome reports; verified error counts are averaged across diagnoses to form normalized pricing records.Failed-gate trajectories are excluded from price fitting and receive an imputed score from same-outcome batch trajectories.Source: §B.2 · pp. 19–20

B.3 Terminal-Anchored Error Pricing

  • Binary pricing conditions on within-task success counts to remove task intercepts, so all-success or all-failure groups contribute no information; graded outcomes instead use ridge regression.Binary estimation uses regularization, including a Firth correction and shrinkage toward a reference; indistinguishable error classes remain unidentifiable.Source: §B.3.1 · p. 20, §B.3.2 · p. 21

B.3 From Coefficients to Prices; B.3.5 Online Refits and Price Updates

  • Negative fitted coefficients set relative error costs, with positive coefficients assigned no reward; online refits use recent verified records and smooth accepted coefficients before new prices take effect.The smoothing weights the previous coefficient 0.7 and the new fit 0.3; unfitted classes or positive new coefficients retain their previous values.Source: §B.3.3 · pp. 21–22, §B.3.5 · p. 22

B.4 Score, Penalty Schedule and Order-Preserving Budget

  • The trajectory score adds a first-error-class boost, while a scheduled, capped penalty converts the score into a bounded budget without reversing binary outcome order when the reward gap exceeds the cap.The first-error score factor is 1.5; the diagnostic penalty vanishes at the final update.Source: §B.4.1 · p. 23, §B.4.2 · p. 23, §B.4.3 · p. 24

B.4.3 Bounded Budget and Order Preservation

  • A penalty cap preserves successful-over-failed episode ordering whenever the original reward gap exceeds the cap, within the diagnostic episode channel.The proposition does not cover additional reward terms; for graded outcomes, the same argument gives a corresponding bound.Source: §B.4.3 · pp. 23–24

B.5 Conserved Step Redistribution

  • Verified error reports weight cited steps by class price, with an earliest positive-priced step boost; normalized allocation preserves the total budget and assigns imputed budgets to the last active step.Repeated reports count separately, while quote length and severity hints do not affect localization weights.Source: §B.5.1 · p. 24, §B.5.2 · p. 24

B.6.2 Centred Advantages and the Effect of Diagnosis

  • A diagnosis penalty lowers a step’s advantage only when it exceeds its comparison group’s mean; a uniform within-group penalty cancels.The episode and context-anchor channels compare the same local cost against different reference sets, rather than assigning separate budgets.Source: §B.6.2 · p. 25

B.6.3 Token-Level Policy Update

  • The policy loss applies the same joint advantage to each response token and excludes context, feedback, and diagnosis tokens.The loss averages over response tokens, with the old policy and advantages held fixed during the update.Source: §B.6.3 · p. 25

B.7 Algorithm

  • FAULT’s RL stage proceeds through rollout and diagnosis, bounded penalty allocation, dual-channel policy updating, and online price refitting.It verifies diagnoses, imputes remaining scores from same-outcome non-imputed scores, allocates capped penalties, and refits prices from recent non-imputed records when checks pass.Source: §B.7 · p. 26

C Additional Experimental Setup

C.1 Benchmarks and Baselines

  • ALFWorld evaluates unseen household-task games, WebShop tests product search through purchase, and the baselines distinguish vanilla, GRPO, GiGPO, SEED, and diagnosis-SFT-started FAULT.The Qwen3-4B GRPO and GiGPO diagnosis-SFT controls share FAULT’s checkpoint while disabling diagnostic shaping and retaining diagnosis monitoring.Source: §C.1 · p. 27

C.2 Evaluation Protocols

Evaluation uses benchmark-specific protocols and cold-start data to initialize diagnosis, taxonomy, and error pricing before RL. Reported metrics and sampling schemes differ by benchmark.

C.2 Evaluation Protocols

  • Each benchmark is evaluated with its own protocol; reported metrics are percentages, and each result is a single-run point estimate, not a significance claim.ALFWorld averages three greedy evaluations per task across 134 unseen tasks; WebShop uses 256 episodes and reports graded score and strict purchase success; Search-based QA reports per-dataset exact match and their unweighted mean.Source: §C.2 · p. 27

C.3 Cold-Start Data and Initialization

  • Cold-start preparation fixes a benchmark-specific error taxonomy, trains diagnosis-SFT data, and estimates initial error prices before RL.The ALFWorld taxonomy contains eight L1 families and 25 L2 mechanisms, with 20 pricing classes retained; its initial fit uses 99 informative groups.Source: §C.3 · p. 27, §C.3.1 · pp. 27–28

C.3.1 ALFWorld

  • ALFWorld cold-start data comprise 2,880 teacher-diagnosed trajectories from 180 tasks, with 200 held out for assessment and 2,680 used for diagnosis-SFT.The eight rollouts per task are sampled from the base model and a checkpoint after 20 GRPO updates.Source: §C.3.1 · p. 27

C.3.2 WebShop

  • WebShop uses its own taxonomy and diagnosis-SFT data, a within-group ridge estimator for graded training reward, and model-size-specific penalty strengths.The recorded ALFWorld cold-start collection description also notes up to 4,320 trajectories across selected early checkpoints.Source: §C.3.2 · p. 28

C.3.3 Search-Based QA

  • Search-based QA uses a benchmark-specific taxonomy and SFT data combining diagnosis examples with task responses, including successful search trajectories.The SFT sets contain 6,974 examples for 1.7B and 6,498 for 4B.Source: §C.3.3 · p. 28

C.4 Training Configuration and Hyper-parameters

FAULT structures diagnoses around a frozen error table and measures how diagnostic scores and penalties behave across rollout groups. The analyses find usable contrast in same-outcome groups while qualifying what those measurements establish.

C.5 Diagnosis Prompt

  • The frozen-table diagnosis prompt requires each reported error to cite a table id, 1-based step, and verbatim quote, and it supplies task, outcome, and rendered trajectory context.Only the environment name and description and table text depend on the benchmark.Source: Figure 7 · p. 28

D.1.1 Coverage and Relative Signal Strength

  • Coverage measures available diagnostic-score contrast, whereas replacing scores with injected budgets reduces FAULT’s run-wide coverage from 95.1% to 84.6%.In the last 20 updates, only 20% of groups retain a budget range above threshold as λu approaches zero.Source: §D.1.1 · p. 30, §D.1.3 · p. 32
  • The same-outcome group index combines each group type’s frequency with its contrast relative to mixed groups, so mixed groups contribute exactly their frequency.Because FAULT and baselines use different credit quantities, the index compares contrast proxies rather than optimizer credit on a common scale.Source: §D.1.1 · p. 31

D.1.2 Credit under Equal Terminal Rewards

  • With equal terminal rewards, GRPO and GiGPO’s episode channel have zero outcome advantage, while FAULT can retain diagnostic contrast unless its step penalties are constant.The proposition isolates terminal rewards and diagnostic penalties; equal outcomes alone do not guarantee diagnostic contrast or semantic correctness.Source: §D.1.2 · pp. 31–32

D.1.3 Observed Coverage and Strength

  • At τ = 0.05, FAULT covers every all-fail group and 90% of all-success groups, while all three methods cover every mixed group.These are empirical coverage results, not guarantees that different diagnoses produce contrast.Source: §D.1.1 · p. 31, §D.1.3 · p. 32
  • The pooled signal-retention Index for FAULT is 0.821, but it is a budget-based contrast proxy rather than a measured optimizer-advantage index or proven bound.Its interpretation differs from baselines because FAULT uses trajectory budgets while baselines use step advantages.Source: §D.1.1 · p. 31, §D.1.3 · p. 32

D.2 Penalties on Successful Trajectories

  • Across training, 61.85% of successful trajectories receive a charge above 0.001 reward units, and failures have higher median budgets in every window.The cap binds on 56 of 20,480 trajectories (0.27%, all successes) and never removes the entire diagnostic-score contrast of a complete group.Source: §D.2 · p. 33, Figure 10 · p. 34

D.3.1 Evaluation Protocol

  • Localization evaluation uses failed trajectories judged by a blind LLM that selects the earliest decisive step, but those labels are model judgments rather than human- or counterfactually verified labels.A blind relabel of 48 trajectories agrees within one step in 75% of cases.Source: §D.3.1 · p. 34

D.3.2 Localization Metrics and Results

FAULT’s negative credit is more concentrated near judged decisive steps and format-valid no-ops than baseline credit, while concentration and targeting measures have important interpretive limits.

D.3.2 Localization Metrics and Results

  • FAULT achieves MRR 0.494 and hit@1 34.1%, versus GiGPO’s 0.303 and 11.9%; GRPO’s MRR 0.306 is close to random ranking’s 0.266.FAULT has 82 exact step matches and GiGPO 21; the evaluation uses blind model judgments, not human or counterfactually verified labels.Source: §D.3.2 · p. 34, Figure 11 · p. 36

D.3.3 Concentration of Step Credit

  • In all-fail groups, mean ENS falls from 11.3 to 3.3 for FAULT, compared with 26.5 to 15.6 for GRPO and 24.0 to 11.3 for GiGPO.FAULT’s concentration statistics describe diagnostic allocation rather than full advantage, and concentration alone does not establish more accurate localization.Source: §D.3.3 · p. 35, Figure 11 · p. 36

D.4 Penalties on Wasted Steps

  • FAULT assigns 3.67× the uniform share of negative credit to format-valid no-ops, versus 0.48× for GRPO and 0.57× for GiGPO.In analysed all-fail groups, the baselines assign zero no-op negative credit; these mechanically identified no-ops are not directly flagged by the invalid-action reward.Source: §D.4 · p. 35, Figure 12 · p. 36

D.5 Cold-Start Data Scale

  • The cold-start scale study varies nested ALFWorld pools by adding 1,440 trajectories from successive checkpoints, so pool size and policy composition vary together.Diagnosis F1 is evaluated on 600 trajectories from 25 unseen tasks; Table 5 also reports initial prices and pre- and post-RL outcomes.Source: §D.5 · p. 36, Table 5 · p. 37

E.1 Training Trajectory Length

Training trajectory-length comparisons are aggregate, single-run summaries and vary with the window used. They do not isolate path efficiency or the effect of diagnostic redistribution.

E.1 Training Trajectory Length

  • The full method has shorter mean training trajectories than GRPO and SEED in each of four windows, but the frozen-price variant is shorter in three windows and longer in the final five updates.The frozen-price variant has the shortest mean over updates 151–160, yet solves fewer tasks in separate greedy evaluation: 120 versus 122 of 134.Source: §E.1 · p. 37, Table 6 · p. 37
  • Mean trajectory length combines successful and failed episodes, so shorter means may reflect either shorter successful paths or more successes replacing horizon-truncated failures.These aggregate summaries cannot distinguish the two effects or establish that diagnostic redistribution alone improves path efficiency.Source: §E.1 · p. 37

E.2 Penalized Behaviors over Training

During training, FAULT’s penalty mass on invalid actions and no-ops declines as these behaviors become less frequent. Format-valid no-ops also fall under FAULT, unlike under the two references, though the comparison is observational.

E.2 Penalized Behaviors over Training

  • 91% to 18%: FAULT’s combined penalty-mass share on invalid actions and no-ops falls as these behaviors become less frequent.Figure 13 summarizes penalty mass in 20-update windows; invalid actions receive a shared format penalty.Source: §E.2 · p. 37, Figure 13 · p. 38
  • 3.0% to 1.7%: format-valid no-ops fall under FAULT, while rising from 2.6% to 3.8% under GRPO and from 2.6% to 4.8% under SEED.The no-op rate uses all recorded steps as its denominator; no-ops are not directly identified by the shared invalid-action penalty.Source: §E.2 · p. 38, Figure 13 · p. 38
  • Invalid actions decline under all three methods, and the single-seed observational comparison does not isolate a causal effect of FAULT’s penalties.GiGPO is omitted because its original logs lack rollout text.Source: §E.2 · p. 38, Figure 13 · p. 38

E.3 Initial Prices and Online Price Dynamics

Online pricing changes the relative allocation assigned to error classes, with search-state amnesia receiving substantial weight despite a zero cold-start price. The plotted coefficients and rankings are evolving associations, not verified causal error costs.

E.3 Initial Prices and Online Price Dynamics

  • Eight of 20 cold-start error coefficients are negative and receive non-zero prices, while the remaining 12 pricing classes have zero price.Success is shorter in 1,085 of 1,107 within-task success–failure pairs and never longer, so the length coefficient reaches its lower bound.Source: §E.3 · p. 38, Table 7 · p. 38
  • Flat or large-magnitude coefficients do not establish calibration quality, and the changing coefficients describe associations rather than causal error costs.The full method deploys updated prices; GRPO and GiGPO track coefficients without applying them, while the frozen-price variant retains its initial prices.Source: §E.3 · pp. 38–40
  • The full method’s search-state amnesia coefficient changes from −0.295 at update 30 to −5.976 at 160; the other illustrative classes end at −1.034 and −1.571.GiGPO (diag-SFT) also shows a changing fitted association without deploying prices; Figure 14 presents three classes on different vertical scales.Source: Figure 14 · p. 39, §E.3 · p. 39
  • For the frozen-price variant, rank agreement between initial prices and online weights fell from 0.63 at update 30 to −0.07 at 160; cold-start prices rescale coefficients by feature standard deviations, whereas online prices use unit scaling.These rankings describe relative-weight changes, not accuracy against known error costs.Source: §E.3 · p. 39
  • 0.46 to 0.73: search-state amnesia’s share of deployed class weight rises through update 70, then declines alongside its occurrence rate, while its own normalized weight stays 1.The two series correlate at +0.996 over updates 70–160; occurrence falls by 55% under FAULT versus 34% under GRPO (diag-SFT).Source: Figure 15 · p. 40, §E.3 · p. 40

F Case Studies

  • The hand-selected ALFWorld case studies compare FAULT’s diagnostic allocation with baseline negative optimizer advantages, using blind-judge labels that are not independently verified ground truth.The judge is an independent LLM that sees only actions and feedback; FAULT’s diagnostic allocation is not its full optimizer advantage.Source: F Case Studies · p. 42

F.1 Credit in an All-Fail Group

In an illustrated all-fail ALFWorld group, FAULT allocates most of a bounded diagnostic budget to a judged decisive revisit, unlike the format-focused baseline credits shown. A separate successful trajectory shows penalties concentrated on ineffective attempts, with some credit also assigned to a corrective examination.

F.1 Credit in an All-Fail Group

  • Approximately 82% of the selected trajectory’s P = 3.18 budget falls on the judge’s decisive step 8, a searched-location revisit, within a group where all eight rollouts fail at the 30-step horizon.The remaining 18% falls on a later cabinet revisit; the case is selected for illustration.Source: Figure 17 · p. 42, §F.1 · p. 42, Table 8 · p. 46
  • GRPO’s negative advantage marks three equal unparseable-step spikes and is zero at the judge’s decisive step; the separately sampled baseline cases are not a paired performance comparison.The GiGPO (diag-SFT) example likewise marks only format-flagged steps, and its transcript was unavailable.Source: §F.1 · p. 42, Figure 18 · p. 43

F.2 Recovery in a Successful Trajectory

  • Approximately 89% of the successful trajectory’s P = 5.52 diagnostic budget falls on three ineffective heating attempts, while 0.56 also falls on the corrective examination.A clean eight-step success in the same group has P = 0, illustrating different penalties among successful trajectories.Source: Figure 19 · p. 44, Figure 20 · p. 45, §F.2 · p. 46

G Extended Related Work

The related-work discussion contrasts outcome-based, numerical process, and language-based approaches to agentic RL credit assignment. It positions FAULT as combining checked diagnoses with outcome-fitted prices and bounded redistribution over verified error steps.

G Extended Related Work

  • GRPO assigns each rollout’s terminal-reward advantage to every step, so same-outcome groups have zero advantage and failed trajectories do not single out the decisive mistake.In the ALFWorld runs, such groups comprise 59%–64% of training groups; DAPO [36] filters them, while Yang et al. [37] use first-visit observation coverage for all-fail groups.Source: G Extended Related Work · p. 46
  • Numerical process signals can distinguish rollouts sharing an outcome, but learned step rewards can be exploited; GiGPO [7] instead compares actions from the same state.GiGPO’s discounted outcome-based credit vanishes in all-fail groups and can spread over steps after a wrong turn.Source: G Extended Related Work · p. 47
  • Language-based trajectory analysis can guide retries, produce corrected trajectories, or distill skills, but often does not directly set RL credit.SEED uses hindsight skills for on-policy distillation; GRSD [20] rescales step advantages with pooled reflections but still has zero advantage in same-outcome groups.Source: G Extended Related Work · p. 47
  • FAULT addresses three gaps by restricting diagnoses to a fixed taxonomy, checking them against step quotes, fitting prices to outcomes, and redistributing a bounded penalty over verified error steps.The cited TRAIL evaluation’s best model reaches about 11% joint accuracy in identifying and locating errors in agent traces [56].Source: G Extended Related Work · p. 47

Related papers

Loading 2610.01161v1…