PaperlayerView exact revision on arXiv

Source-linked AI summary

Agent Error Dataset: Scaling 50,000 Error--Diagnosis Pairs for Failure Analysis and Error-Aware Post-Training

Kunlun Zhu, Xuyan Ye, Yibo Li, Cheng Qian, Beibin Li, Heng Ji

arXiv:2609.40111v1cs.AIcs.CL

TL;DR

Outcome rewards do not identify which agent decision to revise or what action should replace it, and existing failure resources differ in settings, targets, and correction evidence. AED and AET retain and transform natural failures into diagnoses, proposed corrections, replay comparisons, and training views. First-proposal corrections improve matched-replay pass rates by 32.7 percentage points, and full-diagnosis fine-tuning raises exact-step agreement from 47.2% to 63.6%.

  • Problem

    Failed-rollout rewards do not specify which decision to revise or what action should replace it, while existing resources differ in execution settings, annotation targets, and correction evidence.

    Source: §1 · p. 2

  • Method

    AED retains natural-failure traces, and AET links diagnoses and proposed corrections to replay evidence where supported and to separate training views.

    Source: §1 · p. 2

  • Results

    First-proposal corrections improve matched-replay pass rates by 32.7 percentage points, and full-diagnosis SFT raises exact-step agreement from 47.2% to 63.6%.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Stored traces support failure analysis across execution settings and re-diagnosis without new rollouts.

    Source: §1 · p. 2

    Failure-only samples cannot estimate deployment failure rates.

    Source: A. Limitations and Reproducibility · p. 15

Abstract

from arXiv · show

An unsuccessful LLM agent rollout contains more information than its final reward: the observations available to the agent, the actions it chose, and the environment's responses. Reusing this experience for learning requires identifying a decision to revise and testing a concrete alternative. We introduce the Agent Error Dataset (AED), comprising 50,228 error-diagnosis pairs from 9,961 source tasks across 33 environments, 19 harness families, and 23 policy models in text-based agent systems. We retain source traces and execution metadata to support cross-setting failure analysis and re-diagnosis without repeating the original rollout. Our five-stage Agentic Error-to-Training (AET) pipeline collects natural failures, generates diagnoses and proposed corrections, and checks them against recorded evidence. Where replay is supported, we compare corrections with original-action retries from the same checkpoint under matched execution settings. We then construct separate training views for diagnosis and actor recovery. Across 3,062 matched replay pairs, first-proposal corrections raise verifier pass rates from 18.4% to 51.1%, a gain of 32.7 percentage points. Using a separately frozen diagnosis release, full-diagnosis fine-tuning on 1,656 source tasks raises Qwen3-8B's exact-step agreement with internal teacher labels from 47.2% to 63.6%, averaged over three seeds on a 943-case holdout. The strongest prompted reference in this comparison scores 54.7%, and mean agreement improves at each of four increasing training-set sizes. In a single-seed comparison of actor-training recipes, action-only repair training scores 6.67 percentage points higher on WebShop-lite than success-only training.

1. Introduction

Interactive agent failures preserve decisions and feedback, but outcome rewards and existing failure resources do not jointly specify what to revise and what action should replace it. AED and AET connect natural failures to diagnoses, corrections, replay evidence, and distinct training views.

1. Introduction

  • Agent harnesses connect language models to executable tools and persistent state, so failure analysis must consider the execution system as well as the model.Agents use tool feedback while inspecting repositories, running commands, and revising solutions.Source: §1 · p. 2
  • The paper asks how failed agent experience can be analyzed and transformed so models learn to recognize, avoid, and repair errors across execution settings.This question follows from the gap between outcome rewards and the specific decision or replacement action needed for learning.Source: §1 · p. 2
  • Existing failure annotations characterize errors or identify responsible agents, but do not consistently specify corrected actions and support both diagnosis and action learning across execution settings.Failed rollouts preserve observations, chosen actions, and environment responses, enabling explanations and proposed alternatives.Source: §1 · p. 2
  • AED contains 50,228 error–diagnosis pairs across 33 environments, 19 harness families, and 23 policy models; AET links diagnoses and corrections to replay evidence and separate training views.Stored traces support re-diagnosis without new rollouts.Source: §1 · p. 2
  • First-proposal corrections improve matched-replay pass rates by 32.7 percentage points, while full-diagnosis SFT raises exact-step agreement from 47.2% to 63.6%.The diagnosis result is averaged over three seeds and evaluated against internal teacher labels.Source: §1 · p. 2

2. Related Work

Prior work studies failure patterns, attribution, recovery, and reuse of unsuccessful experience, while AED links natural failures to diagnoses, corrections, and available replay evidence. Figure 2 distinguishes construction and training views from cohort-specific coverage and experiments.

2. Related Work

  • Failure-analysis work organizes recurring patterns into taxonomies or identifies responsible agents and error locations, including in long traces.MAST and AdaMAST develop taxonomies; attribution studies locate agents and errors.Source: §2 · p. 2
  • Figure 2 separates coverage, historical error profiles, and experiments into distinct cohorts; its replay bars use proposal-specific controls to measure correction utility.The caption identifies first/selected proposals and restored-harness selected proposals as separate replay groups.Source: Figure 2 · p. 3
  • Recovery work tracks error resolution or connects diagnosis to corrective execution, while AED links natural failures to diagnoses, proposed corrections, and available replay evidence.Prior examples include TrajDebug, AgentDebug, AgentDebugX, and CUADebug.Source: §2 · p. 3
  • Prior unsuccessful-experience methods reuse failed actions, segments, or goals, create reflective recovery trajectories, or guide later behavior with critiques; AED compares corrections with same-checkpoint retries on a replay-supported subset.Trajectory export tests do not establish full-record interoperability.Source: §2 · p. 3

3. Agent Error Dataset

AED organizes failed agent traces into diagnoses, tested corrections, and objective-specific training views. Its experiments find gains in matched correction replay and internal diagnosis agreement, alongside mixed actor-training outcomes and limits on transfer.

3.2. AET: a five-stage data generation pipeline

  • AET links natural-failure collection, trace-cited diagnosis, evidence checks, matched replay where supported, and separate training views with distinct admission rules.Replay compares a correction with a fresh original-action retry from the same checkpoint under matched execution settings.Source: §3.2 · p. 4, §3.1 · p. 4

3.3. Collection scope and training subsets

  • AED indexes 50,228 pairs across 9,961 source tasks, with 33 environments and 19 harness families meeting the ten-task floor; pairs do not count independent executions.The collection contains 38,278 stored source-trace blobs, but blob identities do not establish the number of independent executions.Source: §3.3 · p. 4, Figure 3 · p. 5

4. Training Views

  • Diagnosis targets include the attributed step, responsible agent, explanation, and correction; actor views supervise passing actions before or after an error, optionally with reflection.Post-error context includes the rejected action only for state-preserving rejections, and excludes the later failed suffix.Source: §4 · p. 5

Appendix B.1 analyzes source coverage and diagnosis multiplicity on this same collection index.

  • Exploratory harness profiles show action-error labels lead in 8 of 9 groups, while the π agent concentrates labels on verification and observation; task mixtures differ.These profiles are a subset comparison, not a causal effect or full-collection distribution.Source: Appendix B.1 · p. 5, §4 · p. 6, Figure 4 · p. 6

5.1. Diagnosis production and correction utility

  • 32.7 percentage points: first-proposal corrections raised verifier success from 18.4% to 51.1% over original-action retries across matched replay pairs.The comparison uses first proposals without selecting among repeated proposals.Source: §5.1 · p. 6

5.2. Can AED train a competitive failure-diagnosis model?

  • On the same 943 cases, full-diagnosis SFT raised case-level micro exact-step agreement with recorded teacher labels at all four nested training sizes; even the smallest exceeded the untrained base.The citation-first judge produced trace-cited diagnoses for 93.5% of the common failure pool, compared with 48.4–65.8% for evaluated alternatives; this measures production yield, not independent label accuracy.Source: §5.1 · p. 6, Figure 5 · p. 7, §5.2 · p. 7

5.3. Can repair supervision improve the acting policy?

  • Repair recipes improved WebShop-lite but reduced TextQuest; action-only and reflective WebShop gains and all TextQuest losses survived adjustment, while preventive results lost to the untrained policy on real ALFWorld and ScienceWorld.The recipe comparisons use one seed and differ in task pools, exposure, and optimizer updates.Source: §5.3 · pp. 8–9

6. Discussion and Conclusion

  • The authors note that historical supervision omits policy context and later checks do not validate it retroactively; repository data can expose identifiers, requiring source-specific privacy and licensing review.The release plan includes redaction of sensitive content and unnecessary personal identifiers while preserving required attribution.Source: §6 · p. 9, Ethics statement · p. 10

A. Limitations and Reproducibility

The collection is broad but uneven, and its pair counts, historical analyses, and training experiments use distinct populations and denominators. The reported evidence is limited by objective-specific eligibility, versioned cohorts, and scope boundaries that preclude treating collection coverage as evidence of full-collection learning utility.

A. Limitations and Reproducibility

  • Collection size counts error–diagnosis pairs, not independent executions or training-ready records, and detailed studies retain their own cohort denominators.Source: A. Limitations and Reproducibility · p. 15, §B.1 · p. 16
  • Diagnosis, repair, and preference objectives require different evidence, so observed gains concern frozen objective-specific subsets rather than full-collection utility.Source: A. Limitations and Reproducibility · p. 15
  • The final index, frozen diagnosis subsets, and actor-development pool are versioned populations, not a verified nested filtering chain, and their counts are not additive.Source: A. Limitations and Reproducibility · p. 15
  • Diagnosis experiments use an earlier release missing policy prompts and tool lists and predating later checks; the results do not validate the revised pipeline or full collection.The passage also states that earlier label defects remain unresolved.Source: A. Limitations and Reproducibility · p. 15
  • Failure-only samples cannot estimate deployment failure rates, while unequal task and model mixtures confound harness rankings and model-version trends.Source: A. Limitations and Reproducibility · p. 15

B.1. Coverage and multiplicity in the current collection

  • The five largest harness families supply 78.8% of pairs, while the five largest environments supply 44.0%; collection shares cannot rank reliability without successful-rollout denominators.BFCL is the largest source at 11.0% of pairs.Source: §B.1 · p. 16
  • Multiple diagnoses add 11,950 pairs beyond one per source blob, but do not add independent failure observations; prevalence analyses must group by trace and source task.Source: §B.1 · p. 16

B.2. Repair outcomes and diagnostic revision

The section reports paired correction and retry outcomes, diagnostic revisions, collection profiles, and the evidence boundaries for interpreting them. These analyses use distinct cohorts and denominators, rather than estimating prevalence or validating root causes across the final collection.

B.2. Repair outcomes and diagnostic revision

  • 1,094 correction-only versus 93 retry-only attempts yield the paired contrast; 470 of 1,564 successful corrections also pass after retrying the original action.The figure displays all four paired outcomes; a single paired outcome does not identify an individual causal effect.Source: §B.2 · p. 16, Figure 7 · p. 17
  • Across 405 multi-round attempts, 1,064 of 1,744 adjacent diagnoses retain the same attributed step (61.0%), the largest transition category in each wave.A fixed location still permits the explanation and proposed action to change; later diagnoses are not established as more accurate.Source: §B.2 · p. 16, Figure 7 · p. 17

B.3. Crossed coverage of the current collection

  • Comparisons across models or harnesses require shared tasks and successful-rollout denominators because collection support is uneven and failure-pair counts alone cannot rank reliability.Figure 8 maps pair counts across all environments and harness families; empty cells do not show that a combination is unsupported.Source: §B.3 · p. 18, Figure 8 · p. 18

B.4. Historical error profiles by model and environment

  • Verification is the largest classified category in ALFWorld-lite (239/761), while action and planning together account for 336/423 Text-to-SQL labels; these profiles describe a historical sample.Models generating trajectories need not be the labeling judges.Source: §B.4 · p. 19

B.5. Observable symptoms with successful-rollout denominators

  • 248 of 822 failed executions include a completion action that fails the task verifier, requiring outcome checks beyond executable-action validity.The symptom figure describes an older harness cohort, not current harness quality.Source: §B.5 · p. 20, Figure 10 · p. 20

B.6. Complete distributions behind Figure 3

  • The complete environment and harness distributions report shares over all 50,228 pairs; expanded “Other” groups are already included in the main rows.A task can occur under several harnesses, so harness task counts do not sum to the unique environment–task total.Source: §B.6 · pp. 21–22, Table 5 · p. 21, Table 6 · p. 22

C.1. Interpreting execution evidence

  • A successful trial establishes observed recovery, not a unique root cause or reliable effect, and repeated contrasts still depend on restored state, policy, harness, and verifier.Evidence grades can be recomputed from recorded outcomes without another environment run.Source: §C.1 · p. 23

C.2. Corpus scope and counting

  • An error–diagnosis pair requires a natural failure, a named candidate error step, an explanation, and at least one trace-grounded quotation; successful replay is not required.Executed passing corrections form a smaller, separately reported subset with control outcomes and replay settings.Source: §C.2 · p. 23

C.3. Notes on the resource comparison

The resource comparison distinguishes AED’s populations, counting units, and evaluation targets from those of other resources. The construction and evidence procedures preserve provenance and separate diagnosis support, executed recovery, and stronger matched replay evidence.

C.3. Notes on the resource comparison

  • AED’s diagnosis release and replay study are distinct populations: the former has 4,319 diagnosis rows, none joined to a replay record, while replay uses 3,062 attempts.The resource table’s counts use different reported units; repeated diagnoses do not add independent failures.Source: §C.3 · pp. 23–24
  • AED and AEGIS scores are not a matched training comparison because their attribution targets, labels, base models, budgets, and test conditions differ.Their reported units also differ, so the paper does not import AEGIS scores into its localization tables.Source: §C.3 · pp. 24–25

D. Construction, Replay, and Evidence

  • The five-stage construction collects natural failures, diagnoses and grounds proposed corrections, replays matched correction and original-action branches where supported, then builds separate training views.Known infrastructure or grader faults are screened from agent-error targets; accepted and rejected proposals are retained.Source: §D.1 · pp. 25–26

D.1. Construction algorithm

  • Collection admission requires trace linkage and quotations, but structural checks do not certify semantic validity or training eligibility; recovery targets additionally require an executed passing continuation.Recorded negative outcomes, valid task-failure verdicts, and supported attributions are distinguished.Source: §D.1 · p. 26

D.2. Evidence ladder and statistical claims

  • E2 requires completed arms with at least two trials each, every treatment trial passing, and every matched control failing; lower evidence grades do not establish unique causation.The ladder distinguishes ungrounded records, grounded diagnoses without replay, and partial or observed recovery.Source: §D.2 · p. 26

D.3. Lineage and checkpoint fidelity

  • Lineage checks verify the source trace, prompt, restored state, trial, applied action, and transcript; reconstructed logs remain in a separate fidelity stratum.The exclusion ceiling controls attrition, not semantic accuracy.Source: §D.3 · p. 27

E.1. Infrastructure and task-contract defects

This section documents infrastructure and task-contract defects, limitations in the frozen data, annotation audits, and illustrative traces, alongside revised quality controls. Its examples clarify the limits of what labels and replay results can establish, including the scope of post hoc taxonomy assignments and evidence needed to diagnose UPDATE failures.

E.1. Infrastructure and task-contract defects

  • 440 retrieval diagnoses record universal search/fetch failure, while grader mismatches and missing teacher-contract context also undermine some frozen labels.The 440 rows span GAIA and WideSearch; 234 AgentBench-DB tasks fail to reproduce the table hash, and 617 of 1,016 rows were labelled before the submission contract entered the prompt. Mind2Web’s first-tool-error rule scores 100% on 78 held-out cases, and 219 BFCL rows blame a later turn than the stored first-turn goal.Source: §E.1 · p. 27

E.2. Construction inventory and counting ledger

  • 10,057 configurations had diagnosis records, but only 9,464 had both a step and explanation, and structural completeness did not establish grounding or validity.The eligible records came from 5,376 distinct source tasks; a separate audit found citation grounding in 6,381 of 11,116 diagnosis attempts (57.4%), which cannot be appended to the configuration funnel.Source: §E.2 · p. 28

E.3. Current annotation quality audit

  • 46.8% exact-step agreement and 75.2% citation admission in paired ALFWorld traces show that attribution agreement does not ensure evidence support.Among 297 matched cells, within-one-step agreement is 53.9%; panel citation admission is 243/323 versus solo’s 285/323. Across 3,470 test–retest pairs, admitted first verdicts repeat the exact step at 88.2% versus 58.2% for non-admitted verdicts, an association that does not establish correctness or causality.Source: §E.3 · pp. 28–29

E.4. Model-panel adjudication of the blinded semantic sample

  • 102/120 blinded records received a defensible step attribution or alternative, but only 9/120 received unanimous endorsement of both step and full evidence support.The three model-family judges saw rendered traces and diagnoses but not replay outcomes, grades, label sources, or annotators; expert adjudication remains unrun.Source: §E.4 · p. 29

E.5. Release audit and revision history

  • 37/60 sampled frozen-release rows were rejected in a defect-finding audit, and later checks do not retroactively validate the evaluated inputs.The sample was equally drawn across environments, so it does not estimate corpus-wide defect prevalence; the release also retained rows with faulty grader references and incorrect responsible-agent labels.Source: §E.5 · pp. 29–30

E.6. Quality rubric and certification

  • The 12-item admission rubric requires attributable agent error, supported evidence, an executable corrective change, and faithful, leakage-free inputs and targets.Ambiguity is recorded but does not reject a row; deterministic checks decide items 4 and 11, while both semantic judges must accept each rejecting item after those checks.Source: §E.6 · pp. 30–31

F.1. A failed trajectory, diagnosed and repaired

  • A shared-checkpoint replay of the line-count case passed with the correction that filters matching lines, while retrying the original command failed.The proposed branch returned and submitted 4; the original-action branch submitted 5. This single recorded pair does not establish general reliability or learned self-repair.Source: §F.1 · p. 32

F.2. Trace, target and competing interpretation

  • Illustrative BFCL and tool-use cases show that quoted traces can still support a bad target, transport success can conceal incomplete content, and missing verification alone does not prove failure.These purposively selected examples are not an accuracy sample: the target in Case B repeats the blamed action despite a new request, and Case C’s truncation attribution requires checking the original wire.Source: §F.2 · p. 33
  • An empty SQL result alone does not establish UPDATE failure; the historical audit example was purposively selected and lacked independent adjudication.Inspect the post-update state and grader before accepting the diagnosis.Source: §F.2 · p. 33

G. Taxonomy definitions and validation study

  • Taxonomy construction collects attribution, explanation, evidence, and proposed action, then stores induced error-type labels in versioned sidecars without overwriting earlier assignments.Each version retains its definitions and executable labeling state; error-type frequencies apply only to the specified historical labeled cohort, and machine assignments test labeling reliability rather than serving as training targets or admission criteria.Source: G. Taxonomy definitions and validation… · p. 33

G.1. Independent labeling-method study

Search-selected corrections outperform original actions across replay strata, while the located-recovery results retain their ordering under crash-as-failure scoring. The labeling study finds limited conditional judge agreement, and its tested rubric change does not establish improved reliability.

G.1. Independent labeling-method study

  • Of 357 items classified by both judges, they agreed on 154 (43.1%); mode-level κ was 0.313 and family-level κ was 0.329.The limited, nonrandom cohort is conditional on both judges assigning a mode and does not measure agreement with human judgments.Source: §G.1 · p. 34
  • The tested rubric change yielded mode-level κ +0.069 (95% interval [−0.022, +0.157]) and family-level κ +0.099 ([−0.018, +0.212]); both intervals include zero.The trial does not establish improved labeling reliability.Source: §G.1 · p. 34

H. Result tables for the body figures

  • Table 11 scopes a resource comparison under one base, recipe, and scorer, reporting micro exact-step percentages and unit+step only on AgenTracer’s test split.Trained rows are mean ± sd across three seeds; the paired-contrast interval is the widest per-seed 95% bootstrap interval.Source: Table 11 · p. 35

H.1. Paired replay by environment

  • Search-selected proposals beat original actions by 47.1 points [42.3, 51.7] on 871 fully restored source-harness pairs; substituted and partial-state strata also show positive contrasts.The substituted ReAct (single) and partly restored multi-agent strata show differences of +43.5 and +50.7 points, respectively; these are not first-proposal effects.Source: §H.1 · p. 35, Table 14 · p. 36
  • Only 11 of 3,062 replay attempts exhausted the step budget, and none was a case where only the original action passed; replay tests replacements, not debugger training.The eleventh environment contributed one attempt and no discordant pair.Source: §H.1 · p. 36

H.2. The located-failure cohort and its failure accounting

  • The located-recovery cohort froze 150 attempts from 1,235 eligible tasks, one per source task across 11 environments, with randomized arm order and explicit crash accounting.The four arms differ only in pre-action coach text; the clean and deployed arms use different framings of the recorded explanation.Source: §H.2 · p. 36
  • Crash-as-failure scoring yields 18.6%, 26.0%, 31.3%, and 35.3% across four arms versus complete-case 18.7%, 27.1%, 32.2%, and 36.1%, preserving ordering.The exact McNemar check agrees with intervals for contrasts far from zero, but not for close contrasts.Source: §H.2 · p. 36
  • Corrected continuation-cap estimates put the content effect at +5.6 points (0.0 to 11.3), replacing the superseded estimate of +0.8 (−6.3 to 7.8).The earlier run exhausted the reasoning budget in 52 of 450 coached continuations, compared with 13 after correction.Source: §H.2 · pp. 36–37

H.3. Resource against resource under one recipe

These comparisons examine how training resources and protocols affect diagnosis transfer, actor training, production yield, and construction costs. Across them, results are protocol-specific, and output compatibility or retrospective sensitivity does not by itself establish diagnosis quality or causal effects.

H.3. Resource against resource under one recipe

  • Both resources improve joint attribution on AgenTracer’s test split, but neither establishes an exact-step gain on the other resource’s test set.On the AED holdout, the joint score rewards matching a constant unit label, and AED does not improve exact step over base.Source: §H.3 · p. 37

I.1. External-label transfer, in both directions

  • On TrajErrBench’s 486 cases, the untrained base scored 18.72%, while two of three trained seeds regressed; task overlap further limits transfer claims.The benchmarks define a step differently, and the study did not measure which difference drives the sign.Source: §I.1 · p. 38
  • +7.79 pp was the Who&When mean exact-step contrast after replacement-data retraining, under the original base decode and pre-specified item-bootstrap rule.The result is protocol-specific; the comparisons do not isolate the causal contribution of overlap or bound its share of the original gain.Source: §I.1 · p. 38

I.2. Completed full-fine-tuning recipes / I.3. Diagnosis targets and evaluation

  • Diagnosis training targets explanations, evidence, and proposed actions after attribution fields, while evaluation compares predictions with recorded teacher labels on a frozen holdout.Claims of superiority require a paired interval excluding zero; matching uses a predeclared two-point non-inferiority margin.Source: §I.3 · p. 40

I.4. Agent post-training design

  • Actor evaluation measures initial-state task success without a debugger-supplied location, and the post-error action-only control matches erroneous actions, feedback, and corrected actions to reflection.The historical success-only, preventive, and reflective mixtures differ in task coverage, updates, and token budget.Source: §I.4 · p. 40

I.5. Comparison scope and interpretation

  • Public comparisons support protocol-specific conclusions, not leaderboard rankings; actor recipe contrasts combine differences in task coverage, exposure, and optimizer updates.Human-review samples do not estimate collection-wide accuracy, and teacher agreement does not establish label correctness.Source: §I.5 · pp. 40–41

I.6. Diagnosis production on a common failure pool

  • Diagnosis-production yield measures structured locations with matching trace citations, so it supports choosing an export-compatible route rather than ranking diagnosis quality.Configurations also differ in rendering and teacher access, and a matching quote does not establish correct attribution or a supported correction.Source: Table 19 · p. 41, §I.6 · pp. 41–42

I.7. Construction cost and scale estimates

  • Construction costs depend on task mix, teacher, prices, trace lengths, and acceptance yield; diagnosis-only projections exclude collection, review, repair, and hardware costs.The estimates do not price the final quality-admitted release, and separate cohorts must not be added into a full-collection bill.Source: §I.7 · pp. 42–43

I.8. Cohorts and interpretation of the main results

This section interprets the results across distinct cohorts and evaluation setups, distinguishing replay corrections from less conclusive diagnosis-training and transfer findings. It also reports multi-round diagnosis patterns while noting that the comparisons do not establish equivalence between construction methods or a causal effect of additional repair rounds.

I.8. Cohorts and interpretation of the main results

  • 59.1% of selected replacements passed replay, versus 13.7% for original actions, a paired difference of 45.3 points across 3,062 attempts.The task-clustered 95% interval was 41.3 to 49.2 points over 1,284 source tasks; search stopped at the first recovering proposal.Source: §I.8 · p. 43

I.9. Pipeline labels against a single judge

  • Diagnosis SFT established no attribution gain over the untrained base or consistent advantage of construction-pipeline labels over single-judge labels at either 168 or 948 source tasks.The paired intervals were wide relative to the registered 5-point effect; public-split comparisons likewise showed no consistent transfer benefit, including lower scores on AgentErrorBench.Source: §I.9 · p. 45

I.10. What the diagnosis itself adds

  • Diagnosis content added 5.6 points over generic reconsideration, but its task-clustered interval reached zero, so an advantage over a second chance was not established.The cohort contained 150 failures in 11 environments, resumed at the attributed step’s pre-action state; the arms differed only in what the actor was told.Source: §I.10 · p. 46, Table 27 · p. 46

I.11. Full-diagnosis training: size and seed stability

  • The larger full-diagnosis training rung improved micro scores under both prompts, but the middle-to-948-task seven-field contrast was +2.12 points and established neither further improvement nor saturation.Its family-macro intervals included zero, so aggregate gains need not extend across task families; the ladder describes recorded training procedures, not an isolated causal effect of adding data.Source: §I.11 · pp. 47–48, Table 29 · p. 47

I.12. Full-diagnosis transfer across public attribution protocols

  • Public transfer did not establish general cross-benchmark improvement: the positive unified Who&When result did not survive correction, while three corrected significant results were losses.The TrajErrBench regression persisted among parseable outputs at −11.73 points, but that post-treatment subset is descriptive and cannot explain the full-cohort loss.Source: §I.12 · p. 48, Table 31 · p. 49

I.13. Compact-target training-set size

  • 48.89%, 47.72%, 47.40%, and 46.13% were compact-target exact-step scores at 120, 252, 504, and 1,020 tasks, versus 47.19% for the base and 46.59 ± 0.12% at 1,656 tasks.The one-seed nested subsets scored on the same 943 held-out cases; empty reasoning blocks rose from 0.0% at up to 504 tasks to 90.4% at 1,020 and nearly 100% at 1,656.Source: §I.13 · pp. 49–50, Figure 14 · p. 50

I.14. A reasoning-preserving arm

  • The reasoning-preserving arm scored 48.46% against the base’s 47.19% on seed 17, but its +1.27-point paired difference had an interval of [−1.00, 3.65] and did not establish a gain.It used 1,008 self-distilled rows and one training seed; because target content, length, and retained task support also changed, this was not an isolated test of reasoning preservation.Source: §I.13 · p. 50, §I.14 · p. 50

I.15. Preference optimisation on measured-effect pairs

  • DPO changed relative action likelihoods but barely changed held-out chosen-action agreement: 54.6 ± 0.6% versus the base’s 54.3%, with no task-recovery evaluation.On 99 held-out pairs, 94 compiled; DPO gave positive implicit margin on 63.5 ± 3.3% and a mean margin of 1.29 nats.Source: §I.15 · p. 51
  • The measured-effect preference table reports offline agreement on 94 held-out pairs that compile, not task success, and its base and DPO margins are not comparable.Five pairs were skipped; the 21-pair dev set was a sanity check, not used for selection.Source: Table 33 · p. 51

J. Additional detail for the experiments

  • Neither the pilot null nor the compact scaling curve establishes equivalence between construction methods; separate full-diagnosis scaling results use a different target and must not be pooled.The corresponding result tables preserve full counts and uncertainty.Source: J. Additional detail for the experiments · p. 51

K.1. Multi-round repair yield and diagnosis changes

  • 61.0% of 1,744 diagnosis transitions named the same step again, versus 20.8% moving later and 18.2% earlier.Across 405 multi-round attempts, these observed transitions indicate diagnosis revisiting was more common than progression through independent errors; they do not measure the effect of adding a round.Source: §K.1 · p. 52, Figure 16 · p. 53
  • Table 35 covers 507 diagnosed attempts, including 377 eligible for paired replay; later rounds add observed successful contrasts, without a randomized one-round control.Attempts that stopped while still failing are right-censored.Source: Table 35 · p. 52

L. Error reporting on successful trajectories

On 50 successful trajectories, models prompted to assume failure reported errors on every case, while prompts allowing “none” produced sharply different abstention rates. The small, prompt-specific sample does not estimate false-positive rates or establish transfer to neutral prompts.

L. Error reporting on successful trajectories

  • On 50 successful trajectories, three models named an error in all 150 primed cases; allowing “none” yielded abstention rates of 84%, 86%, and 44%.The first two models still reported errors on all 10 matched failures; 39 of 50 successful trajectories contained a recoverable hiccup.Source: L. Error reporting on successful… · p. 53
  • Paired prompts on the same trajectories compared the AgentErrorBench instruction verbatim with a version removing the failure premise and permitting none, across twelve environments, seven harnesses, and ten policies.There was one trajectory per task, and attribution accuracy still requires semantic adjudication.Source: L. Error reporting on successful… · p. 53
  • Final-visible-step concentration, a small success sample, and one prompt family limit interpretation as general error detection; the study does not estimate false-positive rates or neutral-prompt transfer.The matched failures provide only a sensitivity check.Source: L. Error reporting on successful… · p. 53

M. Sensitivity to the supplied error location

The supplied-location sensitivity study compares recovery against checkpoint-specific controls and finds a significant gain when the proposer receives the labelled step, but not when given a different step. The accompanying controls, release census, and coverage notes delimit what these experiments and collection counts represent.

M. Sensitivity to the supplied error location

  • Told the labelled step, the proposal beat its control by 12.9 points; told a different step, the gain was 3.1 points and did not establish an advantage over that location.Without a supplied step, the proposer named a replayable location on 136 of 150 rows and gained 9.8 points against its own control; the between-arm comparison with the labelled location was not significant. The cohort overlaps training on 55 of 150 tasks, so these are exploratory results, and a null difference is not equivalence.Source: M. Sensitivity to the supplied error… · p. 53

N. Output-contract and abstention controls

  • Output-contract and abstention development studies did not justify either prompt change; repeated identical calls also changed admission outcomes.The schema instruction did not improve citation-grounded admission over retrying the unchanged prompt, and removing abstention did not support the proposed mechanism.Source: N. Output-contract and abstention… · p. 54

O.1. The evaluated release by environment

  • Table 12 counts the frozen v14 diagnosis release by environment, including rows, source tasks, harness families, policy models, and replay-package attempts; release rows are marked not replayed.Replay evidence resides in a separate package.Source: §O.1 · p. 54

O.2. Current collection coverage

  • Failure-only environment counts do not support model-generation comparisons without common tasks, matched execution settings, and successful-rollout denominators.The environment inventory uses the same pair index as Figure 3.Source: §O.2 · p. 54

O.3. Environment fidelity and the real-environment rebuild

  • The 15-environment release plan repeated tasks across policy, harness, and temperature; replay certification is disabled for environments lacking a valid interactive repair endpoint.The collection includes procedural toys, simplified stand-ins, synthetic generators, and Text-to-SQL that are excluded from the frozen release.Source: §O.3 · p. 54

O.4. Replay sample budget

  • Replay sample budgeting used measured pass-rate variance and environment costs; current construction uses one execution per arm and reports paired discordant counts.State-restoration checks determined which environments could be replayed.Source: §O.4 · p. 55

P. Prior evaluation exposure and the current split

This block defines how prior evaluation exposure is recorded and how the current split and replay comparisons are constrained. It also distinguishes supervision interfaces and reports limits on single-trial replication.

P. Prior evaluation exposure and the current split

  • The exposure registry links recorded prior evaluation outputs, including partial runs, to tasks, trajectories, and leakage components; undocumented external use remains unruled out.Construction-time teacher labels and unexecuted prompts are not counted as evaluation exposure.Source: P. Prior evaluation exposure and the… · p. 55
  • The v14 release excludes exposed or quarantined material from train, development, and holdout, but had no separate exposure-registry audit.The authors treat the export rule, not an independent zero-exposure measurement, as evidence; singleton splitting also limits family-disjointness claims.Source: P. Prior evaluation exposure and the… · p. 56

Q.2. Replay harness and state fidelity

  • Replay contrasts require treatment and control to share a restored state and continuation policy, but substituted ReAct does not test transfer to original coordination, memory, or stopping rules.Current contrasts use one sample per arm; repeated branches and unique causation require additional evidence.Source: §Q.2 · p. 57

Q.6. Supervision interfaces

  • Training views distinguish diagnosis of a completed trace from action supervision at an intervention point; each Figure 17 arrow denotes an interface, not measured training benefit.The post-error view retains the erroneous action and feedback as masked context, while action comparison requires executed alternatives from the same state.Source: Figure 17 · p. 58, §Q.6 · p. 58

Q.3. Learning objectives and execution assumptions

  • The appendix specifies masked actor targets and conventional objectives, while noting that the historical pilot does not qualify as a recovery experiment.For the post-error mixture, reflection text is supervised in the corrected response’s Thought field, and the action-only control removes that inserted reflection.Source: §Q.3 · pp. 57–58

Q.4. Metrics by supervision target

  • Offline diagnosis and repair metrics are proxies rather than environment-verifier execution, and benchmark prompts and step-index conventions vary across evaluations.Who&When separates hand-crafted from algorithm-generated cases; AgentErrorBench uses 1-based agent-step indices and predicts a failure module rather than a responsible agent.Source: §Q.4 · p. 58, §Q.5 · p. 58

R. Replication of single-trial contrasts

  • 85/126 certified actions passed again, while the full success/failure contrast repeated on 66/126 revised rows selected after successful revision.These rates apply only to the post-revision certification cohort.Source: R. Replication of single-trial contrasts · p. 58
  • A revised action outperformed the first proposal when that proposal failed, but its advantage disappeared when both proposal and control passed.Same-action repeats showed stochastic reversals; the observations do not isolate a causal revision benefit or establish a corpus-wide replication rate.Source: R. Replication of single-trial contrasts · p. 59

S. Supervision comparison: current and historical cohorts

The supervision comparison reports a pilot contrast with uncertainty that does not establish equivalence, alongside limits on teacher-label interpretation and training-protocol comparability. The appendices describe stored records and which supervision views the frozen release can support.

S. Supervision comparison: current and historical cohorts

  • Pilot arms each contain 168 tasks across 17 families, with shared inputs within each format, but joint admission restricts comparison to examples both procedures can render.Teacher access and execution were not independently controlled, and pilot and final-cohort estimates should not be pooled.Source: S. Supervision comparison: current and… · pp. 59–60
  • Agreement with internal teacher labels measures fit to consensus annotations, including shared conventions and label errors, not an independent-label capability ranking.The consensus arbiter is recorded but is not necessarily the sole author of each label.Source: S. Supervision comparison: current and… · p. 59
  • The CPU loss check matches the per-example-mean reference within 4.7 × 10^-8 loss and 4.9 × 10^-8 gradient, but does not exercise the GPU fused kernel.Without disabling default token-count normalization, the test batch’s loss differs by 0.164; historical GPU runs did not log a runtime assertion.Source: S. Supervision comparison: current and… · p. 59
  • All seed-matched X-minus-J family-bootstrap intervals contain zero, so the pilot establishes neither unchanged behavior nor equivalence.The holdout is reused across seeds, and the intervals describe family-resampling evaluation uncertainty rather than independent tasks.Source: S. Supervision comparison: current and… · pp. 59–60

T.1. Record contents

  • A source trace stores task and execution context, while debugging attempts add attribution, evidence, proposed corrections, replay samples, and construction provenance.Multiple attempts can share a trace; execution transcripts and replay branches preserve trial conditions and outcomes.Source: §T.1 · p. 60

T.2. Eligibility by supervision target

  • The frozen release materializes the diagnosis view, while views requiring executed branches are empty because 9,712 of 9,715 pooled attempts were never replayed.Replay is optional for this natural-failure corpus, and reference-assisted labels remain privileged-teacher supervision rather than unassisted label construction.Source: §T.2 · pp. 60–61

T.3. Relation to the Agent Data Protocol

The section compares AED’s diagnosis-and-correction records with ADP trajectory records, then documents prompt targets and evaluation protocols. It emphasizes that conversion checks and public comparisons have bounded scope.

T.3. Relation to the Agent Data Protocol

  • 672/672 certified rows exported their failing trajectory as ADP trajectories, with zero losses under the exporter’s stated checks, while AED adds grounded attribution, replacement actions, and replay verdicts.ADP schema validation was structural, and use by other ADP consumers was not tested.Source: §T.3 · p. 61, Table 42 · p. 62

U. Core Prompts

  • The prompt defines corrected_action as one concrete replacement tool call with arguments, returning null when no specific valid action can be named or the blamed step was not an incorrect action.It prohibits inventing unavailable tools and requires trajectory-compatible tool names and argument keys.Source: U. Core Prompts · p. 63
  • The construction target may include a proposed correction even when replay was unavailable or failed, and it does not guarantee the earliest error, root cause, or unique decisive step.Thus a recorded correction is not itself evidence of recovery or causal effectiveness.Source: U. Core Prompts · pp. 63–64

U.1. Native-context public evaluation

  • Public evaluation targets differ: TrajErrBench asks for the earliest critical failure and module, while AgentErrorBench predicts a failure module using numbered agent steps.Who&When instead evaluates step and failure-module or responsible-agent predictions; AgentErrorBench’s index is an agent-step index, not a message index.Source: §U.1 · p. 64

V.1. First-call frontier comparison protocol

  • Table 3 scores first responses on the same frozen cases; its seed-17 student follows answer-format continuation on public labels, unlike Figure 5’s three-seed internal holdout, so the displays are not a shared-test ranking.First and final predictions agree case by case; differences between tables must not be interpreted as model gains under one protocol or attributed entirely to forcing.Source: §V.1 · pp. 64–65
  • Repeated base-model exact-step ranges across six runs are maximum-minus-minimum values, not confidence intervals or calibrated significance thresholds.The comparison also excludes model rows that exceed the stated reasoning-token completion screen, so it does not rank those omitted models’ attribution ability.Source: §V.1 · p. 65

V.2. Earlier public benchmark references

This section retains earlier public attribution references but distinguishes their protocols from the completed paired training comparison. Historical results and positional baselines therefore have limited comparability.

V.2. Earlier public benchmark references

  • The earlier Qwen3-8B reference is a historical base invocation and is not combined with the current base to estimate training effects.The best constant step uses each test split’s labels, so it diagnoses positional bias rather than serving as a deployable baseline.Source: §V.2 · p. 65
  • Table 43’s exact-step metric uses the full planned denominator, while MAST reports micro-F1 against released model-judge labels.These references are separate from the completed paired training comparison.Source: Table 43 · p. 65
  • AgenTracer-8B’s author-reported‡ results are 20.7 W&W-HC and 37.3 W&W-AG, while historical prompted GPT-5 mini§ reports MAST micro-F1 57.0 (n = 197).The table identifies these as results under different historical protocols.Source: §V.2 · p. 65, Table 43 · p. 65

W.1. Returned judgments and agreement

The review preserves original judgments from an AI-assisted assessment of historical records. Reviewers often selected the same error step, while diagnosis verdicts showed more disagreement.

W.1. Returned judgments and agreement

  • Reviewers matched preferred steps on 59/69 localizable records (85.5%) and both accepted 41/80 diagnoses (51.3%), with raw verdict agreement of 50/80 (62.5%).For records where both reviewers chose a single step, they agreed on 46/50 (92.0%).Source: §W.1 · p. 66

W.2. Sample, interface and instructions

  • Four author-volunteer raters reviewed 80 historical records with shared AI assistance; the coverage-selected sample is not a random final-release sample.The two reviewers assigned to each record saw identical model advice, so their judgments are not independent external validation.Source: Table 44 · p. 66, §W.2 · p. 67

W.2.1. Instructions supplied to raters

  • A decisive error must be supported by visible evidence, materially contribute to failure, and be correctable with information and tools available at that step.Reviewers retain disagreements and abstentions; agreement on sampled records does not establish correctness against an independent reference label.Source: §W.2.1 · pp. 67–68

X.1. Construction and diagnosis scores

  • Table 47 separates correction pass rates from diagnosis-production yield and cost; Table 48 reports internal 943-case teacher-label agreement and public benchmark scores.Yield counts structured diagnoses with matched trace citations, not independently correct labels.Source: Table 47 · p. 70, Table 48 · p. 70, §X.1 · p. 70

X.2. Complete actor scores

  • Actor scores cover held-out tasks in six development environments, but the one-seed recipes differ in task coverage, exposure, and update count.These comparisons therefore combine supervision type with those training differences.Source: Table 49 · p. 71, §X.2 · p. 71

X.3. Admission and replay populations

  • Table 50 separates admission-rule yield and required spend from verifier pass rates measured on a shared-checkpoint replay cohort.Cost per retained row includes all spending required by the admission rule; reaching 767 AET rows required diagnosing all 19,904 candidates.Source: Table 50 · p. 71, §X.3 · p. 71

X.4. Debugger contrasts and settings

  • The seed-17 1,656-task checkpoint scores 65.93/63.31 macro/micro, versus 65.52/60.34 for the 948-task checkpoint; these are single-seed contrasts.The multi-seed arms use seeds 17, 202, and 828, and each internal checkpoint answers 942/943 cases.Source: §X.4 · p. 71, Table 51 · p. 72

X.5. Public attribution: protocol sensitivity and settings

  • Public attribution results depend on prompt protocol: the diagnosis student loses exact-step accuracy in original Who&When conditions, and point-estimate gains do not establish public state-of-the-art performance.The unified AgentErrorBench prompt requests only a step, so no module score is defined.Source: §X.5 · p. 72, Table 52 · p. 72

X.6. Actor exposure and descriptive statistics

Actor-repair comparisons combine supervision type with differences in task exposure and update count. Results vary by environment, and the preventive-repair arm loses to the base policy on the two tested real environments.

X.6. Actor exposure and descriptive statistics

  • Repair arms use 1,790 rows over 1,183 tasks and 1,490 updates, versus success-only’s 1,093 rows over 1,093 tasks and 1,326 updates; exposure also varies by environment.Aggregate token budgets are similar, but task coverage and per-environment exposure differ.Source: §X.6 · p. 72
  • Table 53 summarizes reference-policy error density and observed changes averaged across three repair arms; the association does not establish a budget mechanism.The nominal exact McNemar tests distinguish repair-versus-success-only comparisons from the reflective-versus-action-only comparison.Source: Table 53 · p. 73, Table 54 · p. 73
  • After Holm adjustment across 18 tests, action-only and reflective WebShop-lite gains remain significant (adjusted p = 0.0115 and 0.0080), as do all three TextQuest losses.The preventive WebShop gain does not survive adjustment (p = 0.5663); this sensitivity does not address seed replication or causal identification.Source: §X.6 · p. 73
  • On real ALFWorld and ScienceWorld, the preventive-repair arm solves 1/134 (0.75%) and 1/294 (0.34%) tasks, versus the base policy’s 8/134 (5.97%) and 29/300 (9.67%).Six ScienceWorld run errors are excluded from the repair arm’s denominator; this comparison concerns the preventive-repair arm alone.Source: §X.6 · p. 73
Loading 2609.40111v1…