PaperlayerView exact revision on arXiv

Source-linked AI summary

From Anomalies to Failures: Constructing Causal Error Graphs for Agentic Trace Diagnosis

Shu-Xun Yang, Yidong Wang, Zhuoer Feng, Bosi Wen, Jiayi Gui, Dayong Yang, Wenbo Yu, Haoke Zhang, Jie Tang, Cunxiang Wang

arXiv:2609.32514v1cs.AI

TL;DR

Long agentic traces make failures difficult to diagnose, while existing methods often conflate anomalies, errors, and failures or leave error propagation unmodeled. CEG-Agent builds typed causal error graphs through tool-augmented inference; on CEG-Bench, it outperforms strong agentic baselines under both relaxed and exact criteria.

  • Problem

    Existing trace diagnosis often conflates anomalies, process-level errors, and task failures and lacks structured accounts of how errors propagate into failures.

    Source: §1 · p. 1

  • Method

    CEG-Agent uses a tool-augmented agentic framework to construct typed Causal Error Graphs linking execution events, anomalies, errors, and failures through causal relations.

    Source: §1 · p. 2, §3 · p. 3, §3.3 · p. 4

  • Results

    CEG-Agent outperforms strong agentic baselines by an average of +9.0 points under semantically relaxed CEG evaluation and +4.9 points under structurally exact CEG evaluation.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The resulting graphs and repair-value estimates help developers understand failure formation and prioritize interventions for agent improvement.

    Source: §1 · p. 2

    On SWE-bench, repair values are not yet sufficiently calibrated for single-shot code patching, and priority-error filtering can omit contextual errors or over-focus on an incomplete local fix.

    Source: §5.4 · p. 9

Abstract

from arXiv · show

LLM-driven agents are increasingly deployed in complex applications, where long agentic traces make failures difficult to diagnose. Existing trace diagnosis methods often conflate anomalies, errors, and failures, making diagnostic targets ambiguous; they also lack structured modeling of how causally relevant errors propagate and amplify into final task failures, resulting in unreliable failure attribution. To address these problems, we propose CEG-Agent, a tool-augmented agentic framework for causal diagnosis of agentic traces. Specifically, CEG-Agent introduces an explicit taxonomy of anomalies, errors, and failures, and constructs Causal Error Graphs (CEGs), a unified typed representation that links execution events, diagnostic nodes, and failure outcomes through causal relations. To evaluate causal trace diagnosis, we further construct CEG-Bench, a fully agent-annotated benchmark with high-confidence, consensus-derived CEG annotations obtained through an Adversarial Agentic Adjudication Protocol (AAAP). We validate the resulting annotations against an expert-curated human gold set, which shows close agreement with the automatic annotations. Experiments on CEG-Bench demonstrate that CEG-Agent achieves state-of-the-art performance under both semantically relaxed and structurally exact evaluation criteria. Our code is publicly available.

1 INTRODUCTION

CEG-Agent distinguishes anomalies, causal errors, and task failures, then models their relationships in causal graphs. On CEG-Bench, it outperforms strong agentic baselines under both relaxed and exact evaluation criteria.

1 INTRODUCTION

  • Long agentic traces can fail through process-level errors that propagate and amplify, while existing diagnosis often conflates those errors with surface anomalies and final failures.Flat classification and step scoring provide limited accounts of causal structure and error propagation.Source: §1 · pp. 1–2
  • CEG-Agent separates anomalies, process-level errors, and outcome-level failures in typed causal graphs that link trace events to error propagation and task failures.The framework also produces repair-value estimates to help developers prioritize interventions.Source: §1 · p. 2
  • CEG-Bench retains 100 consensus-annotated traces from 150 candidates across BrowseComp, Tau2Bench, and SWE-bench using adversarial agentic adjudication.Two heterogeneous agentic annotators cross-examine and refine each graph until both accept it without remaining objections.Source: §1 · p. 2
  • CEG-Agent gains an average of +9.0 points under semantically relaxed CEG evaluation and +4.9 points under structurally exact CEG evaluation over strong agentic baselines.The reported gains concentrate on distinguishing anomalies from causally relevant errors and constructing error-propagation chains.Source: §1 · p. 2

2 RELATED WORK

Prior work evaluates agent task completion, tool use, and intermediate process quality, while trace diagnosis has begun to analyze execution trajectories. CEG connects process-level errors to execution events and task failures through typed causal relations.

2 RELATED WORK

  • Agent benchmarks assess task completion and tool use, while process-oriented evaluations and LLM judges assess intermediate reasoning, actions, and step quality.Examples include BrowseComp (Wei et al., 2025), Tau2Bench, and SWE-bench.Source: §2 · pp. 2–3
  • Trace diagnosis methods such as TraceSIR (Yang et al., 2026) analyze execution traces, but often focus on outcomes, local step quality, or flat error categories.The paper identifies limited support for distinguishing anomalies, causal errors, and failures or explaining how errors propagate.Source: §2 · p. 3
  • CEG adapts causal-graph and root-cause-analysis ideas by grounding process-level errors in execution events and linking them to failures through typed causal relations.CEG-Agent also specializes tool-augmented and critic-based LLM frameworks for structured diagnosis and repair-value estimates.Source: §2 · p. 3

3 METHOD

CEG-Agent formulates trace diagnosis as constrained inference over typed heterogeneous causal graphs. Its tool-augmented procedure builds and validates these graphs, grounding diagnostic nodes in trace events and connecting errors to failures.

3 METHOD

  • CEG-Agent frames trace diagnosis as constrained inference over typed heterogeneous causal graphs, with graph construction, validation, semantic refinement, and repair-value estimation exposed as tools.The method section presents the modeling and inference principles, with implementation details deferred to Appendix A.Source: §3 · p. 3

3.1 PROBLEM FORMULATION

  • Constrained MAP estimation seeks a schema-valid graph grounded in the trace, requiring typed diagnostic categories, event grounding, and explicit causal links from errors to failures.The objective uses a tool-augmented agentic policy to model the graph probability.Source: §3.1 · p. 4

3.2 CAUSAL ERROR GRAPH

  • The CEG uses event, anomaly, error, and failure nodes with typed edges; errors are grounded in events and linked causally, while anomalies are excluded from causal attribution.Well-formed graphs also require causal connectivity to failures and distinct anomaly and error content.Source: §3.2 · p. 4

3.3 CEG AS TOOL-AUGMENTED INFERENCE

  • A top-level policy incrementally updates a diagnosis state by invoking registered tools that write to graph components under schema constraints.The loop ends with a finalize action or after at most K iterations.Source: §3.3 · pp. 4–5

3.3.1 CORE DIAGNOSIS TOOLS

  • Core tools construct ordered trace events, identify target failures, generate event-grounded error hypotheses, induce causal edges, and surface anomalies outside causal attribution.Causal edges explain how errors propagate and accumulate into failures; temporal and grounding edges are populated deterministically.Source: §3.3.1 · p. 5
  • Graph validation checks well-formedness, repairs structural violations, and surfaces semantic inconsistencies for critic-led refinement; repair-value estimation scores whether fixing an error would mitigate the primary failure.The terminal graph is projected to satisfy key schema constraints.Source: §3.3.1 · p. 5

3.3.2 AUXILIARY TOOLS

  • Auxiliary tools support trace and graph inspection, progress tracking, isolated read-only sub-agent investigation, graph visualization, and finalization with schema projection.Visualization serializes CEGs into DOT, HTML, or SVG for inspection of error-propagation chains.Source: §3.3.2 · p. 6

4 CEG-BENCH

CEG-Bench provides consensus-annotated traces through adversarial agent adjudication, with annotations audited against expert judgments. Its evaluation uses semantically relaxed and structurally exact criteria for comparing predicted graphs with consensus annotations.

4.1 TRACE COLLECTION

  • The benchmark retains 100 consensus traces from 150 failed GLM-4.6 candidates drawn equally from BrowseComp, Tau2Bench, and SWE-bench.The released set contains 29 BrowseComp, 36 Tau2Bench, and 35 SWE-bench traces, preserving their original message format.Source: §4.1 · p. 6

4.2 BENCHMARK CONSTRUCTION

  • AAAP retains a trace only after iterative adversarial graph revision reaches dual-adversary consensus; otherwise, the trace is discarded.One drafter and one independent reviewer construct and refine candidate graphs, while two adversaries stress-test them.Source: §4.2 · p. 6
  • 71.5 overall CEG-Sim agreement with the human gold set, including 88.1 for anomalies and 76.3 each for errors and failures.Eight experts independently cross-annotated all 100 traces; AAAP annotations are the primary benchmark reference, while the human set audits their reliability.Source: §4.2 · p. 7

4.3 EVALUATION CRITERIA

  • SRE aligns predicted and gold nodes using type-specific similarity, while SEE requires exact node and typed-edge matching.SRE aggregates scores across structural, textual, and utility dimensions; SEE averages five discriminative dimensions.Source: §4.3 · pp. 7–8

5 EXPERIMENTS

CEG-Agent outperforms same-backbone agentic baselines on both semantic and structural graph evaluation. Its gains extend across backbones and trace types, while repair feedback results vary by task and remain preliminary.

5.1 OVERALL PERFORMANCE

  • CEG-Agent achieves the best total SRE and SEE scores for both backbone families, with gains over +Process of +2.0/+1.2 points for Claude and +13.6/+6.5 for GPT-5.5.The +Process baselines already receive deterministic event segmentation, so the reported gains are not merely attributed to improved event parsing.Source: §5.1 · p. 8

5.2 FINE-GRAINED ANALYSIS

  • On substantive non-empty traces, CEG-Claude gains +5.6 SRE/+1.4 SEE and CEG-Codex gains +13.0 SRE/+6.2 SEE over same-backbone baselines.The analysis also reports closer matching to gold graph scale, stronger recall on difficult and propagated errors, and greater advantage on medium-to-large traces; even the best system reaches only 51.6 SRE and 20.5 SEE.Source: §5.2 · p. 8

5.3 CROSS-BACKBONE GENERALIZATION

  • Qwen3.7-Max reaches 43.7, well above the strongest Codex baseline at 32.9; this supports the claim that the validator-guided construction loop offers a backbone-agnostic structural scaffold.Qwen3.7-Max outperforms GPT-5.5 and Gemini-3.1-Pro on structurally exact reconstruction despite lower semantic similarity, indicating that fluent diagnostic descriptions and exact causal structure are distinct capabilities.Source: §5.3 · p. 9

5.4 CLOSED-LOOP REPAIR WITH CEG FEEDBACK

  • On Tau2Bench, all CEG-derived repair hints beat generic retry feedback, and the failure-level hint gives the largest gain.The probe compares generic retry feedback with failure, causal-error, and repair-prioritized hints; the preferred feedback granularity differs by task.Source: §5.4 · p. 9
  • Repair results are directional rather than conclusive because per-source samples are small, and absolute BrowseComp success remains low.On SWE-bench, priority-error filtering hurts; the authors suggest it may omit contextual errors or over-focus on an incomplete local fix.Source: §5.4 · p. 9

6 CONCLUSION

The paper presents CEG-Agent for causal diagnosis of agentic execution traces and CEG-Bench for systematic evaluation. It reports that CEG-Agent outperforms strong agentic baselines under both semantic and structurally exact criteria.

6 CONCLUSION

  • CEG-Agent distinguishes anomalies, process-level errors, and outcome-level failures, linking events and diagnostic nodes in typed Causal Error Graphs.The graphs represent causal relationships among trace events, errors, and failures.Source: §6 · p. 9
  • CEG-Bench supplies consensus CEG annotations produced through AAAP and audited against the expert-curated CEG-Bench-Gold reference.The paper describes its annotations as agent-constructed and reports close agreement with the human reference.Source: §6 · p. 9, AI USE STATEMENT · p. 10
  • The authors characterize CEG-Agent’s visualizable propagation structures and repair-oriented recommendations as actionable diagnostic evidence for debugging agent executions and improving future behavior.This is the paper’s stated practical conclusion.Source: §6 · p. 9

A CEG-AGENT IMPLEMENTATION DETAILS

The implementation normalizes traces, equips an agentic runtime with diagnosis tools, and combines deterministic schema repair with LLM-based semantic refinement. It also estimates repair value and packages graphs and execution records for downstream evaluation.

A CEG-AGENT IMPLEMENTATION DETAILS

  • Trace normalization converts provider messages into ordered thought–action–observation turns, preserving the initial task specification and subsequent feedback or observations.The implementation overview also identifies a registered tool set, runtime, validator, critic loop, repair-value function, delegation interface, and output bundle.Source: §A.1 · p. 11, A CEG-AGENT IMPLEMENTATION DETAILS · p. 11

A.2 TOOL SET

  • Fourteen registered tools update diagnosis state or provide inspection, memory, delegation, visualization, and repair-value functions.The repair-value tool can be disabled, and the number of critic refinement rounds is configurable.Source: §A.2 · p. 12, Table 5 · p. 12

A.3 AGENTIC RUNTIME

  • At each runtime iteration, the LLM receives the conversation history, diagnosis state, and tool specifications, then tool calls update state until finalization or the iteration budget.The experimental iteration budget is K = 24; after exit, schema projection is applied even if the model did not explicitly finalize.Source: §A.3 · pp. 12–13

A.4 SCHEMA PROJECTION

  • The validator separates mechanically repairable structural violations from semantic issues that require trace-level judgment.Deterministic projection removes or regenerates invalid edges and recomputes structural roles; the projection is idempotent.Source: §A.4 · p. 13

A.5 LLM CRITIC LOOP

  • For semantic issues, an LLM critic proposes compact patches rather than rebuilding the graph, and projection follows each patch.The critic budget defaults to C = 2; local patches are described as auditable and as limiting the potential damage of a single revision.Source: §A.5 · pp. 13–14

A.6 COUNTERFACTUAL REPAIR VALUATION

  • Repair valuation estimates whether correcting an error alone would avoid or substantially mitigate the primary failure, combining graph-structural and lexical signals.The score rewards counterfactual severance, root-like role, amplification, and confident descriptions while penalizing propagated symptoms with unresolved upstream causes; the gold annotations use graded signals, while inference uses binary surrogates for severance and upstream hazard.Source: §A.6 · pp. 14–15

A.7 SUB-AGENT DELEGATION

  • Sub-agents answer narrow diagnostic questions using restricted, effectively read-only tools, and return findings, evidence, and suggested patches for the parent to consider.The parent agent decides whether to apply a proposed patch.Source: §A.7 · p. 15

A.8 OUTPUT BUNDLE

  • Each trace output bundle contains the final graph, agentic steps, call-level token and latency logs, metadata, console log, and available graph visualizations.These bundles are the units used to compute benchmark-level statistics and metrics.Source: §A.8 · p. 15

B CEG-BENCH DATASET ANALYSIS

CEG-Bench retains traces only when two adversarial annotators certify a graph, yielding a 100-trace benchmark with diverse causal structures and source profiles. Its distributions show frequent propagated errors but few errors with high counterfactual repair value.

B CEG-BENCH DATASET ANALYSIS

  • Across 100 traces, CEG-Bench contains 3,488 events, 391 errors, 125 failures, 253 anomalies, and 4,540 typed edges, including 1,152 non-temporal edges.The four-role AAAP separates graph proposal and refinement from adversarial certification; both adversaries must approve a trace, and ambiguous cases are rejected.Source: B CEG-BENCH DATASET ANALYSIS · p. 16

B.1 PER-SOURCE SIZE DISTRIBUTIONS

  • The three source benchmarks differ substantially: SWE-bench traces reach 250 events, while BrowseComp averages 6.2 errors per trace versus 2.0 for Tau2Bench.Tau2Bench has a median of 14 events and SWE-bench a median of 35; the longest SWE-bench trace is 50 times the shortest Tau2Bench trace.Source: §B.1 · p. 16

B.2 VOCABULARY DISTRIBUTIONS

  • Planning (19.7%), evidence integration (18.4%), execution (16.9%), and self-evaluation (15.6%) account for 70.6% of errors; completion and correctness account for 76.8% of failures.All eight error mechanisms appear; completion (42.4%), correctness (34.4%), and constraint failures (13.6%) together make up 90.4% of failures, while PROPAGATED errors are the most common role.Source: §B.2 · p. 17

B.3 EDGE STRUCTURE OF THE CAUSAL SUBGRAPH

  • The causal subgraph has 749 causal edges, averaging 7.5 per trace; AMPLIFIES edges are roughly 1:4.4 as common as CAUSES edges.Most causal structure flows through error-propagation chains ending in CONTRIBUTES-TO edges; AMPLIFIES is reserved for errors that meaningfully magnify another error’s impact.Source: §B.3 · p. 17

B.4 REPAIR-VALUE DISTRIBUTION

  • Only 13 errors (3.3%) receive HIGH repair value, versus 244 (62.4%) LOW and 134 (34.3%) MEDIUM.HIGH requires a strong counterfactual bottleneck for the primary failure and no hard override; all four environment errors receive HIGH values.Source: §B.4 · p. 18

C CEG EVALUATION PROTOCOLS

The appendix describes two released reference evaluation protocols: a semantically relaxed evaluator and a structurally exact evaluator.

C.1 SEMANTICALLY RELAXED CEG EVALUATION

  • The relaxed evaluator aligns predicted and gold nodes by bipartite matching, translates identifiers, and scores precision, recall, and F1 across eight dimensions.Its dimensions are grouped into structural, textual and sparse, and utility tiers, with total weights of 0.60, 0.30, and 0.10, respectively.Source: §C.1 · pp. 18–19
  • For the relaxed evaluator, the main-chain score compares canonical root-to-primary-failure paths using node and edge overlap and a length-normalized longest common subsequence.The consistency dimension also checks graph invariants, including reachability.Source: §C.1 · p. 19

C.2 STRUCTURALLY EXACT CEG EVALUATION

  • The structurally exact evaluator uses no fuzzy matching; it resolves errors by event-grounding and mechanism tuples and failures by type bucket, then scores seven dimensions.Its composite is the unweighted mean of five model-discriminative dimensions; parser-determined event and EVENT-NEXT scores are reported separately.Source: §C.2 · pp. 19–20

D HUMAN GOLD VALIDATION OF CEG-BENCH

Against the expert-curated human reference, CEG-Agent retains the best within-family composite SRE and SEE rankings for both backbone families. The appendix also reports AAAP agreement with human annotations and performance patterns across source benchmarks and trace-complexity levels.

D HUMAN GOLD VALIDATION OF CEG-BENCH

  • Eight domain experts annotated and verified all 100 traces, resolving disagreements through discussion into consensus graphs; the gold set serves as an audit rather than a scalable replacement.The experts marked events, failures, anomalies, causally relevant errors, mechanisms, roles, and causal edges.Source: D HUMAN GOLD VALIDATION OF CEG-BENCH · p. 20
  • AAAP achieves CEG-Sim 71.5 against the human gold, with SRE agreement of 88.1 on anomalies and 76.3 each on errors and failures.SEE agreement is lower on exact causal wiring: 20.1 for causes and 36.2 for contributes_to; SEE agreement rises from 43.0 on small traces to 52.4 on large traces.Source: D HUMAN GOLD VALIDATION OF CEG-BENCH · p. 20, Table 9 · p. 21
  • Against the human gold, CEG-Agent has the best composite SRE and SEE scores within both backbone families.Absolute scores are lower than against AAAP, but the within-family ranking is preserved.Source: D HUMAN GOLD VALIDATION OF CEG-BENCH · p. 20, Table 10 · p. 21, Table 11 · p. 22

E.1 DETAILED EXPERIMENTAL SETUP

  • The experimental comparison uses raw coding-agent baselines, baselines with parser-segmented events and EVENT-NEXT edges, and CEG-Agent with the same backbone on the same 100 traces.The evaluation reports semantically relaxed CEG-Sim and structurally exact CEG-Exact scores; the appendix supplies detailed setup and analyses.Source: E EXPERIMENTAL ANALYSES · pp. 21–22, §E.1 · p. 22

E.2 PER-SOURCE PERFORMANCE

  • On SWE-bench and BrowseComp, CEG-Claude beats Claude Code (+Process) by +9.4 and +4.4 SRE points, and +6.2 and +1.7 SEE points, respectively.For GPT-5.5, CEG-Codex leads SRE on all three sources and SEE on two of three, including a +21.8-point SRE gain over Codex (+Process) on Tau2Bench.Source: §E.2 · p. 22, Table 12 · p. 23

E.3 ROBUSTNESS TO TRACE COMPLEXITY

  • On medium traces, CEG-Codex scores 15.0 SEE versus 3.9 for Codex (+Process), an +11.1-point gain.CEG-Claude also exceeds the +Process baseline on medium and large traces; CEG-Agent’s benefit is greatest when diagnosis requires sustained causal graph construction.Source: §E.3 · p. 23, Table 13 · p. 23

E.4 DIMENSION-LEVEL DIAGNOSIS

The dimension-level analysis finds that CEG-Agent’s gains center on recovering causal structure, separating anomalies from errors, and diagnosing traces with substantive causal graphs. It also identifies limits in failure-link coverage and repair-value calibration, and shows that empty graphs and hard traces affect evaluation.

E.4 DIMENSION-LEVEL DIAGNOSIS

  • CEG-Agent improves causal-chain and exact error-grounding dimensions, while Claude’s textual failure scores trade off against detailed descriptions and exact failure-link coverage remains a gap.The main remaining structural gap is SEE contributes_to: richer error graphs do not always include explicit links from every error to the failure node.Source: §E.4 · pp. 23–24

E.5 GOLD-EMPTY TRACES AND ABSTENTION BEHAVIOR

  • On 85 non-empty traces, CEG-Agent raises composite scores over same-backbone baselines; empty-trace abstention can mechanically reward systems that return empty graphs.Claude Code (+Process) abstains on 11/15 gold-empty traces, while CEG-Claude usually emits a diagnosis; the authors identify a calibrated empty-graph option as a possible improvement.Source: §E.5 · pp. 24–25

E.6 GRAPH-SIZE AND STRUCTURAL CALIBRATION

  • Baselines under-generate causal structure, whereas CEG-Agent produces graphs closer to gold scale, supporting stronger structural-dimension scores.Codex variants produce 1.64–1.82 causal-subgraph edges per trace versus a gold average of 7.49.Source: §E.6 · p. 26

E.7 PER-MECHANISM AND PER-ROLE RECALL

  • CEG-Agent improves recall on most mechanisms and all three structural roles, with its largest gains on cumulative mechanisms and PROPAGATED errors requiring causal-chain reconstruction.Representation, evidence integration, and planning require evidence aggregation across multiple trace steps; Claude’s main exceptions include omission and the four-example environment category.Source: §E.7 · pp. 26–27

E.8 FAILURE-TYPE COVERAGE

  • CEG-Agent broadens failure-type coverage, but every system misses the three gold safety failures, suggesting trace-only evidence may be insufficient for safety-specific diagnosis.Most systems overuse completion when uncertain; Codex (+Process) emits no constraint failures across the benchmark.Source: §E.8 · pp. 27–28

E.9 REPAIR-VALUE CALIBRATION

  • CEG-Claude provides useful within-trace repair-priority ordering (Kendall’s τ = +0.400), while CEG-Codex’s τ = 0.0 despite high bucket accuracy from mostly predicting LOW.CEG-Claude over-predicts HIGH (20.0% versus 3.3% gold prevalence); fine-grained calibration is left for future work.Source: §E.9 · pp. 28–29

E.10 INTER-SYSTEM AGREEMENT AND IRREDUCIBLY HARD TRACES

  • All six systems score SEE = 0 on 19 traces, including 15 gold-empty cases and four difficult non-empty BrowseComp traces involving global evidence integration.Tau2Bench supplies eight of the ten traces where some system reaches maximum SEE of at least 0.58; the authors characterize it as structurally easier than SWE-bench and BrowseComp.Source: §E.10 · p. 29

E.11 PER-BACKBONE DIAGNOSTIC CAPABILITY PROFILE

Diagnostic profiles vary by backbone: Claude is balanced, while other models show different strengths in exact reconstruction, local error identification, causal edges, and anomaly coverage. Cost and tool-use profiles also vary, with repeated trace inputs and provider latency shaping runtime.

E.11 PER-BACKBONE DIAGNOSTIC CAPABILITY PROFILE

  • Claude-Opus-4.7 is the most balanced diagnostician; GLM-5.1 is strong on exact reconstruction and failure typing, while GPT-5.5 identifies local errors better than it wires them into causal structure.Gemini-3.1-Pro scores well on SRE edges but lowest on SEE contributes_to; Qwen3.7-Max has the lowest SRE anomaly score but best SEE cause and contributes_to scores.Source: §E.11 · pp. 29–31

E.12 COST AND TOOL-USE PROFILE

  • Across backbones, normalized traces are repeatedly attached during diagnosis, making input tokens dominant; wall-clock time also depends on provider latency, and optional repair and delegation use varies.Claude-Opus-4.7 and GPT-5.5 take about 13–14 iterations and roughly 19 LLM calls per trace; CEG-Claude costs $225 over the benchmark versus $300 for Claude Code (raw) and $244 for Claude Code (+Process).Source: §E.12 · p. 31

F CLOSED-LOOP REPAIR STUDY

The closed-loop probe tests whether CEG-derived feedback helps a previously failed agent recover, with outcomes varying across interactive and single-shot settings. Representative visualizations also show how the event sequence and causal graph can be inspected together.

F CLOSED-LOOP REPAIR STUDY

  • On Tau2Bench, all CEG-derived hints beat generic retry, with the failure-level hint scoring 91.7 versus 77.8; feedback gains are strongest when an agent can act interactively.The failure hint exposes the outcome-level failure, while error hints expose causal error nodes and their mechanisms, roles, and descriptions.Source: F CLOSED-LOOP REPAIR STUDY · p. 32
  • In single-shot settings, BrowseComp scores highest with error hints (17.2), while SWE-bench priority-error filtering hurts; missing retrieval evidence and removed contextual errors constrain recovery.On SWE-bench, concise failure and error hints each solve one additional instance over generic retry, but priority filtering can focus the model on an incomplete local fix.Source: F CLOSED-LOOP REPAIR STUDY · p. 32, Table 28 · p. 33
  • The study is directional rather than statistically decisive: it covers 29–36 traces per benchmark, with the strongest consistent gains in interactive Tau2Bench.The authors leave larger-scale optimization, feedback formatting, and more robust repair-value calibration to future work.Source: F CLOSED-LOOP REPAIR STUDY · p. 33

F.1 CASE STUDIES

  • Representative visualizations align chronological events with induced causal graphs, making errors, anomalies, propagation chains, and failures inspectable in one view.The examples are presented as human-readable diagnostic artifacts alongside machine-evaluable CEGs.Source: §F.1 · pp. 33–34, Figure 5 · p. 33, Figure 6 · p. 34

G LIMITATIONS

The paper notes limits in annotation certainty, benchmark coverage, repair-value calibration, and breadth of baseline comparisons.

G LIMITATIONS

  • AAAP annotations are automatic rather than exhaustive human labels and may retain shared annotator biases; some traces have empty CEGs when causal errors cannot be inferred without outcome feedback.Source: G LIMITATIONS · p. 34
  • CEG-Bench covers 100 traces from three benchmarks, all sourced from GLM-4.6, and excludes broader deployment settings, safety-critical domains, and multi-agent environments.Source: G LIMITATIONS · pp. 34–35
  • Repair-value estimation remains preliminary: deterministic graph-structural scores define gold values, while inference-time LLM buckets can be wording- and confidence-sensitive.Source: G LIMITATIONS · p. 35
  • Baseline comparisons cover Claude Code and Codex, while the cross-backbone study uses five representative LLMs, leaving broader comparisons for future work.Source: G LIMITATIONS · p. 35

Related papers

Loading 2609.32514v1…