PaperlayerView exact revision on arXiv

Source-linked AI summary

DAAF: From Failure Localization to Editable System Assets in LLM Agents

Xiaoyang Yuan, Qi Liu, Yubin Ruan, Xinyi Mou, Zhuomeng Zhang, Wenjin Wang, Hanying Jiao, Di Wu, Mingye Xu, Yi Bin, Ke Feng, Zixun Sun

arXiv:2609.32498v1cs.AI

TL;DR

Failure localization can identify where an agent error appears without identifying which persistent, editable asset to change. DAAF learns replacement effects from controlled task replays and uses them to diagnose new executions; on held-out Telecom tasks, it achieves 80.72% attribute Hit@1 and 62.65% failure recovery with 3.23% clean-task regression.

  • Problem

    Failure-localization methods identify agents, steps, or modules, but do not necessarily identify an editable, versioned system asset whose change is expected to improve the task outcome.

    Source: §1 · pp. 1–2

  • Method

    DAAF combines weak-signal failure detection with component-type-conditioned predictions of registered replacement effects learned from controlled replays, and shares evidence across compatible requests.

    Source: §1 · p. 2

  • Results

    80.72% attribute Hit@1 and 62.65% failure recovery with 3.23% clean-task regression were achieved on held-out Telecom tasks.

    Source: §1 · p. 2, §5 · p. 9

  • Takeaways & Limitations

    Intervention-grounded attribute attribution connects failure localization to executable system repair on held-out Telecom tasks.

    Source: §1 · p. 2

    The controlled benchmark results are not sufficient evidence for unsupervised deployment in safety-critical settings.

    Source: ETHICS STATEMENT · p. 10

Abstract

from arXiv · show

Deployed LLM agents increasingly rely on persistent, versioned system assets such as routing rules, knowledge segments, prompt instructions, and reusable skills. Failure-localization methods can identify where an error manifests in an agent or execution trace, but repair requires a different decision: which editable system asset should be changed, and is that change expected to improve the task outcome? We study this gap through component-attribute failure attribution, where diagnosis targets versioned, addressable items rather than execution locations. We propose the Detection-Aware Attribution Framework (DAAF), which learns the effects of valid attribute replacements and amortizes this intervention evidence into deployment-time diagnosis. DAAF combines sparse and noisy failure signals to decide whether intervention is warranted, learns component-type-conditioned replacement effects from controlled replays evaluated by executable task outcomes, and shares supervision across requests with compatible intervention responses. At diagnosis time, DAAF uses only the observed execution, registered candidates, and available failure signals; it requires neither counterfactual replay nor task reward and returns no_change, a repair target, or an unresolved decision when evidence is insufficient. On held-out tau^2-bench Telecom tasks, DAAF achieves 80.72% attribute Hit@1, recovers 62.65% of failed executions while limiting clean-task regression to 3.23%, and reaches 71.93% overall task success. These results show that intervention-grounded attribute attribution can connect failure localization to executable system repair.

1 INTRODUCTION

DAAF shifts diagnosis from locating execution failures to selecting versioned, editable assets whose replacement effects can be evaluated at task level. It learns from controlled interventions to make repair decisions for new executions.

1 INTRODUCTION

  • Failure localization identifies agents, steps, or modules, but these labels need not identify a versioned, addressable object that can be changed and reevaluated.Persistent assets may affect multiple requests, and edits can be beneficial, neutral, harmful, or interaction-dependent.Source: §1 · p. 1
  • Repair requires evidence about what happens when persistent objects are modified, motivating the use of intervention outcomes collected during training to predict useful changes for new executions.The challenges include incomplete failure signals, heterogeneous interacting edits, and selectively transferring evidence across compatible requests.Source: §1 · p. 2
  • DAAF targets the expected task-level effect of valid registered replacements, combining weak-signal detection, type-conditioned effect prediction, interaction-aware replays, and cross-request consistency.At diagnosis, it returns no_change, a registered repair, or an unresolved outcome without task evaluation or counterfactual replay.Source: §1 · p. 2
  • 80.72% attribute Hit@1, 62.65% failure recovery, and 3.23% clean-task regression show DAAF connecting attribute diagnosis with executable repair on held-out Telecom tasks.The framework also combines those results with 71.93% overall task success.Source: §1 · p. 2

2 RELATED WORK

Prior work addresses failure detection, execution-level attribution, intervention-based debugging, and system optimization. DAAF instead predicts the effects of changing registered persistent assets, then uses those estimates to inform diagnosis.

2 RELATED WORK

  • Weak-signal methods combine noisy sources to detect failure, but DAAF uses intervention outcomes—not weak failure evidence alone—to supervise repair attribution.MSWSD combines signals with different availability and reliability to decide whether an execution enters repair.Source: §2 · p. 2
  • Execution-level attribution and counterfactual debugging localize causes or decisions, whereas DAAF targets versioned, addressable assets whose registered replacements can be evaluated.DAAF uses learned replacement effects at diagnosis without replaying candidate edits for each new request.Source: §2 · p. 3
  • System-optimization methods improve prompts or pipeline variables once an objective is available; DAAF addresses the preceding decision of which registered asset has evidence supporting a change.A selected asset can subsequently be passed to an optimizer or human maintainer to construct a replacement.Source: §2 · p. 3

3 DAAF: LEARNING WHICH ATTRIBUTES TO CHANGE

DAAF frames diagnosis as deciding whether to intervene and which registered asset change is expected to improve task outcomes. It learns from controlled replays during training and makes diagnosis without replay or task reward.

3 DAAF: LEARNING WHICH ATTRIBUTES TO CHANGE

  • DAAF has three learned parts: MSWSD estimates failure evidence, CTCA predicts signed replacement effects, and joint learning shares repair evidence across compatible requests.Their outputs support an abstaining repair decision.Source: §3 · p. 3

3.1 PROBLEM SETUP AND REGISTERED EDITS

  • Attributes are stable, addressable registered items with validated replacements, grouped into Node, Harness, Knowledge, Prompt, and Skill roles.The taxonomy need not be exhaustive, but candidate attributes must be addressable and replaceable through a validated interface.Source: §3.1 · pp. 3–4

3.2 WEAK-SIGNAL FAILURE DETECTION

  • MSWSD combines weak failure signals—including trajectory anomalies, user feedback, and disagreement across matched repeated runs—while explicitly representing unavailable observations.Its estimate determines whether an execution should enter the repair stage; noisy seed annotations and labeling functions provide soft failure targets.Source: §3.2 · p. 4

3.3 INTERVENTION-SUPERVISED ATTRIBUTE EFFECTS

  • CTCA learns signed replacement effects from controlled replays evaluated by executable task outcomes, estimating each edit’s marginal effect across composed replacement settings.Positive effects indicate likely improvement and negative effects indicate harm; unavailable comparisons are omitted rather than treated as zero.Source: §3.1 · p. 4, §3.3 · p. 5

3.4 JOINT LEARNING AND REPAIR DECISION

  • Joint learning shares evidence only across requests with compatible system versions, candidate sets, replacement policies, evaluators, and intervention responses.Diagnosis returns no_change when failure evidence is low, an unresolved decision when no candidate has enough positive support, or the strongest supported registered edit; it uses no task reward or replay.Source: §3.4 · pp. 5–6

4 EXPERIMENTS

The experiments evaluate whether DAAF’s asset-level diagnoses lead to executable repair on held-out Telecom tasks, alongside analyses of intervention context, transfer, evidence sharing, and conventional localization. The main results connect diagnosis to task recovery and show that replacement effects vary with context.

4.1 EXPERIMENTAL SETUP

  • The evaluation uses τ 2-bench Telecom for executable repair and Who&When only for auxiliary trajectory-level localization, which lacks editable system assets.The held-out Telecom set contains 114 official tasks, with 83 natural failures and 31 natural successes; repair metrics use three matched repeats.Source: §4.1 · pp. 6–7, §4 · p. 6

4.2 ATTRIBUTE DIAGNOSIS AND EXECUTED REPAIR

  • DAAF reaches 80.72% Attribute Hit@1, repairs 62.65 ± 3.19% of original failures, and achieves 71.93 ± 1.52% Overall Success on held-out Telecom tasks.It compares with 40.96% Attribute Hit@1 and 35.74 ± 4.56% recovery for the component-aware LLM; unchanged-system reruns recover 12.05 ± 3.62%.Source: §4.2 · p. 7

4.3 INTERACTION-AWARE ATTRIBUTION AND TRANSFER

  • Composed settings make 31.02% of registered replacements change task outcomes, versus 11.36% for standalone replacements, supporting context-dependent effect modeling.Standalone probing misses 88.54% of task–attribute pairs with measurable effects somewhere in their replacement chain.Source: §4.3 · p. 8
  • Training-derived conditional effects preserve the held-out recovery ordering; adding Prompt Instruction raises Failure Recovery from 62.65% to 81.53%.The gain is 18.88 percentage points (95% CI [10.84, 26.91]); other candidates’ recovery differences are smaller and their intervals include zero.Source: §4.3 · p. 8, Table 3 · p. 8

4.4 SUPPORTING ANALYSES

  • Zero-filling unavailable failure signals lowers natural-cohort AUPRC from 75.77% to 71.31% and AUROC from 56.06% to 44.58%.Mean imputation matches the missingness-aware representation on both reported splits, so the evidence supports distinguishing unavailable signals from observed zeros, not superiority over all imputation.Source: §4.4 · p. 8
  • Intervention-response neighborhoods reach 62.63% held-out agreement, versus 60.09% for request-similarity neighborhoods and 58.59% for random edges.The intervention-based construction has the highest agreement on all ten attribute hold-out splits.Source: §4.4 · p. 8

4.5 CONVENTIONAL FAILURE LOCALIZATION

  • On Algorithm traces, the DAAF frontend reaches 48.02% Step and Joint accuracy versus AgentDebugX’s 42.06%; on Handcrafted traces, it reaches 25.86% Step and 24.14% Joint versus 18.97% and 17.24%.This is an auxiliary localization check, not a model-matched executable repair evaluation.Source: §4.5 · p. 9, Table 4 · p. 9

5 CONCLUSION

DAAF links failure localization to repair decisions over persistent, editable system assets by learning replacement effects from controlled interventions. Its Telecom results connect attribution to executable repair, while the formulation assumes registered candidates.

5 CONCLUSION

  • On held-out Telecom tasks, DAAF achieves 80.72% attribute Hit@1, recovers 62.65% of failed executions, and limits clean-task regression to 3.23%.Controlled replay analysis finds non-zero replacement effects in 31.02% of composed settings versus 11.36% under isolated edits.Source: §5 · p. 9
  • DAAF learns the effects of registered replacements from controlled interventions as offline supervision, requiring no per-candidate replay or task reward at diagnosis time.The framework connects weak failure evidence to versioned, editable repair targets.Source: §5 · p. 9
  • The formulation assumes a registered candidate set; automatic asset discovery and replacement generation remain future work.Source: §5 · p. 9

ETHICS STATEMENT

The work studies failure diagnosis and repair in controlled benchmark environments, not live deployment or human-subject experiments. Its results are not sufficient evidence for unsupervised use in safety-critical settings.

ETHICS STATEMENT

  • The study uses controlled benchmark environments and involves neither live deployment nor human-subject experiments.Source: ETHICS STATEMENT · p. 10
  • Interventions modify registered assets only within executable evaluation environments and do not affect production systems or real users.Source: ETHICS STATEMENT · p. 10
  • The authors caution that these results are insufficient evidence for unsupervised deployment in safety-critical settings.They recommend validation, access control, monitoring, and human oversight for practical deployment.Source: ETHICS STATEMENT · p. 10

AI USE STATEMENT

The paper reports its use of generative AI tools and details how failure signals, replay evidence, and registered assets support DAAF’s training pipeline. The appendix specifies the supervision and replay procedures used to construct attribution targets.

AI USE STATEMENT

  • Generative AI assisted with literature checks, drafting, derivation checks, code, and analysis; the authors say they developed the core research design and reviewed the outputs.Language models also contribute weak annotations and serve as agents, simulators, or diagnosticians in the experimental pipeline.Source: AI USE STATEMENT · p. 10

A SIGNAL EXTRACTION AND SUPERVISION

  • Behavioral disagreement uses repeated runs from the same request and starting state, comparing normalized final answers and action sequences.When only one matched run is available, disagreement is unavailable rather than assigned a default value.Source: A SIGNAL EXTRACTION AND SUPERVISION · p. 12
  • CTCA trains on signed marginal effects from adjacent coalitions replayed under matched conditions, retaining positive, zero, and negative effects.Official executable task outcomes construct offline targets, not evaluation-time diagnostic features.Source: A SIGNAL EXTRACTION AND SUPERVISION · p. 13

B REGISTRY, INTERVENTIONS, AND ATTRIBUTION HEADS

  • The registry identifies versioned editable attributes and replacement policies, while validated replacements preserve component-specific execution constraints.The Telecom corpus uses 220 tasks, five candidates, two replacement permutations, and two matched repetitions, yielding 24 valid replays per task and 5,280 total.Source: B REGISTRY, INTERVENTIONS, AND… · pp. 13–14

C LEARNING AND OPTIMIZATION DETAILS

The section describes fixed supervision and neighborhood choices for joint training, then gives finite-sample results for detector and attribution estimates. It also outlines the training and diagnosis procedure.

C LEARNING AND OPTIMIZATION DETAILS

  • Training pairs enter the consistency neighborhood only when system version, candidate set, replacement policy, and task evaluator are compatible.The graph keeps five highest-weight neighbors per case, is symmetrized, and is frozen before joint optimization.Source: C LEARNING AND OPTIMIZATION DETAILS · p. 15

D ESTIMATION PROPERTIES AND OPTIMIZATION DETAILS

  • The detector’s sampling deviations around its mean-signal output obey a tail bound, with separate calibration error added when comparing that output to true failure probability.The attribution-estimation bound decreases as O(M^-1/2) and grows only logarithmically with the number of candidate attributes, but can be loose at small replay budgets.Source: D ESTIMATION PROPERTIES AND… · pp. 15–16
  • The diagnosis algorithm freezes training targets and neighborhoods, alternates detector and attribution updates, calibrates on development data, then returns no_change, a repair, or unresolved.The finite-budget bound does not guarantee that closely spaced effects are distinguishable, consistent with allowing unresolved outcomes.Source: D ESTIMATION PROPERTIES AND… · p. 16

D.2 PROOF OF PROPOSITION 2

The proof separates prediction error from finite-budget replay-estimation error and gives a condition for preserving candidate rankings. A descent-compatible alternating variant also bounds average update gradients.

D.2 PROOF OF PROPOSITION 2

  • If the two largest true effects differ by more than twice the total uniform error, their ranking is preserved.The total error combines predictive approximation error and finite-budget replay-estimation error.Source: §D.2 · p. 17

D.3 A DESCENT-COMPATIBLE ALTERNATING VARIANT

  • The descent-compatible alternating variant bounds average block-update squared gradient norms by 2Lmax(L0 − Linf)/T over T rounds.The result applies to the smoothed objective, or locally away from the positive-part map’s kinks.Source: §D.3 · p. 17

E EVALUATION PROTOCOL

The protocol evaluates failure detection, independently validated repair locations, and executed repair outcomes using held-out tasks and matched repeats. Its controls test whether detection alone suffices, whether composed edits reveal effects missed by isolated probing, and whether partial LLM-plus-CTCA recovers the full pipeline’s gain.

E EVALUATION PROTOCOL

  • The evaluation separates failure detection, independently validated repair labels, and executed repair metrics, with development-only calibration and matched task–repeat seeds.Repair labels are excluded from fitting and ranking; principal repair metrics average three matched repeats.Source: E EVALUATION PROTOCOL · p. 17

F.1 EXTENDED ATTRIBUTE-LOCALIZATION CONTROLS

  • The partial LLM + CTCA composition does not reproduce the gain of the full decision pipeline.Uniform candidate choice averages over accepted repair sets, so it can exceed 20% when failures admit multiple validated repairs.Source: §F.1 · p. 18, Table 8 · p. 18

F.2 DETECTION-ONLY CONTROL

  • Failure detection without learned replacement effects is insufficient for DAAF’s repair policy, with the paired interval excluding zero by a wide margin.The control holds the broader setup constant while freezing typed attribution parameters.Source: §F.2 · p. 18, Table 9 · p. 19

F.3 INTERACTION STATISTICS IN THE REPLAY CORPUS

  • Across the replay corpus, 515 task–attribute pairs exhibit measurable effects somewhere in their replacement chains, but standalone probing identifies only 59, missing 88.54%; sampled marginal contributions differ in sign across replacement orders for 472 of 1,100 targets.Source: §F.3 · p. 19

F.4 INCOMPLETE-EVIDENCE ANALYSIS

The incomplete-evidence analysis finds no demonstrated advantage of explicit masks over mean imputation, while zero-filling unavailable signals degrades detection. The neighborhood and online-cost analyses describe held-out agreement comparisons and the separation between training replay costs and online diagnosis.

F.4 INCOMPLETE-EVIDENCE ANALYSIS

  • Missingness-aware encoding and mean imputation are identical on both reported splits, while treating unavailable sources as observed zeros degrades failure detection, especially on balanced development data.The evidence therefore supports avoiding zero-fill, but not a mask advantage over mean imputation.Source: §F.4 · p. 19, Table 11 · p. 19

F.5 NEIGHBORHOOD QUALITY UNDER ATTRIBUTE HOLD-OUT

  • Intervention-response neighborhoods have the highest held-out agreement on every attribute split, testing transfer beyond the attributes used to construct the graph and beyond lexical request similarity.The comparison includes intervention-response, request-only, and random neighborhoods.Source: §F.5 · pp. 19–20, Table 12 · p. 20

F.6 ONLINE EXECUTION COST

  • The online execution count excludes the shared offline CTCA replay corpus and reflects diagnosis plus the selected-repair protocol, not enumeration of all five candidate replacements.Table 13 reports online environment executions per natural failure, with lower values better.Source: §F.6 · p. 20, Table 13 · p. 20

G.1 TASK PARTITIONS AND SUPERVISION

The experiments partition Telecom tasks and training supervision by base-task identity, then freeze model selection before evaluation. They use controlled replays for conditional effects and matched seeds for repair comparisons; Who&When is an auxiliary localization check rather than the basis for executable repair claims.

G.1 TASK PARTITIONS AND SUPERVISION

  • The evaluation uses all 114 official Telecom tasks, while a disjoint 50-task development set supports model selection and calibration; training excludes both partitions’ base-task identities.The evaluation set contains 83 natural failures and 31 natural successes; development contains 25 of each.Source: §G.1 · p. 20

G.2 OPTIMIZATION AND MODEL SELECTION

  • Thresholds and calibration are selected on the development configuration and frozen before evaluation.MSWSD uses three base-task folds; CTCA and the joint model use full-batch gradient descent with learning rate 0.2 and gradient-norm clipping at 5.0.Source: §G.2 · p. 21

G.3 MATCHED-SEED REPAIR EVALUATION

  • Principal repair comparisons use three matched execution repeats with the same task–repeat seeds across methods, enabling paired outcome analysis.Seeds are derived deterministically for each base-task and repeat pair.Source: §G.3 · p. 21

G.4 WHO&WHEN REPORTING

  • Who&When is used only as an auxiliary localization check because it exposes no editable system assets and is not a model-matched repair evaluation.Executable repair claims are established by the Telecom study.Source: §G.4 · p. 21

Related papers

Loading 2609.32498v1…