PaperlayerView exact revision on arXiv

Source-linked AI summary

Debugging the Debuggers: Failure-Anchored Structured Recovery for Software Engineering Agents

Chenyu Zhao, Shenglin Zhang, Yihang Lin, Wenwei Gu, Zhimin Chen, Yongqian Sun, Dan Pei, Chetan Bansal, Saravan Rajmohan, Minghua Ma

arXiv:2605.08717v2cs.SEcs.AI

TL;DR

Post-failure recovery for software engineering agents lacks a systematic way to turn failed-run evidence into grounded guidance. PROBE structures telemetry into diagnoses and bounded recovery guidance, and across three settings exceeds the strongest non-PROBE baseline by 43.58 percentage points in Top-1 diagnosis accuracy and 12.45 percentage points in recovery rate on 257 initially unresolved cases.

  • Problem

    Existing recovery workflows lack a systematic way to preserve fine-grained failed-run evidence and turn it into grounded guidance for a subsequent attempt.

    Source: §1 · p. 1

  • Method

    PROBE organizes failed-run telemetry into structured evidence, diagnosis, and bounded recovery guidance through a Telemetry Layer, a Diagnosis Layer, and a Guidance Gate.

    Source: §1 · p. 2

  • Results

    43.58 percentage points in Top-1 diagnosis accuracy and 12.45 percentage points in recovery rate over the strongest non-PROBE baseline across three settings on 257 initially unresolved cases.

    Source: §1 · p. 2, §7 · p. 10

  • Takeaways & Limitations

    Accurate diagnosis alone does not ensure executable recovery; guidance must also be grounded and bounded.

    Source: §4.3.2 · p. 8, §7 · p. 10

    The one-repair-attempt protocol measures single-step recoverability, leaving performance across iterative recovery loops uncharacterized.

    Source: §6 · p. 10

Abstract

from arXiv · show

Software engineering agents are increasingly deployed in evaluable engineering environments, yet post-failure recovery remains costly, manual, and ad hoc. Existing systems expose traces or generate follow-up feedback, but they do not convert heterogeneous runtime evidence into grounded, bounded recovery guidance for a subsequent attempt. We present PROBE, a failure-anchored framework for structured recovery in software engineering agents. PROBE organizes failed-run telemetry into structured evidence, structured diagnosis, and bounded recovery guidance through a Telemetry Layer, a Diagnosis Layer, and a Guidance Gate. The Telemetry Layer preserves fine-grained runtime signals, the Diagnosis Layer fuses cross-signal evidence into grounded diagnoses, and the Guidance Gate produces diagnosis-derived guidance only when it is evidence-grounded, actionable, and within the scope of agent-side behavior. We evaluate PROBE across three settings: repository-level software repair, enterprise workflow recovery, and AIOps service mitigation. On 257 initially unresolved cases, PROBE achieves 65.37% Top-1 diagnosis accuracy and a 21.79% recovery rate, outperforming the strongest non-PROBE baseline by 43.58 and 12.45 percentage points. The results reveal a diagnosis-recovery gap: accurate diagnosis is necessary but insufficient unless translated into bounded guidance that a subsequent attempt can execute and verify. Beyond controlled evaluation, a Microsoft IcM prototype shows that PROBE can attach as a non-intrusive side channel to existing service-diagnosis workflows without changing the agent policy, toolset, or execution budget. These results suggest that telemetry-grounded, failure-anchored recovery can improve post-failure recoverability under realistic engineering constraints.

1 Introduction

PROBE addresses ad hoc post-failure recovery by structuring failed-run telemetry into evidence, diagnosis, and bounded guidance, with a gate that permits guidance only when it is grounded, actionable, and in scope.

1 Introduction

  • Existing recovery is fragmented: naive retries lack targeted feedback, benchmarks discard fine-grained signals, and telemetry, diagnosis, and intervention remain disconnected.Failures often reflect recurring behavioral patterns that targeted guidance can address.Source: §1 · pp. 1–2
  • Recovery must preserve provenance-rich signals, integrate heterogeneous metrics, logs, traces, and context into traceable diagnoses, and turn diagnoses into grounded, verifiable, in-scope actions.Guidance specifies a target, operation, verification signal, and boundary condition.Source: §1 · p. 2
  • PROBE organizes failed-run telemetry into typed signals, structured evidence, anchor-first diagnoses, and bounded guidance filtered for telemetry support, actionability, and agent-side scope.Its Telemetry Layer, Diagnosis Layer, and Guidance Gate implement this progression.Source: §1 · p. 2
  • On 257 initially unresolved cases, PROBE achieves 65.37% Top-1 diagnosis accuracy and a 21.79% recovery rate, exceeding the strongest baseline by 43.58 and 12.45 percentage points.Evaluation spans SWE-bench [18], EnterpriseOps-Gym [23], and AIOpsLab [9].Source: §1 · p. 2
  • The portable Python package adds side-channel recovery without changing agent policy, toolset, executor, or evaluator; a Microsoft IcM prototype also avoids changing the execution budget.The framework is evaluated across repository repair, enterprise workflow automation, and AIOps service mitigation.Source: §1 · p. 2

2 Background and Motivation

The section argues that failed engineering runs are process-level events that final verdicts alone cannot explain; effective recovery therefore depends on structured, heterogeneous runtime evidence and bounded guidance.

2.1 Agent Stacks and Execution Layers

  • Across repair, workflow, and AIOps settings, tool-using agents expose recurring execution boundaries, motivating PROBE’s framework-agnostic shared telemetry schema [34] [36] [39] [47].The same schema supports SWE-agent, EnterpriseOps-Gym, and AIOpsLab implementations while keeping the recovery pipeline unchanged.Source: §2.1 · pp. 2–3

2.3.1 Finding 1. Initially unresolved runs are dominated by process-level failures.

  • 172 of 257 initially unresolved runs (66.93%) fall into insufficient validation, tool/subprocess failure handling, or state/workflow error, indicating process-level failures dominate.Failures often depend on how agents interpret tool feedback, verify intermediate results, react to runtime conditions, and decide whether to stop or submit.Source: Table 1 · p. 3, §2.3.1 · pp. 3–4

2.2 Related Work

  • Prior work separately offers trace inspection [12], failure localization [3], trace-based replay [20], or retry feedback [32] [22], rather than telemetry-anchored diagnosis and bounded guidance.The missing process treats failed-run telemetry as its anchor and admits guidance only when it is grounded, actionable, and in scope.Source: §2.2 · p. 3

2.3 Motivation

  • The motivation analysis covers 257 first-attempt reports—102 SWE-bench [18], 106 EnterpriseOps-Gym [23], and 49 AIOpsLab [9]—classified by two engineers blind to PROBE diagnoses and guidance.Annotators reviewed saved trajectories, runtime records, and outcome evidence, resolving disagreements through discussion and category consolidation.Source: §2.3 · p. 3

2.3.2 Finding 2. Effective recovery requires heterogeneous runtime signals.

  • Final verdicts rarely identify causes: validation needs action, evaluator, and stop evidence; tool failures need call sequences, return codes, and adaptation; environment failures require distinguishing recoverable configuration from infrastructure blockers.PROBE is motivated to preserve fine-grained records, structure heterogeneous evidence, diagnose actionable causes, and pass only bounded, evidence-grounded guidance to the next attempt.Source: §2.3.2 · p. 4

3 PROBE · 3.1 Overview

PROBE transforms failed-run telemetry into bounded recovery guidance through a structured, three-layer pipeline. Its benchmark-agnostic schema supports instrumentation of existing agents without changing their policy, toolset, executor, or evaluator.

3.1 Overview

  • PROBE structures post-failure recovery as a transformation from failed-run telemetry to bounded guidance for a subsequent attempt.Source: §3.1 · p. 4
  • The Telemetry Layer records failed-run telemetry in typed signal families, with each bundle component corresponding to one family.Source: §3.1 · p. 4
  • The Diagnosis Layer localizes and fuses failure-relevant signals into structured evidence before deriving a structured diagnosis.Source: §3.1 · p. 4
  • The Guidance Gate validates and constructs bounded recovery guidance from the diagnosis pipeline.Source: §3.1 · p. 4
  • External evaluator signals are included in the telemetry bundle only when available.Source: §3.1 · p. 4
  • PROBE represents its pipeline as T → E → D → G: telemetry, structured evidence, diagnosis, and bounded recovery guidance.Source: §3.1 · p. 4
  • The benchmark-agnostic schema abstracts model-access, orchestration, and tool-interface events into shared telemetry, enabling instrumentation without changing an agent’s policy, toolset, executor, or evaluator.Source: §3.1 · p. 4

3.2 Telemetry Layer

PROBE preserves runtime evidence as typed, provenance-aware telemetry, then organizes it into signals that expose agent behavior, environment conditions, and task outcomes.

3.2 Telemetry Layer

  • Each run is recorded as spans—the smallest execution units, with event type, timestamp, payload, status, and available tool, model, or error metadata—organized into typed families preserving provenance.The telemetry bundle preserves signal provenance for downstream evidence construction and diagnosis.Source: §3.2.1 · p. 4, §3.2 · p. 4

3.2.1 Telemetry Representation.

  • A trace orders spans over time, while intent and tool–environment state distinguish agent goals from what the environment exposed or changed.When phase labels are unavailable, PROBE infers intent from model messages, tool choices, commands, verifier calls, and submission events.Source: §3.2.1 · p. 4

3.2.2 Signal Families.

  • Metrics track cost and capacity, recovery progress, progress–cost coupling, and behavioral stability to flag low-progress, resource-heavy nonprogress, repetitive, or unstable runs.Progress can be measured from state changes, with intent transitions as a fallback when explicit workflow progress is unavailable.Source: §3.2.2 · p. 4
  • Logs deduplicate, truncate, and canonicalize tool returns, command outputs, parser/runtime failures, and system messages into reusable signatures anchoring repeated errors, invalid arguments, subprocess failures, and environment breakdowns.Source: §3.2.2 · p. 4
  • Temporal and intent signals expose repeated failures, missing progress, unstable transitions, and role shifts; environment signals capture tool and workflow context, while optional outcomes ground telemetry in task objectives.Outcome evidence may include benchmark verdicts, test results, incident-resolution checks, or task-specific scoring scripts, and can reveal mismatches between agent claims and evaluator observations.Source: §3.2.2 · pp. 4–5

3.3 Diagnosis Layer

The Diagnosis Layer turns runtime telemetry into structured evidence and then a run-specific, evidence-grounded diagnosis designed to support bounded recovery guidance.

3.3.1 Failure Localization.

  • Failure localization uses signal-specific detectors to identify relevant execution steps, events, patterns, and outcome conditions.Metrics use MAD-based robust z-scores and empirical-quantile thresholds, with Isolation Forest for aggregate evidence when no metric dominates. Logs and tool-environment state capture execution errors and repeated failures; traces and intent use smoothed n-gram surprise and repeated-action patterns, while available evaluator verdicts and failing checks become typed findings.Source: §3.3.1 · p. 5

3.3.2 Cross-Signal Fusion.

  • Cross-signal fusion aligns findings around shared failure events or outcomes, preserving their anchors, provenance, temporal scope, severity, support, and conflicts.Compatible evidence units are grouped by anchor, time, or semantic content; agreement can strengthen an unresolved-failure anchor, while contradictory evidence is retained as conflict.Source: §3.3.2 · p. 5

3.3.3 Structured Diagnosis.

  • Structured diagnosis starts from a telemetry-supported failure anchor and separates the primary cause from the agent’s behavioral mistake.PROBE’s default schema-constrained LLM protocol can fall back to a deterministic summarizer [35] [42] [46]. Its fixed diagnosis object includes contributing factors, an evidence summary, and confidence; confidence estimates evidential support, not calibrated correctness [43], and the Guidance Gate separately checks grounding, actionability, and intervention scope.Source: §3.3.3 · pp. 5–6

3.4 Guidance Gate

The Guidance Gate validates diagnoses before they shape a subsequent attempt, injecting recovery guidance only when it is evidence-grounded, actionable, and within agent-side intervention scope.

3.4 Guidance Gate

  • The deterministic gate constructs bounded recovery guidance only when a structured diagnosis is evidence-grounded, actionable, and within the agent’s intervention scope.It preserves the distinction between diagnosis and guidance by validating diagnoses before they shape the subsequent attempt.Source: §3.4 · p. 6
  • A diagnosis is injectable only when its primary cause and failure anchor trace to structured evidence; unsupported artifact paths, entities, or locations are excluded.Without both a supported primary cause and failure anchor, the diagnosis-derived guidance component is marked non-injectable.Source: §3.4 · p. 6
  • Operationalizable guidance specifies a target, operation, verification signal, and boundary condition, with interventions limited to agent-side recovery [22] [32].Infrastructure-dominated failures do not receive speculative corrective instructions; feedback is bounded around verification, evidence collection, or avoiding premature completion.Source: §3.4 · p. 6
  • After validation, the formatter uses injectable guidance as the main hint; otherwise, it may offer conservative telemetry-supported checks rather than direct corrective instructions.The guidance record captures injectability, target, operation, verification signal, and boundary condition.Source: §3.4 · p. 6

4 Experiments · 4.1 Experimental Setup · 4.2 RQ1: End-to-End Recovery Effectiveness

Under a fixed one-repair-attempt protocol, PROBE improves diagnosis accuracy and recovery across 257 initially unresolved cases, while the results reveal that accurate diagnosis does not guarantee successful recovery.

4 Experiments

  • The experiments ask whether failed-run telemetry improves recovery, whether PROBE produces actionable bounded guidance, and what telemetry, token, and latency overhead it adds.The three research questions address recovery effectiveness, guidance actionability and gating, and practical overhead.Source: §4 · p. 6

4.1 Experimental Setup

  • The evaluation covers repository repair, enterprise workflow recovery, and AIOps service mitigation using SWE-bench [18], EnterpriseOps-Gym [23], and Microsoft AIOpsLab [9].SWE-bench uses mini-SWE-agent; EnterpriseOps-Gym uses its default task executor; AIOpsLab evaluates autonomous agents in interactive microservice cloud environments.Source: §4.1 · pp. 6–7, §4.1.1 · p. 7

4.2 RQ1: End-to-End Recovery Effectiveness

  • 65.37% Top-1 diagnosis accuracy and 21.79% recovery rate across 257 initially unresolved cases; PROBE exceeds the strongest non-PROBE baselines by 43.58 and 12.45 percentage points, respectively.The stronger diagnosis gain than recovery gain indicates a diagnosis–recovery gap: accurate failure-cause identification is necessary but insufficient under fixed environment and budget constraints.Source: Table 2 · p. 7, §4.2 · p. 7

4.1.1 Benchmarks and Recovery Protocol.

  • Only initially unresolved tasks enter the repair phase, which allows at most one repair attempt while holding the agent, task, environment, evaluator, and budget fixed.This protocol isolates the marginal effect of recovery feedback from repeated retries; the one-repair limit is an evaluation protocol, not a framework limitation.Source: §4.1.1 · p. 7

4.1.2 Baselines.

  • Outcome Only supplies evaluator feedback, AgentOps Summary [12] and LangSmith Summary supply trace summaries plus evaluator feedback, and PROBE Full Pipeline supplies gated hints from telemetry and diagnosis.All conditions use the same subject agent, task, evaluator, execution budget, and repair-attempt interface, differing only in feedback; diagnosis and summary baselines use Claude Sonnet 4.5.Source: §4.1.2 · p. 7, §4.1.1 · p. 7

4.1.3 Evaluation Metrics.

  • The evaluation measures telemetry volume by recorded spans and tool calls, prompt overhead by added tokens, and feedback-generation latency from initial failure through formatted recovery hints.Table 2 defines Rate as the fraction of initially unresolved cases recovered after repair and Top-1 as agreement between diagnosed and independently reviewed failure causes.Source: §4.1.3 · p. 7

4.2 RQ1: End-to-End Recovery Effectiveness

  • PROBE outperforms the best non-PROBE baseline on SWE-bench and EnterpriseOps-Gym; on AIOpsLab, Top-1 accuracy ties at 44.90%, but recovery is 30.61% versus 16.33%.SWE-bench results are 83.33% Top-1 accuracy and 28.43% recovery, with gains of 53.92 and 13.72 percentage points; EnterpriseOps-Gym results are 57.55% and 11.32%, with gains of 53.78 and 5.66 points.Source: §4.2 · p. 7
  • Outcome Only recovers 6 of 257 initially unresolved tasks, while observability-summary baselines recover 24 overall and have Top-1 accuracy of 14.79% and 21.79%.The summaries provide runtime context but often miss signal-specific anchors for recovery-relevant causes; in interactive AIOps, recovery also depends on operationally specific guidance such as checks to rerun and premature actions to avoid.Source: §4.2 · pp. 7–8, Figure 4 · p. 8

4.3 RQ2: Actionable and Gated Recovery Guidance · 4.4 RQ3: Practical Overhead

PROBE turns failed-run evidence into structured diagnoses and bounded recovery guidance, but diagnosis alone does not ensure recovery. Across three settings, its side-channel recovery process adds telemetry, prompt, and feedback-generation overhead.

4.3.2 Category-wise Results.

  • 168 of 257 cases aligned with the independently reviewed failure cause, while 56 were recovered, showing that correct diagnosis is important but insufficient for executable recovery.Table 3 defines Aligned as the fraction matching the reviewed cause and Recovered as the fraction of initially unresolved cases solved on the subsequent attempt.Source: Table 3 · p. 8, §4.3.2 · p. 8

4.3.1 Case Study: From Diagnosis to Bounded Guidance.

  • In the AIOpsLab case, PROBE diagnoses a misconfigured targetPort and premature submission, then guides port correction and connectivity verification before submission.The Guidance Gate applies grounding, actionability, and intervention-scope checks; guidance specifies the target, operation, verification signal, and boundary condition.Source: §4.3.1 · p. 8

4.3.2 Category-wise Results.

  • Insufficient validation and state/workflow error reach 97.59% and 83.33% diagnosis alignment, respectively, while state/workflow error reaches only 14.29% recovery.Explicit anchors such as failing checks, premature submission, missing state transitions, or incorrect entity linkage can make diagnosis easier than repair under the same execution budget.Source: §4.3.2 · p. 8
  • Tool/subprocess failure handling has the lowest recovery rate at 6.38%, while patch/submission workflow recovery is highest; runtime/environment guidance stays within verifiable agent-side actions when possible.When direct repair is unsupported or runtime conditions are outside the agent’s control, guidance can instead prioritize verification, evidence collection, or avoiding premature completion.Source: §4.3.2 · pp. 8–9

4.3 RQ2: Actionable and Gated Recovery Guidance

  • PROBE converts failed-run evidence into structured diagnosis and evidence-grounded recovery feedback, but repair still depends on whether the required recovery path is executable by the agent and environment.RQ2 examines this diagnosis–recovery gap through a representative case study and category-wise alignment and recovery outcomes.Source: §4.3 · p. 8, §4.3.2 · p. 9

4.4 RQ3: Practical Overhead

  • Median failed-run telemetry contains 578 spans and 164 tool calls on SWE-bench, 46 spans and 17 tool calls on EnterpriseOps-Gym, and 135 spans and 64 tool calls on AIOpsLab.The differences reflect task structure: repository repair involves repeated file inspection, editing, and tests, while enterprise and service tasks typically involve shorter environment-interaction sequences.Source: §4.4 · p. 9
  • PROBE adds an average of 1.07K prompt tokens on SWE-bench, 0.45K on EnterpriseOps-Gym, and 0.39K on AIOpsLab, keeping feedback prompts compact.Rich telemetry is retained, while diagnosis, Guidance Gate processing, and hint formatting select recovery-relevant content for the subsequent attempt.Source: §4.4 · p. 9
  • Across 257 cases, median feedback-generation latency is 210.43 seconds, with setting-level medians of 190.45 seconds on SWE-bench, 300.68 seconds on EnterpriseOps-Gym, and 130.50 seconds on AIOpsLab.Latency occurs once between the failed initial attempt and repair attempt, covers report construction through hint formatting, and does not modify the subject agent’s execution budget.Source: §4.4 · p. 9

5 Deployment-Oriented Prototype for IcM Service Diagnosis

IcM diagnosis failures often require manual trace and evidence review; PROBE’s proof of concept tests a side-channel recovery artifact while preserving the existing agent workflow.

5.1 Prototype Scope and Input–Output Contract

  • The prototype observes diagnosis-agent runs and produces recovery artifacts after failure or unresolved outcomes, without replacing triage, mitigation, escalation, or human approval.The agent retains the same model, tools, evaluator, prompt budget, and execution logic.Source: §5.1 · p. 9

5.2 Integration Design

  • Lightweight adapters record boundary events and finalize reports, while fail-open behavior lets the original workflow continue if PROBE is unavailable.The integration also attempts to finalize telemetry after exceptions so failed runs can still produce execution records.Source: §5.2 · p. 9
  • 210.43 seconds is the median feedback-generation latency across 257 failed runs; it is off the critical path for human review and outside the agent execution budget for automated reruns.The latency is treated as feedback-generation time before a subsequent attempt.Source: §5.2 · p. 10

5.3 Lessons Learned

  • Useful recovery requires boundary events—not only final outcome labels—to locate lost progress, and adoption should keep PROBE as a wrapper outside the agent’s trusted execution path.Relevant events include model and tool calls, tool returns, validation attempts, exceptions, and evaluator outcomes.Source: §5.3 · p. 10
  • The same report supports human incident review and iterative-agent retries, with the latter receiving a compact hint block containing verifiable actions and stopping conditions.Human-facing reports identify the failure anchor, primary cause, and supporting evidence.Source: §5.2 · p. 9, §5.3 · p. 10
  • The proof of concept validates integration and artifact flow, not production impact; live deployment should assess engineer inspection effort, guidance acceptance, and robustness under load.These measures evaluate operational usefulness beyond diagnosis accuracy and artifact generation.Source: §5.3 · p. 10

6 Discussion and Limitations

PROBE’s evaluation does not establish its iterative-recovery ceiling or how recovery depends on diagnosis-model capability. Future evaluations will test multiple guided attempts and compare diagnosis models.

Limitations

  • PROBE’s one-repair-attempt protocol measures single-step recoverability, not the practical ceiling of iterative recovery.Multi-round protocols could show whether PROBE provides marginal gains across debugging loops or recovery saturates after the first guided attempt.Source: §6 · p. 10
  • The dependence of recovery on diagnosis-model capability remains unquantified.Same-model and model-swap ablations, including using the task-execution model for diagnosis generation, could separate the structured pipeline’s contribution from the diagnosis model’s.Source: §6 · p. 10

Future evaluations

  • Future evaluations will extend PROBE’s assessment to iterative recovery and diagnosis models with different capabilities.These extensions are intended to better characterize its robustness and generality.Source: §6 · p. 10

7 Conclusion

PROBE turns heterogeneous runtime telemetry into structured diagnoses and bounded recovery guidance through three coupled stages. Across 257 initially unresolved cases, it improves diagnosis and recovery over the strongest non-PROBE baseline, while a Microsoft IcM prototype demonstrates integration without changing agent policy, tools, or execution budget.

7 Conclusion

  • On 257 initially unresolved cases, PROBE achieved 65.37% Top-1 diagnosis accuracy and 21.79% recovery rate, exceeding the strongest non-PROBE baseline by 43.58 and 12.45 percentage points, respectively.The evaluation covered three settings.Source: §7 · p. 10
  • PROBE converts heterogeneous runtime telemetry into structured diagnosis and bounded recovery guidance through a Telemetry Layer, Diagnosis Layer, and Guidance Gate.These are three coupled stages of the framework.Source: §7 · p. 10
  • The results show accurate diagnosis is necessary but insufficient without guidance that is grounded, bounded, and executable.Source: §7 · p. 10
  • A lightweight Microsoft IcM service-diagnosis prototype integrates PROBE without modifying agent policy, toolset, or execution budget.Its design principles include fail-open instrumentation, deterministic guidance gating, and boundary-level adaptation for broader industrial software engineering agent workflows.Source: §7 · p. 10

Related papers

Loading 2605.08717v2…