PaperlayerView exact revision on arXiv

Source-linked AI summary

DoVer: Intervention-Driven Auto Debugging for LLM Multi-Agent Systems

Ming Ma, Jue Zhang, Fangkai Yang, Yu Kang, Qingwei Lin, Saravan Rajmohan, Dongmei Zhang

arXiv:2512.06749v3cs.AIcs.SE

TL;DR

Log-only attribution in LLM multi-agent systems leaves failure hypotheses untested, and multiple trials can make a single responsible step ambiguous. DoVer tests hypotheses with targeted edits and replay, recovering failed trials across the reported datasets and frameworks while measuring progress and hypothesis validation.

  • Problem

    Log-based attribution of a failure to one agent or step can be uncertain when a session contains multiple trials with distinct possible errors.

    Source: §1 · p. 1, §3 · p. 4

  • Method

    DoVer segments interaction logs into trials, generates failure hypotheses, applies targeted interventions while preserving prior context, and replays the trajectory to measure success or progress.

    Source: §1 · p. 2, §4.1 · p. 6, §4.2 · p. 7

  • Results

    DoVer recovers 18–28% of failed trials under Magnetic-One on AssistantBench and GAIA, and 49% on GSMPlus with AG2; it also validates or refutes 30–60% of hypotheses.

    Source: §1 · p. 2, §6 · p. 11

  • Takeaways & Limitations

    The reported results support intervention as a practical debugging approach in the evaluated agent frameworks and datasets.

    Source: §6 · p. 11

    The current implementation requires sufficiently rich interaction logs and checkpoint/replay support to reconstruct trials and splice in edited messages.

    Source: §7 · p. 11

Abstract

from arXiv · show

Large language model (LLM)-based multi-agent systems are challenging to debug because failures often arise from long, branching interaction traces. The prevailing practice is to leverage LLMs for log-based failure localization, attributing errors to a specific agent and step. However, this paradigm has two key limitations: (i) log-only debugging lacks validation, producing untested hypotheses, and (ii) single-step or single-agent attribution is often ill-posed, as we find that multiple distinct interventions can independently repair the failed task. To address the first limitation, we introduce DoVer, an intervention-driven debugging framework, which augments hypothesis generation with active verification through targeted interventions (e.g., editing messages, altering plans). For the second limitation, rather than evaluating on attribution accuracy, we focus on measuring whether the system resolves the failure or makes quantifiable progress toward task success, reflecting a more outcome-oriented view of debugging. Within the Magnetic-One agent framework, on the datasets derived from GAIA and AssistantBench, DoVer flips 18-28% of failed trials into successes, achieves up to 16% milestone progress, and validates or refutes 30-60% of failure hypotheses. DoVer also performs effectively on a different dataset (GSMPlus) and agent framework (AG2), where it recovers 49% of failed trials. These results highlight intervention as a practical mechanism for improving reliability in agentic systems and open opportunities for more robust, scalable debugging methods for LLM-based multi-agent systems. Project website and code will be available at https://aka.ms/DoVer.

1 INTRODUCTION

LLM multi-agent systems can fail without crashing, making their errors difficult to diagnose from long interaction logs. DoVer tests failure hypotheses through targeted interventions and evaluates debugging by recovery and progress, rather than attribution alone.

1 INTRODUCTION

  • LLM agent systems increasingly support collaborative tasks, and debugging is needed when they produce incorrect or unsatisfactory results despite continuing to run.The paper includes both benchmark underperformance during development and user-reported dissatisfaction in deployed systems as debugging contexts.Source: §1 · p. 1
  • DoVer tests log-derived failure hypotheses by editing suspected steps, messages, or plans while preserving prior context, then re-executing from the intervention point.A resolved failure supports the hypothesis; a persistent failure refutes it.Source: §1 · p. 2
  • Under Magnetic-One, DoVer recovers 18% of failed AssistantBench trials and 28% of failed GAIA trials; on GSMPlus with AG2, it recovers 49%.It also validates or refutes 30–60% of failure hypotheses, depending on task complexity.Source: §1 · p. 2
  • The paper identifies uncertain ground-truth annotations as a limitation of log-based attribution and frames debugging around resolving failures and validating hypotheses.Its stated contributions include analyzing annotation uncertainty and experimentally demonstrating recovery and hypothesis validation or refutation.Source: §1 · p. 2

2 RELATED WORK

Prior work studies agent failures through taxonomies, trace-level evaluation, and log-based attribution, but attribution remains an untested hypothesis. Debugging work includes manual replay tools and automated repair approaches, while DoVer tests edits through replay and progress measures.

2.1 FAILURE ANALYSIS AND ATTRIBUTION FOR LLM-BASED AGENT SYSTEMS.

  • Failure-analysis work catalogs agent errors across task interpretation, planning, tool interaction, and verification, while trace-level studies find that strong long-context models still struggle to debug traces.Evaluation frameworks also argue that end-to-end pass/fail is too coarse and use requirement graphs or task-utility criteria to reveal where progress stalls.Source: §2.1 · p. 2
  • Log-based failure attribution identifies a decisive step or agent from traces, but remains an untested hypothesis until validated by execution.DoVer tests targeted edits by rerunning the system and assessing success or milestone and utility gains, aligning with trajectory-aware evaluation.Source: §2.1 · p. 2
  • Contemporaneous attribution approaches include reasoning-driven judges, counterfactual scaffolding, causal-inference formulations, spectrum-based methods, hierarchical attribution, and graph-guided tracing.The cited works appeared during the work’s preparation or subsequent peer-review period.Source: §2.1 · pp. 2–3
  • Specialized failure-tracer models trained on curated success and failure trajectories are described as orthogonal to DoVer and potentially usable in its initial attribution stage.The passage presents this integration as a possibility, not an evaluated result.Source: §2.1 · p. 3

2.2 DEBUGGING APPROACHES FOR LLM-BASED AGENT SYSTEMS

  • Manual trace tools support rewind, edit, and re-execution, while AgentDebug uses a similar intervention-driven approach but does not particularly emphasize multi-agent settings.The cited manual tools are described as difficult to scale.Source: §2.2 · p. 3
  • Executable agent-issue benchmarks and industrial program-repair studies motivate debugging that verifies edits in the original run and measures intermediate progress.DoVer chooses a minimal intervention, reruns the trajectory, and scores milestone or utility gains.Source: §2.2 · p. 3

3 FROM LOG-BASED ATTRIBUTION TO INTERVENTION-BASED DEBUGGING

The section argues that single-step log attribution is difficult to evaluate when labels are uncertain and sessions contain multiple trials or inter-agent misalignment. The authors use this analysis to motivate trial-aware interventions and explicit validation of hypothesized errors.

3 FROM LOG-BASED ATTRIBUTION TO INTERVENTION-BASED DEBUGGING

  • The section frames its analysis around the prevailing single-agent, single-step attribution task and sources of uncertainty in its ground-truth annotations.The task defines a decisive error as a step whose incorrect action, if replaced with a correct one, would be followed by trajectory success.Source: §3 · p. 3
  • 6% to 24%: GPT-4o step-attribution accuracy on WW-HC rises with explicit step indices and an annotator-guidance reminder in the All-at-Once setting.The reported setting includes ground-truth answers; the work notes that absolute accuracy remains low for practical deployment.Source: §3 · pp. 3–4
  • 14 of 29 GAIA cases in WW were judged to have uncertain ground-truth annotations, while WW reports 15–30% annotator uncertainty and about 20% average initial disagreement.Attribution accuracy was 24% for GPT-4o and 7% for GPT-5 on uncertain cases, versus 44% and 53% on the remaining certain cases, respectively.Source: §3 · p. 4
  • 9 of 14 uncertain cases involved multiple trials, each of which could contain distinct decisive errors, making one session-wide step attribution ambiguous.Trials are contiguous planning-and-execution spans; the illustrated session has four trials with different strategies and separate possible error points.Source: §3 · p. 4, Figure 1 · p. 4
  • In 5 of 14 uncertain cases, inter-agent misalignment made responsibility ambiguous because both an Orchestrator instruction and a WebSurfer action were faulty.In 7 uncertain WW cases, reviewers found no clear error at the ground-truth-designated step even after adjudication.Source: §3 · p. 5
  • The authors propose testing hypothesized failure steps with interventions and checking whether the trajectory succeeds, reducing dependence on noisy human labels.They recommend trial-aware interventions and distinguish edits to the Orchestrator’s plans or instructions from direct capability improvements for sub-agents.Source: §3 · p. 5

4 METHODOLOGY

DoVer turns failure-attribution hypotheses into controlled edits and replays, then evaluates task success, progress, and whether the hypothesis was validated or refuted. Its pipeline segments long traces into trials so multiple hypotheses can be tested independently.

4 METHODOLOGY

  • DoVer has four stages: trial segmentation, failure-hypothesis generation, intervention generation, and execution with differential evaluation.Evaluation compares the intervention run with the original using task success and a progress score.Source: §4 · p. 5

4.1 DOVER PIPELINE

  • Re-plan steps divide execution logs into trials, shortening the context for each causal chain and enabling independent, parallel interventions.This allows multiple hypotheses to be proposed for a single trace, since a failure may admit more than one viable repair.Source: §4.1 · p. 6
  • For each trial, DoVer generates a suspected failure step and agent with a rationale, then turns the hypothesis into a targeted edit to the failing context.It focuses primarily on orchestrator-level message edits, including revised sub-agent instructions and plan updates; this supports architecture agnosticism but cannot directly enhance sub-agent capabilities.Source: §4.1 · p. 6

4.2 EVALUATION METRICS

  • Trial Success Rate measures task completion after intervention, while Progress Made measures movement toward success even when a trial remains incomplete.When annotated solution steps are available, progress can be measured through milestones extracted from human-annotated traces.Source: §4.2 · p. 7
  • Without human-annotated intermediate steps, the milestone-based progress metric cannot be applied directly; trace comparison with an LLM judge is left for future work.Source: §4.2 · p. 7
  • DoVer classifies intervention outcomes as validated, partially validated, refuted, or inconclusive using trace comparisons and whether the intervention was faithfully executed.The system uses three independent runs per intervention to reduce execution randomness.Source: §4.2 · pp. 6–7

5 EXPERIMENT RESULTS

Experiments evaluate DoVer through intervention outcomes and hypothesis validation across datasets and agent frameworks, with ablations, case studies, and analysis of inconclusive trials. Results vary by setting, and some failures point to sub-agent capability limits that orchestrator-level interventions cannot resolve.

5.1 EXPERIMENT SETUP

  • M1 experiments used checkpointed replay and message edits across WW-derived AssistantBench and GAIA sets; AG2 experiments used 2,400 GSMPlus testmini examples.The WW sets included 28 AssistantBench cases, 29 GAIA cases, and 45 additional GAIA Level-1 validation cases.Source: §5.1 · p. 8

5.2 QUANTITATIVE EVALUATION RESULTS

  • Trial Success Rate was 17.6% in WW and 27.5% in GAIA-Level-1; Progress Made reached 15.7% in GAIA-Level-1, while GSMPlus approached 50% trial success.Progress Made cannot be computed for GSMPlus because the dataset lacks human-annotated solution steps.Source: §5.2 · pp. 8–9, Table 2 · p. 9
  • Validated and Refuted hypotheses were each around 15% in WW-AB and WW-GAIA, where about 60% were Inconclusive; GAIA-Level-1 had about 30% Inconclusive cases.The authors suggest that harder WW cases make reliable execution of intended interventions more difficult.Source: Table 3 · p. 9, §5.2 · p. 9

5.3 ABLATION STUDY

  • Qwen3-8B achieved 11.3% trial success, rising to 14.3% with 3-shot prompting; Qwen3-32B achieved 16.9%, compared with GPT-4o's 17.6%.The model comparison used the WW-GAIA setting.Source: §5.3 · pp. 9–10
  • In WW-GAIA, Self-Refine and CRITIC recovered 0% of failed cases, while DoVer recovered 17.6% of trials.The authors attribute the baselines' shortfall to long, noisy, divergent trajectories that make end-of-trace refinement insufficient.Source: §5.3 · p. 10

5.4 QUALITATIVE CASE STUDIES

  • Case studies distinguish refuted hypotheses after faithfully executed but ineffective edits from inconclusive trials where tooling prevents faithful execution of the intended intervention.Across the cases, outcomes also include validation and partial validation when edits repair trajectories or produce measurable progress despite external frictions.Source: §5.4 · p. 10

5.5 ANALYSIS OF INCONCLUSIVE CASES AND TARGETED TOOL ENHANCEMENT

  • Inconclusive cases comprised 29–67% of trials in Table 3 and exposed sub-agent capability gaps; targeted WebSurfer fixes later solved Cases 20, 21, and 26.The fixes addressed missing scroll-to-bottom functionality and PDF handling.Source: §5.5 · p. 11
  • The authors identify two future directions: automated debugging loops that implement sub-agent fixes and capability-aware interventions when code changes are not allowed.Source: §5.5 · p. 11

6 CONCLUSION

DoVer reframes multi-agent debugging as a do-then-verify process, testing failure hypotheses through targeted interventions rather than relying solely on log attribution. It reports recovery across multiple datasets and frameworks, alongside measurable progress and hypothesis validation.

6 CONCLUSION

  • DoVer recovers 18–28% of previously failed trials on AssistantBench and GAIA under M1, and 49% of failures on GSMPlus under AG2.The paper also reports measurable improvements and verification or falsification of most hypotheses.Source: §6 · p. 11
  • DoVer tests failure-attribution hypotheses by applying targeted interventions and examining whether the system succeeds.This do-then-verify process differs from relying solely on log-based attribution.Source: §6 · p. 11
  • The authors present intervention as a practical and scalable tool for improving agent reliability and reducing dependence on ambiguous human annotations.Source: §6 · p. 11

7 LIMITATIONS AND GENERALIZABILITY

The evaluation covers two specific agent frameworks and task settings, so the reported results are evidence of feasibility rather than universal guarantees. DoVer also requires replay-capable logs and currently limits interventions to orchestrator messages.

7 LIMITATIONS AND GENERALIZABILITY

  • The evaluation covers Magentic-One on GAIA/AssistantBench and AutoGen2-based MathChat on GSMPlus, so results are feasibility evidence, not universal guarantees.The tested tasks are web-based information seeking and math problem solving, with sequential, centrally orchestrated topologies and GPT-4o and Qwen3 models.Source: §7 · p. 11
  • DoVer requires rich interaction logs and checkpoint/replay interfaces to reconstruct planning–execution trials and splice in edited messages.Adding checkpointing required non-trivial engineering effort in M1 and AutoGen2; asynchronous or black-box orchestrators would need additional instrumentation or higher-level abstractions.Source: §7 · p. 11
  • Interventions are restricted to orchestrator text, limiting repairs for failures rooted in missing sub-agent capabilities.The current method neither modifies sub-agent code nor synthesizes new tools.Source: §7 · p. 11
  • Milestone progress and “intervention fulfilled” labels rely partly on LLM-as-a-judge assessments, which may introduce bias.Source: §7 · p. 11

A REVISIT LOG-BASED FAILURE ATTRIBUTION ON THE WHO&WHEN DATASET

The appendix revisits log-based failure attribution on Who&When (WW), reproducing the All-at-Once protocol and examining prompt refinements and uncertainty in ground-truth labels. Attribution improves with the refinements but remains low at the step level, including with GPT-5.

A REVISIT LOG-BASED FAILURE ATTRIBUTION ON THE WHO&WHEN DATASET

  • The appendix focuses on the All-at-Once method because it uses one LLM call and performs comparably to Step-by-Step and Binary Search; it also provides trial-level annotation notes for GAIA cases in WW.Case 25 is excluded because its question is ambiguous; Table 6 records trial observations, potential errors, ambiguous attributions, and API issues.Source: A REVISIT LOG-BASED FAILURE… · pp. 14–15, Table 6 · p. 17
  • Step-level attribution accuracy remains around 20% even with GPT-5, despite gains from explicit step indices and annotator guidance.Table 5 reports the reproduced WW results across scenarios and models, including the prompt refinements.Source: Table 5 · p. 15, A REVISIT LOG-BASED FAILURE… · p. 15
  • 13.8% of Hand Crafted cases and 20.6% of Algorithm Generated cases had hallucinated failure-step indices; explicit indices improved step-level accuracy and eliminated out-of-range indices.Figure 3 marks the explicit step indices in orange.Source: A REVISIT LOG-BASED FAILURE… · p. 15, Figure 3 · p. 16
  • Embedding annotators’ guidance in the prompt further improves attribution performance by aligning the criteria used by models and annotators.The concise reminder is shown in blue in Figure 3.Source: A REVISIT LOG-BASED FAILURE… · p. 15, Figure 3 · p. 16
  • Model-output agreement with ground-truth labels is substantially higher for cases whose ground-truth annotation is uncertain, indicating that log-based attribution can be confounded by label uncertainty.Source: A REVISIT LOG-BASED FAILURE… · p. 15

B PROMPTS USED IN DOVER

DoVer’s prompts divide debugging into trial segmentation, failure analysis, intervention generation, progress measurement, and post-intervention diagnosis. Together, these stages locate candidate failures, propose fixes, measure outcomes, and assess whether an intervention supports the initial hypothesis.

B PROMPTS USED IN DOVER

  • The Trial Segmenter partitions the session log into plan–execution trials and identifies initial and updated planning steps as anchors for later analysis.Source: B PROMPTS USED IN DOVER · p. 19
  • The Failure Proposer summarizes each trial’s plan and execution, then identifies the earliest error, responsible agent, and error reason when a trial fails.Source: B PROMPTS USED IN DOVER · p. 19
  • The Intervention Recommender proposes a minimal executable fix using the task description, ground-truth answer, and localized failure context.Its JSON output specifies an intervention category and replacement text.Source: B PROMPTS USED IN DOVER · p. 19
  • The Milestone Extractor creates up to five tool-agnostic milestones, which the Milestone Evaluator labels achieved, partial, or missed to quantify progress.The evaluator also assesses whether a new path was explored, its feasibility, and its contribution.Source: B PROMPTS USED IN DOVER · p. 19
  • After rerunning with an intervention, the final proposer distinguishes success, failure to execute the instruction, and a mislocalized or insufficient fix, supplying evidence for hypothesis validation.Source: B PROMPTS USED IN DOVER · p. 19

C ENABLING DOVER ON AUTOGEN2

DoVer was integrated with AG2 MathChat by adding checkpoint-and-replay support around its conversation manager, allowing interventions to edit a message and resume execution. The integration reuses DoVer’s pipeline after adapting prompts to MathChat.

Overview of MathChat in AG2

  • AG2 MathChat uses a group-chat manager that selects among specialized workers, rather than Magentic-One’s explicit planner–executor loop.Reasoning attempts and verification cycles emerge through specialist turns; there is no dedicated planner agent.Source: C ENABLING DOVER ON AUTOGEN2 · pp. 19, 21

Enhancing AG2 with Checkpointing and Replay

  • AG2 checkpoints serialize conversation history, agent and LLM configurations; MathChat needs no additional tool-state snapshot because it has no persistent external environment.Checkpoints are keyed by step index and can restore the system at a past turn.Source: C ENABLING DOVER ON AUTOGEN2 · p. 21

Enabling DoVer on MathChat in AG2

  • DoVer restores a checkpoint, edits the target-step message, and resumes the original AG2 run loop without changing core scheduling or tool interfaces.Trial segmentation and intervention generation are reused after adapting prompts to MathChat.Source: C ENABLING DOVER ON AUTOGEN2 · p. 21

Implementation Effort

  • The integration took roughly 1,000 lines of code and a few days for one engineer, with new AG2 applications needing checkpoint capture, replay entry points, and intervention mappings.The work was aided by LLM-based coding assistance.Source: C ENABLING DOVER ON AUTOGEN2 · p. 21

Web-based Intervention Interface for AutoGen2

  • The web interface lets users inspect traces, edit a selected message or plan, resume execution, and review continuation histories; experiment logic does not depend on it.It is intended as a lightweight visualization and human-in-the-loop debugging tool.Source: C ENABLING DOVER ON AUTOGEN2 · p. 21, Figure 4 · p. 22

Guidelines for Enabling DoVer in Other Agent Frameworks

  • The integration preserves existing architecture with checkpoint-aware wrappers, captures relevant system state, and applies interventions directly to structured checkpoint data.The stated design principles include restoring agent configuration, LLM settings, and conversation history.Source: C ENABLING DOVER ON AUTOGEN2 · p. 21

D SAMPLE “VALIDATED” AND “PARTIALLY VALIDATED” CASES

The cases illustrate how targeted intervention can resolve a retrieval-strategy failure, while another intervention makes forward progress but leaves execution blocked by environment and tooling.

Validated case

  • A date-bounded, keyword-based APOD search replaced aimless scrolling, found Holabird as the architect, and flipped the failed trial to success.The case validates both the strategy-error hypothesis and the focused-retrieval remedy.Source: D SAMPLE “VALIDATED” AND “PARTIALLY… · p. 22

Partially validated case

  • Case 56’s Alpha Vantage price investigation stalled before source verification; the intervention redirected the agent to check data availability and access requirements.The case is labeled partially validated.Source: D SAMPLE “VALIDATED” AND “PARTIALLY… · p. 22

Validated case

  • In Case 3, re-planning reached the correct information source, but script errors and API-key friction blocked completion despite validating the re-planning hypothesis.The case is described as clear forward progress without a final result.Source: D SAMPLE “VALIDATED” AND “PARTIALLY… · p. 22
Loading 2512.06749v3…