PaperlayerView exact revision on arXiv

Source-linked AI summary

ResonAct: Streaming Metrics for Runtime Diagnosis and Self-Healing in Multi-Agent Systems

Tarun Chintada, Neelamadhav Gantayat, Ishaan Romil, Renuka Sindhgatta, Soujanya Soni, Sameep Mehta

arXiv:2609.34701v1cs.AIcs.LG

TL;DR

Multi-agent systems can fail in diverse ways, while observability often leaves diagnosis and remediation until after execution. ResonAct uses streaming metrics as runtime signals for failure diagnosis and policy-driven intervention; task completion improved across evaluated configurations, with a maximum reported gain of 10.00 percentage points.

  • Problem

    Multi-agent systems experience diverse execution failures, while existing observability commonly diagnoses them after execution rather than intervening during it.

    Source: Introduction · p. 1

  • Method

    ResonAct is an external control plane that streams execution events into operational metrics for runtime failure detection, diagnosis, and policy-driven intervention.

    Source: Introduction · p. 1, The ResonAct Framework · p. 2

  • Results

    Task completion improved across all four benchmark–model configurations, with a +10.00 pp gain on Sales MAS with Qwen3-8B, from 73.67% to 83.67%.

    Source: RQ2: Runtime Remediation and Task… · p. 5

  • Takeaways & Limitations

    ResonAct’s results show improved task completion alongside trade-offs among detection quality, recovery difficulty, and runtime cost across workloads.

    Source: RQ3: Runtime and Resource Overhead · p. 6

    Controlled evaluation of remediation costs, recovery benefits, and user-facing effects with pilot customers remains planned.

    Source: Deployment Plan · p. 6

Abstract

from arXiv · show

Multi-agent systems (MAS) are increasingly used to automate enterprise workflows involving multiple specialized agents, external tools, and long-running task execution. Failures may arise from tool degradation, context propagation errors, coordination breakdowns, or repeated agent interactions that prevent task completion. While existing observability frameworks provide traces and logs, diagnosis and remediation are largely performed after execution completes, limiting opportunities for recovery during runtime. We present ResonAct, a runtime self-healing framework that enables continuous monitoring, diagnosis, and remediation of multi-agent systems through streaming operational metrics. ResonAct ingests execution traces, agent interactions, and tool invocations into a streaming analytics layer that continuously derives task progress, context health, and tool reliability metrics. These metrics serve as runtime control signals for detecting anomalous execution patterns and localizing root causes using a structured failure model. Based on the diagnosed failure, ResonAct dynamically selects remediation policies and performs actions. The framework operates as an external control plane, enabling intervention without modifying application agents or orchestration logic. We evaluate ResonAct across enterprise workflow scenarios and AppWorld benchmarks. The results show that the streaming metric-based analysis identifies execution degradations and localizes faults. Furthermore, policy-driven remediation improves task completion rates by up to 10.00 percentage points, with detection precision ranging from 70.59% to 82.91%, recall from 63.09% to 100%, recovery rates from 10.48% to 46.67%, and runtime overhead ranging from $-0.25%$ to 14.12% across the evaluated configurations.

Introduction

Multi-agent systems face diverse runtime failures, while observability often diagnoses them only after execution. ResonAct uses streaming metrics to detect and diagnose emerging failures and apply policy-driven interventions during execution.

Introduction

  • Failures can stem from tool problems, context propagation errors, coordination breakdowns, and deviations from intended execution, alongside conventional software faults.Reported execution-time behaviors include step repetition, lost conversation history, reasoning-action mismatch, task derailment, and premature termination.Source: Introduction · p. 1
  • Existing traces and logs provide visibility, but diagnosis often follows failure, leaving long-running workflows exposed to unnecessary computational cost and latency.The paper notes that recursive behavior, context growth, and repeated tool interactions can make token consumption highly variable, even for identical inputs.Source: Introduction · p. 1
  • ResonAct continuously turns traces, agent interactions, and tool invocations into health signals for detecting anomalies and localizing potential failure causes.Signals include task progress, context reliability, and tool health.Source: Introduction · pp. 1–2
  • Policy-driven remediation targets diagnosed conditions during execution, while an external control plane separates monitoring and remediation from agent and orchestrator implementations.The evaluation covers enterprise workflows and AppWorld, measuring detection, completion, recovery, and runtime overhead.Source: Introduction · pp. 1–2

Related Work

Prior work covers agent coordination, failure characterization, and observability, but the described approaches leave a need for runtime failure detection and recovery mechanisms tailored to multi-agent systems.

Multi-Agent Systems

  • Multi-agent frameworks support specialization, communication, task decomposition, and orchestration, but runtime reliability is generally not their primary focus.The passage identifies a lack of general mechanisms for continuous failure detection, diagnosis, and automated recovery during execution.Source: Multi-Agent Systems · p. 2

Failure Analysis and Reliability

  • MAS failure research characterizes diverse failures and propagation, but primarily analyzes or injects failures rather than providing automated runtime recovery.MAST draws on 150 traces and was evaluated on 1,600 traces across seven frameworks.Source: Failure Analysis and Reliability · p. 2

Observability and Self-Healing Systems

The ResonAct Framework

ResonAct is an external control plane that streams execution events into runtime metrics, detects and diagnoses failure conditions, and selects corrective guidance for agent invocation boundaries.

The ResonAct Framework

  • The external control plane separates monitoring, failure analysis, and remediation from task-specific agents and orchestrators in a six-stage closed loop.The stages are event capture, metric construction, failure detection, stability filtering, remediation selection, and guidance injection.Source: The ResonAct Framework · p. 2

Failure Taxonomy and Runtime Events Capture

  • Runtime conditions are grouped as common, silent, or critical failures, and execution evidence includes agent interactions, tool invocations, and LLM execution traces.The categories span transient errors and timeouts; context decay and redundant execution; and coordination breakdowns, unbounded loops, and cascading failures.Source: Failure Taxonomy and Runtime Events… · p. 2

Streaming Metric Construction

  • Metrics update incrementally from events and characterize query-, agent-, or tool-level state using count-, embedding-, or LLM-as-judge computation.Metrics describe observed behavior independently of remediation policy, allowing reuse across detectors and policy changes without modifying streaming analytics.Source: Streaming Metric Construction · p. 3

Failure Assessment

  • Failure assessment applies failure-specific detectors to metric snapshots and requires detection across r consecutive cycles before forwarding a stable signal.The persistence parameter r trades early intervention against resistance to transient detections.Source: Failure Assessment · pp. 3–4

Runtime Reflection and Diagnosis

  • An LLM interprets the stable failure signal alongside recent interactions, tool outcomes, execution history, and metric state to produce a failure pattern and supporting evidence.The output is an evidence-based interpretation, not a definitive root-cause determination.Source: Runtime Reflection and Diagnosis · p. 4

Remediation Decision Engine

  • The remediation engine combines the stable signal and diagnostic output into an intervention decision, using either LLM-based or deterministic policies.The decision specifies whether to intervene, the target, strategy, guidance, and retry authorization; deterministic policies provide bounded actions and a fallback when LLM output is unavailable or fails.Source: Remediation Decision Engine · p. 4

Runtime Guidance Injection

  • When intervention is selected, guidance is injected into the applicable agent’s prompt before its next invocation or retry, not into an ongoing generation.Guidance may redirect execution, switch agents or tools, restore missing context, or limit retries.Source: Runtime Guidance Injection · p. 4

Experiments

ResonAct is evaluated for detection, remediation, and efficiency across Sales MAS and AppWorld. It improves task completion across evaluated configurations, while detection and runtime costs vary by workload and model.

Evaluation Metrics

  • Recovery counts a detected failed execution only when it completes successfully after remediation.Detection is evaluated against independently labelled execution outcomes.Source: Evaluation Metrics · p. 4

Experimental Setup

  • Evaluation used Granite-8B and Qwen3-8B, with settings including a ten-agent Sales MAS and an AppWorld MAS extended to an orchestrator and nine application-specific workers.The validation dataset is a 600-query ground-truth benchmark spanning ten domain agents, evenly split across Medium, Difficult, and Complex tiers; approximately 60% were failure-oriented stress scenarios.Source: Experiments · p. 4, Experimental Setup · pp. 4–5

Results and Discussion

  • Failure composition differs by workload: agent degradation and agent loops dominate jointly, while AppWorld has a comparatively larger proportion of agent failures.The reported variation helps explain why one detector configuration does not behave identically across benchmarks.Source: RQ1: Runtime Failure Detection · p. 5, Figure 2 · p. 5
  • 70.59–82.91% precision and 63.09–100% recall: detection varies across workloads, with AppWorld reaching 100% recall for both models but showing elevated false positives.Granite-8B’s AppWorld false-positive rate is 72.97%.Source: RQ1: Runtime Failure Detection · p. 5
  • 4.76–10.00 percentage points: ResonAct improves completion across configurations, with stronger recovery on Sales than AppWorld.Table 3 reports end-to-end detection, task completion, recovery, and runtime performance.Source: RQ2: Runtime Remediation and Task… · p. 5, Table 3 · p. 6
  • −0.25% to 14.12%: runtime overhead ranges from negligible to substantial by configuration, while CPU and memory changes remain comparatively modest.The AppWorld Qwen3-8B run increases execution time by 14.12%; Sales execution time increases by 2.04% and 5.66% for Qwen3-8B and Granite-8B, respectively.Source: RQ3: Runtime and Resource Overhead · pp. 5–6

Deployment Plan

  • The WXO technical preview streams runtime events to compute agent-loop indices and per-tool failure rates; remediation evaluation and operational-cost assessment remain planned for pilot customers.The planned evaluation includes latency, token consumption, cost, recovery, and user assessment of task outcomes.Source: Deployment Plan · p. 6

Conclusion

ResonAct acts as an external control plane, deriving operational metrics from multi-agent execution events to drive failure-specific runtime interventions.

Conclusion

  • ResonAct derives continuous operational metrics from multi-agent execution events and uses them to drive failure-specific runtime interventions.Integration with the agent runtime platform provides a technically grounded path to controlled operational deployment.Source: Conclusion · p. 6
  • Its streaming architecture enables detection and remediation at agent invocation boundaries without embedding control logic in individual agents.Source: Conclusion · p. 6
  • Integration with the agent runtime platform provides a technically grounded path to controlled operational deployment.Source: Conclusion · p. 6

Metric Definitions and Computation

Metrics are computed over recent observation windows; the section describes measures of repeated agent calls and execution steps, including their use and interpretation limits.

Metric Definitions and Computation

  • Default metric windows span 20 queries, 50 calls per tool, and 50 calls per worker agent unless otherwise specified.Source: Metric Definitions and Computation · p. 7
  • The agent loop index uses worker-agent calls and calls after an agent’s first occurrence in a context; a high value signals repeated activation but does not establish error.Source: Metric Definitions and Computation · p. 7
  • Average steps counts worker-agent calls and tool invocations, and ResonAct uses the metric to identify executions exceeding the expected workload distribution.The current implementation uses the empirical P95 as the upper reference for unbounded execution.Source: Metric Definitions and Computation · p. 8

Execution and Path Efficiency

Execution and path-efficiency metrics summarize wasted activity, successful delegation, execution length, and recent tool-call volume. Their interpretations are bounded: some are heuristic, and tool-call volume is a drift signal rather than a measure of success or optimality.

Execution and Path Efficiency

  • The unnecessary path ratio proxies wasted execution by combining repeated worker-agent calls with failed tool invocations.It is heuristic: legitimate repeated agent calls may count as unnecessary, while repeated successful tool calls do not contribute to the numerator.Source: Execution and Path Efficiency · p. 8
  • Delegation efficiency measures successful worker completion relative to delegation activity, but does not match individual assignments to completions or verify answer quality.Successful worker completions and worker-agent delegation events are the quantities it relates.Source: Execution and Path Efficiency · p. 8
  • Compactness compares a completed context's execution length with the shortest positive-step completed context in the current window.It is zero for incomplete or zero-step contexts, or when no positive-step completed baseline exists, and measures relative efficiency rather than optimality.Source: Execution and Path Efficiency · p. 8
  • API call efficiency compares recent tool-call volume with a longer moving baseline across short and long context windows.The default windows contain up to 5 and 20 contexts, respectively; available contexts are used as history accumulates.Source: Execution and Path Efficiency · p. 8
  • Values above 1 indicate increased recent tool-call volume relative to the longer baseline, but the ratio does not independently establish success, cost, or optimality.Task difficulty can also change the ratio.Source: Execution and Path Efficiency · p. 8

Agent and Tool Health

ResonAct uses recent outcomes and latency to assess worker-agent and tool health. Tool health and latency are considered jointly because quick failure can also produce low latency.

Agent and Tool Health

  • Tool health includes a failure rate over an observation window, with success rate defined as 1 − Ft and latest status recorded as up after success or down after failure.The supplied passage introduces the failure-rate measure without specifying its formula.Source: Agent and Tool Health · p. 8
  • Tool latency runs from invocation to terminal completion or failure, so low latency can reflect rapid failure rather than healthy operation.ResonAct considers tool health and latency jointly.Source: Agent and Tool Health · p. 8
  • Worker-agent health is based on recent completion outcomes, while agent latency uses recorded durations of completed executions; the host/orchestrator is excluded.Health and latency provide complementary signals for detecting sustained agent degradation.Source: Agent and Tool Health · p. 8

Semantic and Quality Signals

ResonAct supplements operational metrics with semantic tool-routing confidence, model-based quality scores, and agreement with learned execution paths. These signals are computed continuously and aggregated at context, agent, and system levels.

Semantic and Quality Signals

  • Tool-routing confidence uses semantic similarity between a query and tool schemas, then a temperature-scaled softmax to score the selected tool relative to alternatives.The score is not a calibrated probability of correctness; the implementation uses all-MiniLM-L6-v2 with default temperature T = 0.1.Source: Semantic and Quality Signals · pp. 8–9
  • For completed queries, an optional LLM judge scores communication, planning, coordination, and final-answer quality, but these are model-based assessments rather than independently verified correctness labels.Communication, planning, and answer quality are independently scored on a 1–5 scale.Source: Semantic and Quality Signals · p. 9
  • Agent activation accuracy tests whether an observed ordered agent/tool path belongs to accepted paths associated with query intent, measuring agreement with a learned reference rather than independent ground truth.The accepted-path set can incorporate frequent observed paths after bootstrapping.Source: Semantic and Quality Signals · p. 9
  • The metrics are computed continuously and aggregated at the context, agent, and system levels.Source: Semantic and Quality Signals · p. 9
Loading 2609.34701v1…