PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond Fault Localization: A Trajectory-Level Study of LLM Agents for Microservice Root Cause Analysis

Qisheng Lu, Aoyang Fang, Junjielong Xu, Jin'ao Shang, Songhan Zhang, Yifan Yang, Xiaochuan Yan, Pinjia He

arXiv:2608.21310v1cs.SE

TL;DR

Microservice RCA benchmarks largely evaluate final service localization, leaving the evidence and propagation route behind a diagnosis unclear. The paper evaluates observable agent trajectories against manually annotated propagation paths across 3,500 runs, then derives DIAGGUARD from observed failure modes. In an independent setting, DIAGGUARD raises Acc@1 from 43.5% to 52.5%.

  • Problem

    Existing RCA evaluations emphasize endpoint correctness but do not reveal the evidentiary basis or service-level fault-propagation route needed to judge a diagnosis.

  • Method

    The paper normalizes agent trajectories and evaluates them against manually annotated service-level fault-propagation paths, using the resulting failure taxonomy to design DIAGGUARD.

  • Results

    In an independent setting, DIAGGUARD raises Acc@1 from 43.5% to 52.5%.

  • Takeaways & Limitations

    Trajectory-level evaluation exposes limitations hidden by final-answer metrics and provides guidance for improving automated RCA.

  • Takeaways & Limitations

    Residual implementation differences across frameworks and models may confound comparisons, and reported differences are therefore indicative.

Abstract

from arXiv · show

Existing evaluations of automated root cause analysis (RCA) for microservices assess diagnostic performance mainly by endpoint correctness: whether a method localizes the responsible service. This criterion enables comparison but does not reveal the evidentiary basis of a diagnosis or the fault-propagation route connecting the source to observed symptoms, both of which an on-call site reliability engineer needs to judge whether action is warranted. We therefore treat RCA as an observable diagnostic process. Our trajectory-level framework evaluates agent executions against manually curated service-level fault-propagation paths. Applied to a public microservice RCA benchmark, it analyzes 3,500 diagnostic trajectories, characterizing where agents investigate and how they use retrieved telemetry. We find a disconnect between answer correctness and diagnostic quality: an agent may localize the fault source yet fail to reconstruct its propagation. Successful investigations stay on the fault-impact surface, act on retrieved evidence, and broaden their query repertoire as the search deepens. Failures arise when decisive evidence is omitted, retrieved evidence is misinterpreted, or unsupported inference substitutes for missing evidence. We operationalize this taxonomy as DiagGuard, a two-stage defense-in-depth architecture in which grounding surveys available observations before localization and verification audits the diagnosis against them. In an independent setting with a different model, benchmark, and service topology, DiagGuard raises Acc@1 from 43.5% to 52.5%. These results show that trajectory-level evaluation exposes limitations hidden by final-answer metrics and provides actionable guidance for improving automated RCA.

I. INTRODUCTION

Microservice RCA evaluations typically score only the final localized service, leaving the evidence and propagation route needed to assess diagnostic trustworthiness unexamined. This paper introduces trajectory-level evaluation against annotated fault-propagation paths and uses the resulting failure analysis to design and validate DIAGGUARD.

  • Endpoint metrics make RCA comparable but collapse the diagnostic process into a final service label, obscuring evidentiary strength and failed reasoning.
  • Trajectory-level evaluation compares normalized agent messages, telemetry queries, and returned evidence against service-level fault-propagation paths.
  • The study finds that successful runs stay on the fault-impact surface and use retrieved evidence, while failures drift, omit decisive evidence, or rely on unsupported reasoning.
  • 3,500 reasoning trajectories across seven framework–model configurations link diagnostic behavior and failure modes to final RCA outcomes.
  • DIAGGUARD converts mined failure modes into grounding and verification defenses, then validates the intervention on a held-out model, dataset, and topology.

II. RELATED WORK

Prior microservice RCA methods mainly return root-cause rankings or final labels, while process-level diagnosis remains difficult to assess without suitable ground truth. Related work spans statistical, graph-based, causal, learning-based, workflow, agentic, and trajectory-analysis approaches.

  • Traditional RCA families include statistical correlation, graph-based propagation, causal inference, and learning-based methods over multimodal observability data.
  • Workflow-based LLM methods structure telemetry for single-pass inference, but a snapshot cannot represent progressive, hypothesis-driven diagnosis.
  • Agentic RCA methods use multi-turn tool calls, telemetry queries, and hypothesis refinement, including MABC and OPENRCA.
  • Despite these advances, RCA methods are assessed almost exclusively at the outcome level because process-level ground truth is lacking.

D. Agent Trajectory Analysis

The study examines general-purpose investigative and RCA-specific agents alongside trajectory-analysis methods, using a common harness to compare their diagnostic behavior. It includes six frameworks and three backbone models with different tool and reasoning designs.

  • Trajectory inspection supports visualization, debugging, and historical optimization of agent behavior.
  • The study selects six frameworks spanning general-purpose investigative agents and RCA-specific agents.
  • THINKDEPTH.AI and AIQ use ReAct-style investigative loops, while TASKWEAVER uses executable plans and tool-backed actions.
  • CLAUDECODE is a terminal-style coding agent; OPENRCA drives an LLM telemetry loop; MABC uses staged multi-agent analysis and consensus.
  • The study uses qwen3.5-plus-2026-02-15, claude-sonnet-4.6, and doubao-seed-2.0-pro-2026-02-15 as backbone models.

C. Benchmarks

The evaluation uses two public microservice RCA benchmarks with different topologies and fault granularities. RCABench supports the main trajectory study, while AIOps 2025 provides an independent validation setting for DIAGGUARD.

  • The study uses RCABench and AIOps 2025, two public benchmarks with different topologies and fault granularities.
  • RCABench contains hierarchical service-to-code-level labels, and the study evaluates a randomly sampled 500-case subset of its 1,430 cases.
  • AIOps 2025 is an e-commerce microservices benchmark containing 400 incidents over ten core services.
  • The RCABench empirical study compares six frameworks under Qwen and reruns one framework under Sonnet, producing seven framework–model configurations.
  • The validation builds the baseline and DIAGGUARD on THINKDEPTH.AI, then tests them on AIOps 2025 with the held-out Seed 2.0 Pro model.
  • Propagation-path annotation reconstructs a graph from injection metadata, telemetry, and the static service-call graph, while trajectories record thought-action-result triples and a final causal graph.

B. Metrics Definitions

The study evaluates RCA with outcome, cost, propagation-reconstruction, and causal-depth measures. These metrics distinguish root-cause localization from recovery of the service-level fault path.

  • Metric inputs: A trajectory is an ordered sequence of rounds, each containing thought, action, and result.For four frameworks, actions generate SQL over a shared telemetry interface; MABC and OPENRCA retain native data access.
  • Metric inputs: The reasoning budget reports effective rounds, mean input-plus-output tokens, and token cost at the backbone model’s list price.Effective rounds are rounds issuing at least one tool call.
  • Localization: Acc@1 counts a case as correct when the predicted root cause belongs to its ground-truth root-cause set.Each case has a ground-truth set containing one or two services.
  • Localization: pass@k measures the fraction of incidents for which at least one of k runs predicts a root cause in the ground-truth set.This metric is reported for configurations run multiple times, including DiagGuard validation.
  • Propagation reconstruction: Node F1 measures service-set recovery, while Edge F1 measures reconstruction of directed propagation links.Both compare the agent’s predicted graph with the ground-truth service-level fault-propagation graph; metrics are also split by Acc@1 outcome.
  • Case difficulty: Causal-chain depth is the shortest-path length from the root cause to the SLO-alarming service along the ground-truth propagation graph.The study stratifies cases by this depth.

C. Behavior and Intent Analysis

The framework analyzes diagnostic behavior through fault-impact-surface navigation and evidence utilization, then characterizes query intent and evidence-related failure modes. It links each trajectory’s actions and retrieved telemetry to whether the investigation advances toward the root cause.

  • Behavior view: The behavior view tracks whether an investigation advances across the fault-impact surface and uses surfaced evidence.The intent view uses each action’s SQL to expose the diagnostic intent behind that behavior.
  • Behavior view: Each trajectory round receives labels for its primary service, navigation transition, and evidence utilization.These labels provide a compact process-level characterization at round granularity.
  • Behavior view: The primary service is selected from services touched by a round, prioritizing proximity to a root cause, newly visited services, and then repeated concentration.This reduction tracks how far the investigation advanced while preserving new directions and concentrated activity.
  • Behavior view: Navigation transitions classify paired rounds as advancing toward, staying level with, or moving away from a root cause.The labels use the rounds’ on-path status and distance to the root.
  • Behavior view: Evidence utilization checks whether the next round’s action services act on on-path services returned by the preceding round.The comparison uses the next action services and the prior round’s returned services.
  • Intent view: Each SQL action receives one intent label and a multivalued set of labels across six deterministic tag axes.A fixed LLM classifier labels 92,501 SQL actions using the SQL operation and returned evidence rather than the agent’s narrative.
  • Intent view: Intent distributions summarize frequencies, while intent n-grams reveal recurring motifs and repeated-intent loops.The n-grams preserve contiguous subsequences of diagnostic intents.
  • Failure analysis: Failed trajectories are evaluated against expected fault-specific evidence to distinguish omitted, misread, and untethered reasoning.A correct diagnosis must recover sufficient evidence for at least one anomalous path from the observed symptom to a root cause.

E. Annotation Reliability

The study combines process annotations with outcome and cost analyses to compare RCA agents, revealing that deeper causal chains and incomplete propagation reconstruction remain challenging despite correct localization.

  • Manual annotation reliability was high for propagation graphs, intent labels, and failure-mode codes, with 93%, 99%, and 93.5% agreement respectively.Intent-label agreement also reached Cohen’s κ = 0.83.
  • Adaptive agents achieved higher localization accuracy than constrained designs, with THINKDEPTH.AI, AIQ, and CLAUDECODE scoring 79.2%, 77.6%, and 79.6%.TASKWEAVER scored 65.2%, OPENRCA 46.6%, and MABC 42.6%.
  • 90.0% Acc@1 for Sonnet THINKDEPTH.AI exceeded Qwen by 10.8 percentage points, while same-model framework differences spanned only 2.0 percentage points.The model swap therefore produced a larger accuracy difference than the top-band framework comparison.
  • Accuracy declines as causal-chain depth increases, although the stronger backbone model largely avoids this decline on the matched THINKDEPTH.AI comparison.For example, Qwen CLAUDECODE falls from 89.5% at depth 2 to 57.1% at depth 5, while Sonnet remains at 85.7–96.8%.
  • Propagation-surface reconstruction is harder than final localization: directed-edge F1 remains below node F1 and never exceeds 0.67, even for correct diagnoses.Incorrect runs show larger relative losses for edge reconstruction than for node identification.

B. RQ2: Agent Behavior

Trajectory analysis shows that successful RCA stays on the fault-impact surface and uses retrieved evidence, while model differences emerge through evidence grounding and diagnostic intent breadth.

  • Surface Navigation and Evidence Grounding: Correct runs re-probe on-path services, whereas incorrect runs more often drift off the fault-impact surface through stray, wandering, or retracing transitions.
  • Intent Repertoire and Fault-Type Sensitivity: Sonnet pivots mid-trajectory toward resource- and topology-oriented intents, while Qwen persists in shallow trace- and log-oriented loops.
  • Intent Repertoire and Fault-Type Sensitivity: Resource probing appears in 82.8% of Sonnet trajectories versus 16.6% for Qwen, aligning with Sonnet’s higher accuracy on RESOURCE and CODE (JVM) faults.RESOURCE accuracy is 94% versus 75%, and CODE (JVM) accuracy is 93% versus 66%.
  • Intent Repertoire and Fault-Type Sensitivity: Qwen’s narrow repertoire produces recurring shallow loops, including sustained trace-following at 0.96 per trajectory versus at most 0.12 for Sonnet.

C. RQ3: Failure Taxonomy

Wrong diagnoses fall into three evidence-handling families: decisive evidence is omitted, misread, or overridden by unsupported reasoning. These failures expose why endpoint correctness alone cannot characterize diagnostic quality.

  • General-reasoning failures include anchoring on early signals, fabricating out-of-schema causes, and abandoning failed or empty evidence channels.
  • Qwen fails predominantly by omitting evidence, remaining confined to superficial signals and rarely redirecting exploration.
  • Sonnet fails predominantly by misreading evidence, with hallucination and difficulty resolving contradictory signals despite broader investigative hypotheses.
  • Three failure families distinguish whether decisive evidence is never gathered, gathered but misread, or overridden by ungrounded reasoning.
  • The taxonomy motivates testing whether recurring failure modes can be distilled into defenses that improve localization beyond the characterization setting.

VI. INTERVENTION AND VALIDATION

DIAGGUARD converts the failure taxonomy into a grounding-and-verification architecture and validates it on a held-out model, dataset, and topology. It improves localization substantially, with complementary gains from both defenses.

  • A. DIAGGUARD Design: A held-out TiKV disk-I/O incident shows the baseline anchoring on chronic Ad-service noise while DIAGGUARD descends to the true storage-node root cause.
  • A. DIAGGUARD Design: DIAGGUARD wraps a reasoning core in two answer-agnostic defenses that address omitted, misread, and overridden evidence.
  • B. Validation: DIAGGUARD is validated on a held-out setting across localization effectiveness and component contribution.
  • A. DIAGGUARD Design: The Grounder confronts the agent with existing telemetry, while the Verifier performs a diagnostic-discipline self-audit and soundness check before commitment.
  • B. Validation: Acc@1 rises from 43.5% to 52.5% (+9.0), while pass@3 rises from 56.9% to 67.1% (+10.2) and pass@5 from 62.3% to 73.0% (+10.7).
  • B. Validation: The Grounder alone raises Acc@1 by +4.5 and the Verifier alone by +4.1, while the full stack reaches +9.0, indicating complementary contributions.

VII. THREATS TO VALIDITY

The study identifies threats involving external, internal, construct, and conclusion validity. Held-out validation partly mitigates transfer concerns, but characterization differences remain indicative because configurations were run once without significance testing.

  • External validity: Characterization may not transfer beyond RCABench’s TrainTicket topology, fixed sample, chosen agents, or similar deployments.Validation on a held-out backbone model, dataset, and service topology partly mitigates this external-validity concern.
  • Internal validity: Residual implementation differences across frameworks and models may confound comparisons despite varying one factor at a time.
  • Construct validity: Manual annotation, LLM-assisted classification, and human review leave subjectivity and judge bias unresolved despite mitigation procedures.Fixed protocols, double coding, and released artifacts reduce but cannot eliminate these concerns.
  • Conclusion validity: Each characterization configuration ran once without significance testing, making its reported differences indicative.
  • Conclusion: The study’s process-level conclusions are supported by trajectory analysis against manually annotated fault-propagation paths, not final-answer correctness alone.
Loading 2608.21310v1…