PaperlayerView exact revision on arXiv

Source-linked AI summary

From General Agents to RCA Experts: A Self-Evolving Harness for Root Cause Analysis

Haiyu Huang, Jiewei Lyu, Zhihan Jiang, Jinyang Liu, Xiao He, Tieying Zhang, Wu Xiang, Michael R. Lyu

arXiv:2608.25661v1cs.SEcs.AI

TL;DR

LLM-based RCA still falls short of production needs, with the main gap attributed to the external harness rather than the general agent’s capabilities. OpsHarness is a self-evolving RCA harness that improves diagnosis by reusing operational knowledge and experience, achieving 59.0% top-1 accuracy, a 63.4% relative gain over a bare general agent and 4.02× over specialized RCA agents.

  • Problem

    LLM-based RCA lacks production-level accuracy, motivating external harnesses that adapt general agents to diagnosis-specific infrastructure and accumulate system-specific experience.

  • Method

    OpsHarness combines layered operational knowledge and idea-card tools with control workflows that mine correct and incorrect trajectories and verify proposed updates.

  • Results

    59.0% top-1 accuracy was achieved across two public benchmarks and an industrial deployment, a 63.4% relative gain over a bare general agent and 4.02× over specialized RCA agents.

  • Takeaways & Limitations

    The results support shifting LLM-based RCA effort from rebuilding agents toward self-evolving harnesses around capable general-purpose agents.

  • Takeaways & Limitations

    Because LLM agents are non-deterministic, the evaluation reports aggregate accuracy across multiple backbones, agent frameworks, and sub-datasets rather than single runs.

Abstract

from arXiv · show

Automated root cause analysis (RCA) with large language models (LLMs) has drawn growing attention. Today, SREs typically automate RCA with LLMs in one of two ways: directly using a general-purpose agent (e.g., Codex or Claude Code) for diagnosis, or building a specialized RCA agent from scratch. As mainstream general agents grow more capable and iterate quickly, our quantitative study finds that the former now often surpasses the latter. Its accuracy, however, still falls short of production needs, and this gap stems mainly from the external adaptation layer outside the agent's general capabilities, namely the harness. We therefore argue that LLM-based RCA should focus on this external harness, reusing the strong general capabilities of a modern agent rather than rebuilding an agent from scratch. A key capability of such a harness is to self-evolve, accumulating system-specific experience from past diagnoses so that it gets better the more it is used. We introduce OpsHarness, a self-evolving RCA harness that turns diagnosis experience into reusable expertise. Its data plane combines layered operational knowledge with an idea-card tool library, while its control plane coordinates setup, diagnosis, evolution, and verification. During evolution, OpsHarness contrasts successful and failed trajectories, converts their evidence into atomic proposals, and admits updates only through a dual-gate verification process designed to prevent overfitting and regression. Across two public benchmarks and an industrial deployment, OpsHarness achieves 59.0\% top-1 accuracy, improving over a bare general agent by 63.4\% and over baseline RCA agents by 4.02$\times$.

I. INTRODUCTION

Modern microservice complexity makes RCA difficult, while mature general-purpose agents increasingly provide strong reusable capabilities. The paper argues that task-specific external harnesses—not rebuilding agents—are key, and introduces OpsHarness, a self-evolving harness that achieves 59.0% top-1 accuracy across benchmarks and industrial deployment.

  • Background: RCA concerns locating causes from metrics, logs, and traces when microservice symptoms surface far from their causes and propagate across dependencies.The introduction frames RCA as a time-pressured site reliability engineering task.
  • Motivation: General-purpose agents now offer mature tool use, code execution, sandboxing, and long-horizon planning, making agent reconstruction from scratch difficult to improve upon.The introduction identifies Claude Code, OpenAI Codex, and OpenCode as examples.
  • Motivation: Codex (GPT-5.5) reaches only 51.2% average accuracy on the test datasets, leaving substantial room for RCA improvement in the external harness.The paper attributes the remaining gap primarily to the harness: the external software layer supplying task-specific tools, domain knowledge, and control logic –.
  • Approach: OpsHarness is a self-evolving external RCA harness that adds diagnosis-specific skills, tools, and control loops while reusing a general-purpose agent’s capabilities.Its data plane contains a layered operational knowledge store and idea-card tool library, while its control plane manages the harness lifecycle.
  • Results: 59.0% top-1 accuracy yields a 63.4% relative gain over a bare general agent and a 4.02× improvement over specialized RCA agents across two benchmarks and an industrial deployment.The result is averaged over four backbones, and each design component contributes to the overall gains.
  • Approach: Its self-evolution process learns from positive and negative diagnosis trajectories through dual-gate verification, distilling system best practices while resisting overfitting.The process is intended to improve diagnosis as the harness is used more often.

II. BACKGROUND AND RELATED WORK · III. MOTIVATION · A. External Harness is Now the Key to LLM-based RCA

General-purpose agents increasingly outperform specialized RCA agents because they provide mature planning, tool-use, and context-management capabilities, but their production accuracy remains limited by missing system-specific diagnostic expertise. OpsHarness addresses this gap by adding a self-evolving external harness around general agents, extending harness engineering into the operations domain.

  • II. BACKGROUND AND RELATED WORK: LLM-based RCA research spans classical graph-, causal-, and spectrum-based methods alongside agents that plan, write code, inspect telemetry, and coordinate specialized roles.Examples include MicroRCA, Sage, Eadro, BARO, RCAgent [14], RCA-Agent, mABC, and Flow-of-Action [16].
  • II. BACKGROUND AND RELATED WORK: An agent is increasingly treated as a model plus an external harness supplying tools, context, memory, control, and evaluation.CoALA [31] and recent surveys, [33] frame agent building as externalizing capabilities into memory, skills, and tools.
  • II. BACKGROUND AND RELATED WORK: Software-development harnesses such as Superpowers and OMX provide reusable skills, commands, and workflows, whereas comparable operations-domain harnesses remain largely absent.OpsHarness is presented as the first self-evolving RCA external harness, filling this gap for general-purpose agents.
  • III. MOTIVATION: The motivation is grounded in controlled quantitative experiments and industrial case studies using production-system data from Company A, a large commercial cloud service provider.Together, these sources support the paper’s observations about general-agent capability and the need for external adaptation.
  • A. External Harness is Now the Key to LLM-based RCA: General-purpose agents provide robust tool-use loops, long-horizon planning, and context management out of the box, making RCA agents built from scratch likely to reimplement and underdeliver these capabilities.The comparison uses Codex or Claude Code as general-agent frameworks and evaluates GPT-5.5, Claude Sonnet 4.6, GLM-5.2, and DeepSeek-V4 on OpenRCA and RCAEval [25].
  • A. External Harness is Now the Key to LLM-based RCA: A bare general agent averages 36.1% overall top-1 accuracy, while its strongest configuration, GPT-5.5, reaches 51.2%.The shortfall is attributed to missing diagnostic expertise rather than insufficient general reasoning.
  • A. External Harness is Now the Key to LLM-based RCA: A representative failure shows why external expertise matters: the agent selected a noisy network-error counter instead of the modest CPU-saturation signal that caused the latency spike.The error arose because absolute deviation was misleading relative to each signal’s baseline and chronic noise.
  • A. External Harness is Now the Key to LLM-based RCA: General agents now often outperform previous RCA agents, but their accuracy remains bounded by the external supporting layer.The paper bases this finding on controlled quantitative experiments across four backbone models and two public benchmarks, alongside industrial evidence.

B. The Ability to Self-Evolve is Decisive for RCA

RCA is an ongoing process because incidents recur, making self-evolution essential for mining expert experience and reusing it across diagnoses. Yet operational knowledge and its evolution remain largely manual, motivating harnesses that continuously extract system-specific expertise and improve performance.

  • Why Self-Evolution Matters: 36.2% of incidents with recorded troubleshooting guides recur at least twice, so diagnosis is an ongoing process rather than a one-shot exercise.The passage frames continuous summarization of best practices and experience mining as the distinction between expert SREs and capable newcomers.
  • Current Operational Practice: Over 90% of 152 high-severity incidents are mitigated using troubleshooting procedures, but those procedures are still written and executed manually.At Microsoft, more than 4,000 Troubleshooting Guides are linked to thousands of incidents, and the right guide reduces severity-2 mitigation time from about 18 to 13 hours [45].
  • Harness Implication: An evolving harness should automatically mine each diagnosis for reusable best practices, knowledge, and tools, then apply them to future diagnoses on the same system.This directly addresses the current manual evolution process and supports continuous performance improvement.
  • Why Self-Evolution Matters: Self-evolution lets SREs and agents mine expert experience from diagnostic cases, reuse it in subsequent diagnoses, and improve analysis over time.It enables generalization to unfamiliar systems while progressively extracting system-specific expertise during diagnosis.

IV. OPSHARNESS DESIGN · A. Overview · B. Layered Operational Knowledge

OpsHarness separates system-specific state from lifecycle coordination: its data plane combines layered operational knowledge with an idea-card tool library, while its control plane runs setup, diagnosis, evolution, verification, and observability. Its four-tier knowledge store distinguishes general RCA guidance, system profiles, and trajectory-mined diagnostic knowledge with tier-specific disclosure and update policies.

  • A. Overview: OpsHarness divides its harness into a data plane of adaptable per-system state and a control plane of lifecycle workflows plus observability.The data plane contains a layered knowledge store and idea-card tool library; the control plane records each run.
  • A. Overview: The control plane coordinates setup, diagnose, evolve, and verify, with evolution and verification advancing the harness from h_i to h_i+1.Setup and diagnosis adapt and apply the harness, while evolution and verification form its self-evolution loop.
  • B. Layered Operational Knowledge: OpsHarness organizes operational knowledge into four tiers, each with its own disclosure and update policy, because diagnostic-flow experience differs from concrete step-level best practice.This layered structure reflects that operational knowledge is not flat.
  • B. Layered Operational Knowledge: K0 provides general RCA-process knowledge, defining diagnosis inputs and outputs through root documents loaded on every diagnosis.The knowledge ships with the harness as text such as AGENTS.md or CLAUDE.md.
  • B. Layered Operational Knowledge: K1 automatically profiles each target system’s schema, on-disk layout, and components after setup, then loads fully at diagnosis start to avoid hard-coded dataset adapters.It tells the agent how to read the system.
  • B. Layered Operational Knowledge: K2 and K3 are mined from diagnosis trajectories during evolution and together form a system-specific diagnostic knowledge network analogous to accumulated SRE expertise.This network is organized as a directed graph G = (O, E, R).
  • B. Layered Operational Knowledge: In the diagnostic knowledge graph, operations are nodes, directed edges encode sensible next moves, and rules are typed annotations on diagnostic states.The graph is defined as G = (O, E, R).

END-TO-END USAGE FLOW · C. The Idea-Card Tool Library

OpsHarness organizes reusable operational knowledge into layered workflow and rule tiers, while its idea-card library progressively exposes analytical tools and expands from experience. This design lets agents implement targeted diagnostics without loading all knowledge or tools at once.

  • END-TO-END USAGE FLOW: OpsHarness combines a layered knowledge store K with a progressively disclosed tool library T, as summarized in its overall design overview.Knowledge is stored as text under the harness root rather than loaded wholesale for every diagnosis.
  • END-TO-END USAGE FLOW: The knowledge store separates coarse mined workflows in K2 from fine-grained operations and rules in K3, representing preferred diagnostic paths and reusable diagnostic moves.A workflow skeleton π = (o1, . . . , om) orders operations such as confirming the KPI window, localizing across layers, and separating cause from propagation.
  • END-TO-END USAGE FLOW: K3 rules pair conditions with actions or caveats and polarity, encoding both positive diagnostic hints and negative anti-patterns that should not drive diagnosis alone.Examples include tracing container CPU surges through latency to the front end and avoiding database session counts as a sole basis for diagnosis.
  • C. The Idea-Card Tool Library: Idea cards specify analytical algorithms through their rationale, light pseudocode, input/output contracts, usage conditions, and parameters, which the agent implements as needed during diagnosis.They replace pre-written scripts with natural-language tool specifications that support anomaly detection, feature scoring, and suspect ranking.
  • C. The Idea-Card Tool Library: The library uses progressive disclosure: the agent first reads a one-page algorithm menu, then opens only the cards required for the current diagnosis.This avoids exposing the entire tool library at once.
  • C. The Idea-Card Tool Library: At cold start, the library contains only scaffolding, but evolution distills recurring fixed code logic from correct trajectories into system-tuned diagnostic cards.The tool library therefore grows as diagnoses accumulate.

D. The Diagnosis Lifecycle · V. SELF-EVOLVING IN OPSHARNESS · A. Trajectory Mining

OpsHarness operationalizes diagnosis through setup and diagnose commands, then self-evolves by mining labeled trajectories, synthesizing evolution actions, and applying only proposals that pass staged verification. Trajectory mining isolates diagnosis blocks, incorporates user feedback, and partitions sessions by correctness and diagnostic cost.

  • D. The Diagnosis Lifecycle: Setup profiles telemetry without per-dataset code, detects semantic columns, builds entity and metric inventories, and writes K1 with a profile YAML.It can also fold pre-existing system knowledge and tools into K1 during onboarding or after structural changes.
  • D. The Diagnosis Lifecycle: Diagnose is exposed as a slash command that users or an automated process invoke with a query after setup.Setup runs when a system is first onboarded or when its structure changes.
  • V. SELF-EVOLVING IN OPSHARNESS: The self-evolving loop has three stages: mining trajectories, synthesizing evidence into evolution actions, and applying proposals that pass dual-gate sandbox verification.Evidence mining extracts successful patterns from positive trajectories and actionable fixes from negative trajectories.
  • A. Trajectory Mining: Trajectory mining converts each maximal diagnose-command block into one time-ordered trajectory while discarding spans outside OpsHarness logic as noise.Slash-command markers delimit session activity, and each block closes at its end marker.
  • A. Trajectory Mining: After resolution, lightweight user feedback supplies the actual root cause and labels each trajectory’s accuracy as correct, partial, or wrong.The feedback hook asks for a thumbs-up or thumbs-down, while observability measures token and time cost.
  • A. Trajectory Mining: The labels divide the batch into positive and negative sets, with D+ containing correct trajectories and D− containing wrong trajectories.Trajectory cost is recorded alongside accuracy for each diagnosis.

B. Evidence Mining and Proposal Synthesis · C. Staged Dual-Gate Verification

OpsHarness converts evidence from correct and incorrect diagnoses into atomic proposals that modify reusable data-plane expertise. It admits proposals only through staged inner and outer verification, with refinement or rejection when they fail.

  • B. Evidence Mining and Proposal Synthesis: A proposal is a set of atomic, typed operators, each editing one skeleton path, operation or rule, or tool card in the data-plane state.Applying the proposal produces a candidate harness state h′ = Apply(h_i, P).
  • B. Evidence Mining and Proposal Synthesis: Correct and incorrect diagnosis trajectories are mined into proposals that encode reusable structure and corrections from labeled evidence.Correct runs expose recurring diagnostic order, while incorrect runs reveal divergence points for contrastive correction.
  • B. Evidence Mining and Proposal Synthesis: Recurring whole-diagnosis order is distilled into a pruned K2 skeleton so later diagnoses replay decisive steps instead of rediscovering exploratory detours.The pruning targets common structure shared by correct trajectories and reduces inefficient exploration.
  • C. Staged Dual-Gate Verification: Every proposal is tested in an isolated sandbox, leaving the live harness untouched until verification succeeds.The verifier creates a stage, copies the live harness, applies the proposal there, and evaluates the resulting candidate.
  • C. Staged Dual-Gate Verification: The inner gate requires no accuracy regression, no material cost increase, and strict improvement in at least one objective.Its strict clause excludes no-op changes, while the cost band permits small token increases but rejects runaway exploration.
  • C. Staged Dual-Gate Verification: The outer gate tests non-regression on a diverse held-out testbed maintained across time and fault families, ensuring source-case gains generalize.Passing the source cases alone is insufficient evidence that a proposal improves the harness broadly.
  • C. Staged Dual-Gate Verification: A proposal is promoted atomically only if both gates pass; otherwise it is refined with failing cases for up to three rounds or rejected and discarded.Failed stages are torn down to reclaim space, and the live harness is updated only after dual-gate acceptance.

VI. EVALUATION

The evaluation examines OpsHarness’s effectiveness after self-evolution, the contribution of its design choices, its cost, and its performance in an industrial deployment.

  • The evaluation asks how effective OpsHarness is, particularly after self-evolution.
  • It investigates how much each OpsHarness design contributes.
  • It evaluates the cost of OpsHarness.
  • It examines OpsHarness’s performance on an industrial deployment.

A. Experimental Setup … B. RQ1: Effectiveness

OpsHarness is evaluated across diverse public and industrial RCA data using matched backbone–agent instances and standard accuracy metrics. On public benchmarks, full OpsHarness achieves the strongest effectiveness, reaching 59.0% Final A@1 and substantially outperforming baselines through cold-start knowledge and self-evolution.

  • 1) Datasets:: The evaluation spans two public RCA benchmarks and one industrial dataset covering diverse systems, fault types, and noise profiles.OpenRCA contains 335 real-world cases from telecom, banking, and market systems, with 46 nodes, 68 containers, 176 service meshes, and 28 root-cause categories.
  • 1) Datasets:: A temporal split simulates forward-looking deployment and prevents methods from seeing future cases, with reported results taken from held-out test data.The setup also includes demonstration retrieval for in-context-learning baselines.
  • 2) Baselines and Implementation:: Each method is evaluated as a backbone-model and agent-framework pair across GPT-5.5, Claude Sonnet 4.6, GLM-5.2, and DeepSeek-V4.Claude Sonnet 4.6 uses Claude Code, the other models use Codex, and OpsHarness is integrated into the corresponding general-purpose agent under the same environment and information.
  • 2) Baselines and Implementation:: OpsHarness (no-evolve) measures cold-start ability with self-evolution disabled, whereas full OpsHarness enables self-evolution on the Day-0 substrate.The implementations use vendor-supported general-purpose agents and identical available information.
  • 3) Evaluation Metrics:: RCA effectiveness is measured with A@1, A@3, and Avg Score against labeled root-cause tuples comprising component, fault type, and, where applicable, onset time.A@k requires the complete tuple among the top-k candidates, while Avg Score provides partial credit across root-cause elements.
  • B. RQ1: Effectiveness: 59.0% Final A@1 makes full OpsHarness the strongest framework on every backbone, exceeding no-evolve at 41.4%, ICL at 38.4%, Direct at 36.1%, RCA-Agent at 17.9%, and mABC at 5.6%.Final A@1 averages A@1 across the six sub-datasets; the comparison covers all 24 backbone–framework instances on the public benchmarks.
  • B. RQ1: Effectiveness: Self-evolution adds 17.6 points beyond the cold-start substrate, while the substrate adds 5.3 points over Direct, yielding a total 22.9-point gain over the bare agent.The progression is Direct 36.1% to OpsHarness (no-evolve) 41.4% to full OpsHarness 59.0%.

C. RQ2: Contribution of Each Design … 3) Knowledge and Tool Mechanisms:

Across four backbones, self-evolution steadily improves OpsHarness, while its verification gate prevents overfitting and regression. Evolved artifacts combine broadly recalled high-level knowledge with selectively recalled, highly effective fine-grained rules and tools.

  • C. RQ2: Contribution of Each Design: The ablation compares full OpsHarness with self-evolution disabled and with self-evolution enabled but the verification gate removed, isolating both design contributions.Each variant diagnoses cases in order, and the stream is split into 12 windows for continuous evaluation.
  • 1) Self-Evolution Improves with Use:: Static knowledge cannot learn the target system: no-evolve remains essentially flat, with final-window A@1 around 0.2 to 0.6 across backbones.Full OpsHarness instead climbs steadily across the 12 windows.
  • 1) Self-Evolution Improves with Use:: 0.83 final-window A@1 for full OpsHarness versus 0.43 for no-evolve, with full OpsHarness the most accurate variant in all four panels.Across 12 windows, full OpsHarness roughly doubles A@1 on examples including GPT-5.5 (0.4 to 0.9) and GLM-5.2 (0.5 to 0.9).
  • 2) The Verification Gate Prevents Overfitting:: On DeepSeek-V4, no-verify collapses from a peak A@1 of 0.4 to 0.0 because weaker models can mine spurious patterns from source cases.One evolved rule up-weighted network anomalies, helping source cases but later mislabeling previously correct CPU- and memory-type cases.
  • 3) Knowledge and Tool Mechanisms:: Higher-granularity knowledge is recalled broadly, whereas fine-grained note-rules and tools are recalled selectively but achieve high effectiveness when recalled.Table III reports artifact count, Recall as the fraction of test cases recalling an artifact family, and Precision as the fraction of correct cases in which it was recalled.

D. RQ3: Overhead … 1) Results on the Industrial Dataset:

OpsHarness keeps per-case diagnosis cost near lightweight baselines while requiring substantially more resources from specialized agents. On Company A’s industrial dataset, it improves A@1 over Direct across all six configurations, including open-source stacks.

  • 1) Diagnosis Cost:: 106k tokens and 325s per case keep OpsHarness close to Direct and ICL despite loading skills and knowledge into context.Direct uses 112k tokens and 317s, while ICL uses 106k tokens and 308s, averaged over both benchmarks.
  • 1) Diagnosis Cost:: 1.7×–2.7× more tokens and 2.3×–3.5× more wall-clock make specialized agents substantially costlier than OpsHarness.The higher cost is attributed mainly to more retries and re-planning during diagnosis.
  • 2) Setup, Evolution, and Verification Cost:: 0.82M tokens and 412s for setup, 0.75M and 184s for evolution, and 1.55M and 421s for verification quantify the non-diagnosis overhead per pass.Setup runs once per dataset, while evolution and verification run once per cycle.
  • 2) Setup, Evolution, and Verification Cost:: 49.8% of the non-diagnosis token bill comes from verification, which dominates because it runs held-out diagnoses in parallel.Evolution is the lightest stage; these stages run rarely and in parallel as sidecars, amortizing over many diagnoses.
  • 3) Harness Footprint:: 228KB is the warm-up on-disk footprint, comprising reusable tools, the skill library, and evolved target-system knowledge.The components occupy 106KB, 95KB, and 27KB, respectively; the total is described as negligible relative to per-case telemetry.
  • 1) Results on the Industrial Dataset:: 0.74 A@1 versus 0.24 for Direct: OpsHarness improves over Direct in all six Company A configurations, including open-source agents and models.On Open Code, it reaches 0.73 A@1 with GLM-5.2 and 0.57 with DeepSeek-V4, versus 0.23 and 0.12 for Direct, respectively.

2) Case Study: … IX. CONCLUSION

The case study illustrates OpsHarness’s self-evolution loop from system profiling through diagnosis and feedback-driven evolution. Discussion and validity analyses show graceful degradation under system changes, failure modes concentrated in novel or tangled telemetry, and safeguards against nondeterminism and data leakage, while the conclusion frames harness design as the key leverage point for capable general agents.

  • 2) Case Study:: The production case study traces setup, diagnosis, and evolution across three days, with OpsHarness profiling telemetry, ranking root causes, and learning from SRE feedback.The third-ranked diagnosis correctly identified an upstream database synchronization fault causing repeated empty-record lookups; the SRE confirmed it with one line of feedback.
  • VII. DISCUSSION: OpsHarness degrades gracefully under system changes: even when profile K1 and mined knowledge become invalid, valid K0 supports a cold-start fallback matching or exceeding a bare agent, while setup can rebuild K1 without code.The reported fallback comparison is 41.4% vs. 36.1% Final A@1.
  • VII. DISCUSSION: Most failures are first encounters with unseen fault modes, while a second group involves telemetry too tangled for reusable patterns; later self-evolution progressively mitigates the first group.The passage reports that over 75% of sampled cases in the second group have the stated tangled-telemetry characteristic, but the remainder of that finding is truncated.
  • VIII. THREATS TO VALIDITY: To address internal-validity concerns from nondeterministic LLM agents, the evaluation reports aggregate accuracy across four backbones, two agent frameworks, and six sub-datasets rather than single runs.This aggregation is presented as the paper’s internal-validity safeguard.
  • VIII. THREATS TO VALIDITY: The evaluation limits leakage by using a temporal train-test split and a guard script that blocks ground-truth access during diagnosis and self-evolution.A bare agent on the same model reaches 36.1% Final A@1, so backbone memorization alone cannot explain the gains.
  • IX. CONCLUSION: The conclusion argues that, with capable general agents, RCA progress depends more on the surrounding harness than on building a specialized agent from scratch.OpsHarness is presented as a self-evolving external RCA harness combining layered knowledge and idea-card tools in a data plane with a control plane, while mining practices from correct and incorrect diagnoses and applying dual-gate verification.
Loading 2608.25661v1…