From General Agents to RCA Experts: A Self-Evolving Harness for Root Cause Analysis

I. INTRODUCTION

• Root cause analysis localizes faults whose symptoms propagate across microservice dependencies, often surfacing far from their causes.It uses metrics, logs, and traces; production incidents can take on-call engineers tens of minutes to localize, [5].Source: §I · p. 1
• 45% of OpenRCA queries are answered by bare Codex on GPT 5.5, versus 32% for RCA-Agent using the same backbone.The general-agent advantage holds across every tested backbone when measured across all tested benchmarks. Unlike specialized LLM-based RCA agents designed from scratch [13]–, general agents already provide mature tool use, code execution, sandboxing, and long-horizon planning.Source: §I · p. 1
• 51.2% average accuracy for Codex (GPT-5.5) leaves substantial RCA headroom, attributed to missing diagnostic expertise and target-system familiarity rather than deficient general capabilities.The proposed remedy is harness engineering: adapting the external layer that supplies tools, domain knowledge, and control logic –.Source: §I · pp. 1–2
• OpsHarness combines layered operational knowledge and an idea-card tool library with four control-plane workflows: setup, diagnose, evolve, and verify.Presented as the first self-evolving external RCA harness, it reuses general agents’ capabilities and remains compatible with their rapid evolution rather than rebuilding an agent from scratch.Source: §I · p. 2
• Self-evolution distills successful and failed diagnosis trajectories into atomic proposals, promoting only updates that pass dual-gate sandbox verification to resist overfitting.The harness accumulates system-specific best practices, knowledge, and tools from past diagnoses for reuse in later incidents, including initially unfamiliar systems.Source: §I · p. 2
• 59.0% top-1 accuracy, averaged over four backbones, represents a 63.4% relative gain over the bare general agent and 4.02× performance of specialized agents.Evaluation spans OpenRCA, RCAEval [25], and an industrial production deployment. Each component contributes to the gains, with overhead comparable to a bare general-purpose agent.Source: §I · p. 2

II. BACKGROUND AND RELATED WORK

• OpsHarness is presented as the first self-evolving external RCA harness, combining compatibility with modern general agents and continual evolution during deployment.Software-development precedents include Superpowers and OMX, whereas comparable external harnesses are largely missing in the Ops domain.Source: §II · pp. 2–3
• Most prior LLM-based RCA approaches build specialized, system-specific agents, limiting reuse of rapidly improving general agents and learning or generalization to unseen scenarios.These approaches include agents operating over multimodal telemetry, role-specialized coordination under SOPs, and incident assistance using tickets and logs.Source: §II · p. 2
• An external harness wraps a whole general agent with task-specific support, whereas the broader harness supplies tools, context, memory, control loops, and evaluation.This distinction treats the general agent’s existing capabilities as a foundation rather than rebuilding them inside a task-specific agent.Source: §II · pp. 2–3

A. External Harness is Now the Key to LLM-based RCA

• 36.1% average overall top-1 accuracy makes general agents stronger than RCA-Agent (17.9%) and mABC (5.6%) across four backbones and two benchmarks.Matched-backbone comparisons use Codex or Claude Code on OpenRCA and RCAEval [25]. On RCAEval, general agents achieve 38.0% versus RCA-Agent’s 5.1%, exposing the specialized agent’s generalization weakness.Source: Table I · p. 3, A. External Harness is Now the Key to… · p. 3

III. MOTIVATION

• OpsHarness’s motivation combines controlled quantitative experiments with production-system case studies at Company A, a large commercial cloud service provider.The industrial case studies provide evidence from a real production setting.Source: §III · p. 3

A. External Harness is Now the Key to LLM-based RCA

• 51.2% overall top-1 accuracy for the strongest bare general-agent configuration, GPT-5.5, still falls short of production needs, which the authors attribute to missing diagnostic expertise.The authors distinguish this expertise gap from a lack of general reasoning ability.Source: A. External Harness is Now the Key to… · p. 3
• A Telecom failure shows a general agent mistaking a noisy network-error counter for the root cause while overlooking CPU saturation’s near-zero-baseline shift.The agent correctly localized the anomalous window but blamed host os_022 instead of CPU saturation on container docker_003. Missing system-specific noise knowledge made raw magnitude misleading.Source: A. External Harness is Now the Key to… · p. 3
• The proposed priority is an external RCA harness that reuses mature general-agent capabilities while addressing the external supporting layer that bounds diagnostic accuracy.General frameworks already supply robust tool-use loops, long-horizon planning, and context management, [23]; rebuilding specialized agents risks underimplementing these capabilities.Source: A. External Harness is Now the Key to… · p. 3

• 36.2% of incidents with recorded troubleshooting guides occur at least twice, motivating continuous learning rather than treating diagnosis as a one-shot exercise.The section identifies continually extracting experience and summarizing best practices as what distinguishes expert SREs from capable newcomers.Source: B. The Ability to Self-Evolve is… · p. 3
• Reusing lessons from an initially misdiagnosed incident helped an SRE locate container CPU saturation and resolve a later incident through restart.The initial investigation mistook falling database sessions for the cause; the actual fault propagated from container CPU saturation through RemoteProcess latency to front-end anomalies. Recording that propagation chain and the unreliability of database session counts as standalone signals guided the later diagnosis.Source: B. The Ability to Self-Evolve is… · p. 4
• About 18 to 13 hours: attaching the right troubleshooting guide reduces mean time to mitigate severity-2 incidents at Microsoft [45].Microsoft links over 4,000 guides to thousands of incidents, but guides are still written and executed manually [45]. Over 90% of 152 high-severity incidents were mitigated using such procedures, while deeper root-cause knowledge remains in postmortems that are not directly reusable [28], [46].Source: B. The Ability to Self-Evolve is… · p. 4
• A self-evolving RCA harness should turn diagnostic experience into reusable expertise, helping agents improve over time and generalize to unfamiliar systems.The harness should automatically mine diagnoses for best practices, knowledge, and tools. Both SREs and agents need this experience extraction and reuse, which remain largely manual today.Source: B. The Ability to Self-Evolve is… · p. 4

A. Overview

• OpsHarness separates its evolving per-system data plane into a layered operational knowledge store and an idea-card tool library.The control plane coordinates workflows over this per-system state.Source: A. Overview · p. 4
• The control plane coordinates setup, diagnosis, evolution, and verification, while an observability subsystem records every run.Setup and diagnosis adapt and apply the harness; evolution and verification form the self-evolution loop that advances harness versions.Source: A. Overview · p. 4

B. Layered Operational Knowledge

• Four operational knowledge tiers distinguish diagnosis-flow experience from operation-specific best practices, with separate disclosure and update policies for each tier.Source: B. Layered Operational Knowledge · p. 4
• K0 defines the general RCA process and expected diagnostic outputs, ships in root documents, and is loaded for every diagnosis.Source: B. Layered Operational Knowledge · p. 4
• K1 automatically captures the target system’s profile after setup, replacing a per-dataset adapter without requiring a hard-coded loader.The profile includes metadata such as schema, on-disk layout, and component inventory, and is read in full at the start of each diagnosis.Source: B. Layered Operational Knowledge · p. 4
• K2 and K3 are mined from diagnosis trajectories during evolution, forming a directed diagnostic knowledge graph for the target system.Nodes represent operations, directed edges identify sensible next operations, and rules annotate diagnostic states. A workflow skeleton is a path through this graph.Source: B. Layered Operational Knowledge · p. 4

END-TO-END USAGE FLOW

• K3 contains reusable diagnostic operations and conditional rules, with positive rules recommending actions and negative rules identifying diagnostic anti-patterns.Rules pair a condition with an action or caveat and a polarity. A K3 operation often maps to an idea card supplying the algorithm that realizes the diagnostic move.Source: END-TO-END USAGE FLOW · p. 5, C. The Idea-Card Tool Library · p. 5
• K2 represents preferred diagnosis workflows as ordered paths through operations, separating coarse workflow structure from K3’s fine-grained operations and rules.A workflow may confirm the front-end KPI window, localize top-down across layers, and then distinguish causes from propagation.Source: END-TO-END USAGE FLOW · p. 5
• Progressive knowledge disclosure loads K0 and K1 routinely, selects full K2 workflows from summaries, and retrieves relevant K3 operations and rules on demand.Knowledge is stored as text under the harness root; loading every tier would crowd the agent’s context and degrade reasoning.Source: END-TO-END USAGE FLOW · p. 5

C. The Idea-Card Tool Library

• Idea cards describe algorithms in natural language, enabling the agent to implement analytical tools against the active schema instead of relying on pre-written scripts.Each card specifies the core idea, light pseudocode, input/output contract, usage conditions, and key parameters. Data layouts vary across systems and windows, motivating code written and iterated against the live context.Source: C. The Idea-Card Tool Library · p. 5
• Evolution grows the initially placeholder-only tool library by distilling code logic recurring across correct diagnosis trajectories into system-specific tool cards.At cold start, the library contains no system-specific tools.Source: C. The Idea-Card Tool Library · p. 5
• The tool library uses staged disclosure: the agent first reads a one-page algorithm menu with usage guidance, then opens only the cards it needs.The agent never loads the whole library.Source: C. The Idea-Card Tool Library · p. 5

D. The Diagnosis Lifecycle

• Users invoke /opsharness:setup during system onboarding or structural changes, supplying only a telemetry location to initiate system profiling.Diagnosis is exposed through /opsharness:diagnose <query> and can be initiated by a user or triggered by an alert.Source: D. The Diagnosis Lifecycle · pp. 5–6
• Diagnosis proceeds top-down, using layered knowledge and cross-modal validation to identify the anomaly window, responsible layer and component, and fault type.The agent loads K0 and K1 and consults K2 and K3 where relevant; metrics confirm, logs explain, and traces show the impact path.Source: D. The Diagnosis Lifecycle · p. 6
• Setup infers telemetry semantics from repeated samples, builds entity inventories and metric categories, and writes K1 plus a profile YAML without per-dataset code.When users supply access paths to existing knowledge or tools, setup incorporates what it finds into K1.Source: D. The Diagnosis Lifecycle · p. 6
• The diagnosing agent implements needed idea cards as context-specific code, executes them, and produces a ranked root-cause report while observability captures the trajectory.Each reported root cause specifies a component, fault type, and onset time.Source: D. The Diagnosis Lifecycle · p. 6

V. SELF-EVOLVING IN OPSHARNESS

• Self-evolution mines labeled trajectories, synthesizes proposals from successful patterns and actionable failure fixes, and applies only proposals passing sandboxed dual-gate verification.The three stages are trajectory mining, evidence mining and proposal synthesis, and staged verification.Source: §V · p. 6

A. Trajectory Mining

• Command-delimited diagnosis blocks become trajectories labeled with feedback-derived accuracy and observed cost; spans outside OpsHarness logic are discarded.After resolution, users provide thumbs-up/down feedback and the actual root cause; accuracy is correct, partial, or wrong, while cost records tokens and time. Correct trajectories form the positive set, and wrong trajectories form the negative set.Source: A. Trajectory Mining · p. 6

B. Evidence Mining and Proposal Synthesis

• Each proposal consists of atomic, typed operators that edit one skeleton path, operation, rule, or tool card to produce a candidate data-plane state.Skeleton paths belong to K2, operations and rules to K3, and tool cards to T.Source: B. Evidence Mining and Proposal… · p. 6
• Successful trajectories yield reusable diagnosis skeletons, operation nodes, tool cards, and positive rules through recurring orders, moves, code, and observations.The evolver approximates recurrence thresholds by reasoning over the labeled batch. Skeletons retain decisive steps while dropping exploratory detours.Source: B. Evidence Mining and Proposal… · p. 6
• Failed trajectories produce corrective edits at the first step inconsistent with the confirmed root cause and negative rules from recurring misleading patterns.For similar incidents with correct and incorrect diagnoses, skipped moves after divergence supply fixes. Failures attributable to outdated evolved knowledge trigger updates or deletions.Source: B. Evidence Mining and Proposal… · p. 6

C. Staged Dual-Gate Verification

• OpsHarness promotes proposals atomically only after both verification gates pass in an isolated sandbox that leaves the live harness unchanged.Failed proposals return for refinement using failing cases as fresh negatives, with retries up to three rounds. Proposals still failing are discarded, and their staging environments are removed.Source: C. Staged Dual-Gate Verification · pp. 6–7
• The inner gate requires nondecreasing source-case accuracy, no material token-cost increase, and strict improvement in at least one objective.The cost tolerance is β = 0.05. Strict improvement excludes no-op changes, while the cost bound rejects accuracy gains bought through runaway exploration.Source: C. Staged Dual-Gate Verification · pp. 6–7
• The outer gate checks non-regression on a held-out testbed resampled before every evolution, stratifying accumulated cases by fault family and occurrence time.The sample remains disjoint from mined cases and defaults to the evolve-window size, avoiding repeated fitting to a static testbed. Task prompts omit privately held labels, and baselines refresh whenever the live harness changes.Source: C. Staged Dual-Gate Verification · p. 7

• The evaluation asks how effective OpsHarness is, especially after self-evolution.Source: §VI · p. 7
• The evaluation asks how much each OpsHarness design contributes.Source: §VI · p. 7
• The evaluation asks about OpsHarness's cost and its performance in an industrial deployment.Source: §VI · p. 7

1) Datasets:

• 335 OpenRCA cases, 270 RCAEval [25] cases, and 88 industrial anomalies support evaluation across enterprise, microservice, and production-change settings.OpenRCA covers three enterprise systems and 28 root-cause categories; RCAEval injects CPU, memory, network, disk, and code faults across three microservice benchmarks. Company A contributes 773,340 telemetry data points, with anomaly labels derived from on-call SRE feedback and raw logs.Source: 1) Datasets · p. 7
• 20% of each dataset’s latest cases form the held-out test set; the earlier 80% supports warm-up without exposing methods to future cases.Warm-up supplies self-evolution for OpsHarness and demonstration retrieval for in-context-learning baselines; reported numbers use held-out tests unless otherwise stated.Source: 1) Datasets · pp. 7–8

B. RQ1: Effectiveness

• 59.0% Final A@1 makes full OpsHarness the strongest framework on every backbone, outperforming all five comparison frameworks.Averaged over four backbones, OpsHarness (no-evolve) reaches 41.4%, ICL 38.4%, Direct 36.1%, RCA-Agent 17.9%, and mABC 5.6% Final A@1.Source: Table II · p. 8, B. RQ1: Effectiveness · pp. 8–9

2) Baselines and Implementation:

• 24 instances combine four backbone models with six frameworks: RCA-Agent, mABC, Direct, ICL, OpsHarness (no-evolve), and full OpsHarness.Backbones are GPT-5.5, Claude Sonnet 4.6, GLM-5.2 [42], and DeepSeek-V4 [43]; Sonnet uses Claude Code, while the others use Codex. All methods receive the same environment and information; Direct has no RCA harness, while ICL retrieves three similar correct rollouts.Source: 2) Baselines and Implementation · p. 8

3) Evaluation Metrics:

• A@1 and A@3 measure exact root-cause-tuple accuracy, Avg Score awards partial credit, and average tokens and time per case measure efficiency.The labeled tuple includes the component and, where applicable, fault type and onset time. Final A@1 averages A@1 across the six public sub-datasets.Source: 3) Evaluation Metrics · p. 8, B. RQ1: Effectiveness · p. 8

B. RQ1: Effectiveness

• 22.9 percentage points separate full OpsHarness from Direct: the Day-0 harness adds 5.3 points, and self-evolution contributes another 17.6 points.The no-evolve variant retains only the Day-0 substrate, isolating cold-start capability from experience-driven improvement.Source: 2) Baselines and Implementation · p. 8, B. RQ1: Effectiveness · p. 9
• 20.3% to 31.6% Final A@1 is the largest cold-start lift, on DeepSeek-V4; GPT-5.5 improves only from 51.2% to 52.8%.The substrate clearly improves Direct on GLM-5.2, DeepSeek-V4, and Claude Sonnet 4.6; on GPT-5.5, the harness’s advantage comes almost entirely from self-evolution.Source: B. RQ1: Effectiveness · p. 9
• 38.4% versus Direct’s 36.1% Final A@1 shows only marginal ICL gains, while Avg Score falls to 67.0% from Direct’s 68.8%.The authors attribute this limitation to retrieving raw examples rather than distilling reusable best practices.Source: B. RQ1: Effectiveness · p. 9

C. RQ2: Contribution of Each Design

• Three variants are evaluated across four backbones on both benchmarks’ training splits using 12 continuous-diagnosis windows.Cases are diagnosed in order, with one evolution triggered per window and mean A@1 reported per window. No-evolve disables self-evolution; no-verify promotes proposals that help their source cases without verification.Source: C. RQ2: Contribution of Each Design · p. 9

2) The Verification Gate Prevents Overfitting:

• 0.33 final-window A@1 for no-verify falls below 0.43 for no-evolve, averaged over four backbones.Accuracy initially rises but then stalls or regresses. Weaker backbones suffer most: DeepSeek-V4 collapses from a peak of 0.4 to 0.0.Source: 2) The Verification Gate Prevents… · p. 9

1) Self-Evolution Improves with Use:

• 0.83 final-window A@1 for full OpsHarness exceeds 0.43 for no-evolve, averaged over four backbones.Full OpsHarness finishes best on every backbone and roughly doubles A@1 across the 12 windows, while no-evolve remains essentially flat.Source: Figure 5 · p. 9, 1) Self-Evolution Improves with Use · p. 9

2) The Verification Gate Prevents Overfitting:

• 37% of evolution proposals are rejected by the verification gate, which promotes changes only if they avoid regression on a disjoint held-out testbed.Accepted proposals require 1.7 attempts on average.Source: 2) The Verification Gate Prevents… · p. 9
• Unverified evolution can overfit source cases: up-weighting network anomalies later mislabels previously correct CPU and memory cases.The proposal helped its source cases but subsequently surfaced network noise as the cause.Source: 2) The Verification Gate Prevents… · p. 9

3) Knowledge and Tool Mechanisms:

• Higher-granularity knowledge is recalled broadly, while fine-grained note-rules and tools are recalled selectively but effectively.Artifact families are characterized by counts, Recall—the fraction of test cases recalling them—and Precision—the fraction of correct cases in which they were recalled.Source: 3) Knowledge and Tool Mechanisms · p. 9

1) Diagnosis Cost:

• 106k tokens and 325s per case, approximately, keep OpsHarness comparable to Direct and ICL across both benchmarks.Direct averages 112k tokens and 317s; ICL averages 106k tokens and 308s. Although OpsHarness loads skills and knowledge into context, it reduces diagnostic trial-and-error.Source: 1) Diagnosis Cost · p. 9, Figure 6 · p. 10
• 1.7×–2.7× the tokens and 2.3×–3.5× the wall-clock time are consumed by specialized agents relative to OpsHarness, mainly through more retries and replanning.The comparison covers the two specialized agents evaluated on the benchmarks.Source: 1) Diagnosis Cost · pp. 9–10

2) Setup, Evolution, and Verification Cost:

• 1.55M tokens and 421s per verification pass make verification the largest non-diagnosis token expense, accounting for 49.8% through parallel held-out diagnoses.Setup runs once per dataset at 0.82M tokens and 412s; evolution, the lightest stage, costs 0.75M tokens and 184s per cycle. Evolution and verification run once per cycle; these infrequent stages execute as parallel sidecars and amortize over later diagnoses.Source: 2) Setup, Evolution, and Verification… · p. 10, Figure 7 · p. 10

3) Harness Footprint:

• 228KB is the approximate warmed-up harness footprint, negligible relative to per-case telemetry.Reusable tools occupy 106KB (46.4%), the skill library 95KB (41.5%), and evolved target-system knowledge 27KB (12.0%).Source: 3) Harness Footprint · p. 10, Figure 7 · p. 10

1) Results on the Industrial Dataset:

• 0.74 A@1 versus Direct’s 0.24 is OpsHarness’s average industrial result across six configurations, improving over Direct in every configuration.The Company A change-anomaly evaluation uses Codex and open-source Open Code with three models; the average improvement is roughly threefold.Source: 1) Results on the Industrial Dataset · p. 10
• 0.73 A@1 with GLM-5.2 and 0.57 with DeepSeek-V4 demonstrate OpsHarness’s effectiveness on Open Code, versus Direct’s respective 0.23 and 0.12.These configurations use fully open-source agent-and-model stacks on the production dataset.Source: 1) Results on the Industrial Dataset · p. 10

2) Case Study:

• Under 2 minutes, OpsHarness diagnosed a recurrent production failure, ranking the database-synchronization fault Top-1 after previously ranking it third.Initially, an SRE rejected the first two candidates and confirmed the third with one line of feedback. On recurrence, OpsHarness also localized the faulty component using automatically mined expertise.Source: Figure 8 · p. 10
• Automatic setup inferred telemetry schemas, while subsequent evolution converted an SRE-confirmed diagnosis into reusable causal rules and K2/K3 knowledge.Setup wrote the system profile and K1; K0, K1, and the profile then guided diagnosis. Within the hour after the alert, evolution distilled the causal chain, passed dual-gate verification, and committed knowledge that an SRE confirmed matched the failure.Source: Figure 8 · p. 10, 2) Case Study · p. 10

VII. DISCUSSION

• 41.4% versus 36.1% Final A@1 favors cold-start OpsHarness over a bare agent, providing a fallback when system changes invalidate learned knowledge.K0 remains valid, and rerunning setup rebuilds K1 without code before experience accumulates again. Evolution can update or delete stale rules when later trajectories contradict them.Source: §VII · p. 10
• Over 75% of sampled tangled-telemetry cases are also undiagnosable by human operators; most OpsHarness errors involve first encounters with unseen fault modes.The authors report that accumulating corresponding best practices progressively mitigates unseen-fault failures.Source: §VII · p. 10

VIII. THREATS TO VALIDITY

• Four backbones, two agent frameworks, and six sub-datasets underpin aggregate accuracy reporting; temporal train–test separation and blocked ground-truth access guard against leakage.Aggregation addresses agent nondeterminism. The same-model bare agent reaches only 36.1% Final A@1, which the authors use to argue that backbone memorization cannot explain the gains.Source: §VIII · pp. 10–11

IX. CONCLUSION

• 63.4% improvement in average accuracy over two bare agents and 4.02× improvement over baseline RCA agents summarize OpsHarness’s overall gains.The conclusion reports evaluation across three benchmarks, four backbones, two agent frameworks, and an industrial deployment.Source: §IX · p. 11
• OpsHarness places specialization in an external harness, pairing layered knowledge and idea-card tools with a control plane that verifies experience-driven updates.Self-evolution mines best practices from both correct and incorrect diagnoses, admitting changes through dual-gate verification.Source: §IX · p. 11