Free to Look, Free to Reason: Simple LLM Harnesses Outdiagnose Hand-Crafted Root Cause Analysis Agents

1 Introduction

• Published LLM RCA agents prescribe investigations through SOPs, fixed stages, role sub-agents, or votes, and are often evaluated on only one to three systems against few baselines.Whether handcrafting still pays off as models and harnesses improve is unknown.Source: §1 · p. 1
• The comparison uses three simple harnesses, five published agents, and statistical and causal methods on 90 validated cases from eight public datasets, across six models and three reasoning efforts.The simple harnesses are a one-call evidence package, a SQL agent, and Codex; methods are changed only when they cannot run on the data or have an implementation error.Source: §1 · p. 1
• Simple harnesses reach PR@1 65–77% on three mid-tier models, ahead of published agents (16–53%) and non-LLM methods (6–39%).The simple harnesses leave the model free to choose what to inspect and how to reason.Source: §1 · p. 1
• The study also analyzes why methods succeed or fail, separates model, reasoning-effort, and multi-turn effects, and derives cost- and time-aware recommendations from the Pareto frontier.The analyses include a case study and the share of model calls each method prescribes.Source: §1 · p. 1

2 Related Work

• Classical microservice RCA ranks components using statistical deviation, causal discovery or inference, dependency-graph propagation, or multi-source fusion; LLM agents often prescribe the investigation in the program.Examples of prescribed designs include SOPs, role agents, tree search, and program-side retrieval followed by LLM reranking.Source: §2 · p. 2
• General-purpose harnesses such as Codex let the model act through code without a task-specific procedure; this paper compares Codex, a SQL agent, and a single-call harness with prescribed pipelines and non-LLM methods.The paper takes a complementary approach to OpsHarness by testing harnesses without a self-evolving layer.Source: §2 · p. 2
• RCA datasets typically cover one to three systems and narrow method slices, while some evaluations provide data or a reference agent but not results across methods.RCAtlas pools eight sources, screens cases for validity, and evaluates the method families on the same cases with one scorer.Source: §2 · p. 2
• Evidence from other domains questions elaborate agent pipelines, and this paper tests that question directly on RCA.Prior examples include fixed pipelines competing with software-engineering agents and conversational task splitting reducing reliability.Source: §2 · p. 2

3 Background and Problem

• Microservice cases provide metrics, traces, logs, events, and an inventory of entities and relations across services, pods, containers, nodes, databases, and middleware.Metrics describe entity time series; traces record calls and their timing or status; logs and Kubernetes events supply textual and operational evidence.Source: §3 · p. 2
• For each case, the method returns a ranked list of root-cause entities from the flagged and pre-incident periods, and the first five answers are scored.Labels identify one root-cause entity per case, except that three cases have two; the annotated level may be service, pod, or node.Source: §3 · p. 2
• Answers are mapped upward through inventory parents to the dataset’s annotation level, duplicates are dropped, and the first five distinct entities are matched by name to labels.A finer answer such as a pod can count as its service; multiple pods of that service do not add another match.Source: §3 · pp. 2–3
• PR@1 is the share of cases whose first answer is a root cause, while RR gives credit to the first t_i positions for a case with t_i root causes.A case without an answer scores zero; the paper omits PR@k for larger k because methods return different numbers of answers.Source: §3 · p. 3

4 The RCAtlas Benchmark

• RCAtlas pools eight public RCA datasets into one case format, screens every candidate for validity, and samples survivors across systems and fault types.It comprises ten suites when RCAEval RE1–3 are counted separately.Source: §4 · p. 3
• The converted cases span nine systems and include an entity inventory plus metrics, traces, logs, and Kubernetes events around the fault.Conversion removes traces of fault-injection tools so cases do not reveal their answers.Source: §4 · p. 3
• Of 542 screened cases, 348 pass eight exclusion rules; common exclusions include an unclean pre-fault period and no telemetry evidence of the fault.These two reasons account for 69 and 56 cases, respectively.Source: §4 · p. 3
• RCAtlas-Core samples 90 eligible cases across 23 scenarios and ten fault classes, with root causes at service, pod, and node levels.All experiments use the core set; cases range from 37k to 5.6M telemetry rows.Source: §4 · p. 3
• The authors provide the core-set case list, labels, code, harnesses, and reimplementations, and plan to release converted core data and an extended set of about 500 cases.The case list records source identifiers and windows so cases can be rebuilt from public sources.Source: §4 · p. 3

• Every method receives the same flagged period, candidate inventory, and telemetry, and returns a ranked list scored on its first five entries.Source: §5 · p. 3
• One-shot uses a single evidence-package call, the SQL agent queries raw files with DuckDB, and Codex writes and runs its own shell and Python code.The SQL agent allows at most 100 calls; Codex is constrained to the case folder and a one-hour limit.Source: §5 · p. 3
• The evaluation runs released RCA-Agent and mABC code and reimplements FoA, RCAgent, and DecompRCA, fixing only data-runability barriers or implementation errors.The reimplemented DecompRCA reproduces the paper’s RCAEval RE1 numbers.Source: §5 · pp. 3–4
• Table 1 organizes the RCAtlas core set by source and describes selected cases, telemetry coverage, root-cause levels, median telemetry rows, and median effective candidate counts.Source: Table 1 · p. 4
• RCLAgent-V2, LATS-RCA, EviRCA, MicroRCA-Agent, and supervised methods are excluded for cost, tool, data-adaptation, or training-split constraints.Source: §5 · p. 4
• The non-LLM comparison includes NSigma, BARO, CIRCA, RCD, TORAI, FaultInsight, and SimpleRCA; randomized methods are averaged over three runs.A random ranking provides the floor.Source: §5 · p. 4
• The study varies model coverage and reasoning effort across three settings, repeats selected harness-model combinations three times, and scores unanswered cases zero.All cases have a one-hour limit; reported means use passes, while Figures 1 and 5 and Table 8 use the first pass.Source: §5 · p. 4

6 Experiments

• The evaluation asks five questions about effectiveness, mechanism, model capability and effort, aptitude and stability, and cost-effectiveness.It compares methods on the same cases and examines six models and three reasoning efforts.Source: §6 · p. 4

6.2 Why simple harnesses win: a case study (RQ2)

• In rca100-main-0104, one-shot, SQL, and Codex identify inventory using refusal logs or falling requests, while FoA and mABC stop at cart.The case’s inventory request count collapses, while cart logs refused connections to inventory.Source: §6.2 · p. 5

6.1 Overall effectiveness (RQ1)

• 65–77% PR@1 for simple harnesses, versus 16–53% for published LLM agents and 6–39% for non-LLM methods, averaged over three mid-tier models.Codex is best overall, and one-shot leads the best published agent, FoA, by 12 points.Source: §6.1 · p. 4, Table 3 · p. 6

6.3 How much of the investigation is prescribed (RQ2)

• The three simple harnesses prescribe no model calls; published agents prescribe 0.10–1.00, and call counts alone do not separate the families.mABC’s ratio is a lower bound because its records do not identify which role made each call.Source: §6.3 · pp. 6–7

6.4 Model capability and reasoning effort (RQ3)

• A stronger model reduces calls and gains 14–18 PR@1 points, while more reasoning effort adds only 2–4 points at 1.5–4 times the wall time.In all three contrasts, runs query the root cause in two to five more cases per hundred; the difference is what the model does with that information.Source: §6.4 · p. 7

6.5 Aptitude and stability (RQ4)

• From Luna to Sol, aptitude rises only 2–7 points, but unreliability falls from 32–36 to 7–8; multi-turn agents mainly improve aptitude on Luna.On Luna, SQL and Codex solve 10–11 points more cases at least once than one-shot, with similar unreliability.Source: §6.5 · p. 7

6.6 Cost-effectiveness (RQ5)

• Every cost- or time-Pareto-optimal run above NSigma’s 38.9% PR@1 uses a simple harness; Luna with Codex reaches 68.9% for under one cent per case.Sol with the SQL agent answers in 45 seconds at 85.6%; the most accurate run, Sol at xhigh with Codex, reaches 88.9% at about 26 cents and 277 seconds per case.Source: §6.6 · p. 8

• Only Codex represents general-purpose coding harnesses; other tools, such as Claude Code, may behave differently.Source: §7 · p. 8
• All tested models are from 2026, so the results do not establish whether the ordering holds for weaker models targeted by many prescribed workflows.Source: §7 · p. 8
• Budget constraints excluded current frontier models such as Fable, Opus, and Argon.Source: §7 · p. 8
• Despite careful conversion and reimplementation, the study’s datasets and methods may differ from their originals.Source: §7 · p. 8
• The case study follows one case, and prescribed-call ratios measure fixed investigation rather than showing that it causes lower accuracy.An ablation removing one agent’s prescriptions would test that causal claim directly.Source: §7 · p. 8

• Free-to-investigate models outdiagnose hand-crafted RCA agents, and both mostly outperform statistical and causal methods.Source: §8 · p. 8
• The results do not show that domain knowledge is useless; they show that fixed control flow can hurt when a procedure does not fit the case.In the case study, FoA’s matched procedure never read the logs.Source: §8 · p. 8
• Offering procedures as optional guidance is a natural next step, while practitioners can choose models and harnesses on the cost–accuracy frontier.Source: §8 · p. 8

Published LLM agents

• FoA scores 54.4 / 58.9 / 46.7 PR@1 for Luna / Flash / Haiku, and Luna matches its generic Microservice Issue SOP in 69 of 90 cases.Only four SOPs include a log step, and FoA’s Luna accuracy ranges from 93% on application-code faults to 41% on resource saturation.Source: A Per-method walkthroughs · p. 10
• RCAgent scores 54.4 / 50.0 / 43.3 PR@1 for Luna / Flash / Haiku, while RCA-Agent scores 32.2 / 63.3 / 34.4.RCAgent leaves 7 Luna cases and 19 Flash cases unanswered within 30 steps; RCA-Agent’s executor discards outputs above 16,384 tokens after two tries.Source: A Per-method walkthroughs · p. 10
• DecompRCA scores 24.4 / 32.2 / 28.9 PR@1 for Luna / Flash / Haiku, with its candidate list containing the root cause in only 57.8% of cases.mABC scores 11.1 / – / 20.0 and walks down the call chain, so it never reaches a service receiving no requests.Source: A Per-method walkthroughs · p. 10

Non-LLM methods

• On errors, outages, configuration faults, and network breaks, NSigma and BARO reach 15% and 6%, versus 67% for the SQL agent and Codex on Luna.On resource saturation, NSigma reaches 62%, close to one-shot on Luna at 65%.Source: A Per-method walkthroughs · p. 10
• Short, often constant pre-fault series can misrank normal jumps, deviation scores rank drops last, and RCD misses the root cause in 71 of 89 cases.Source: A Per-method walkthroughs · p. 10

• Figure 7 plots tool actions per case against the first query step naming the root cause, marking first-ranked hits as filled and misses as hollow.Contours show the densest 65% of a run’s hits; cases that never query the root cause are omitted.Source: Figure 7 · p. 10

Significance

• 12 of 13 cross-tier model pairs differ, versus 1 of 9 within-tier pairs (20 of 21 versus 1 of 14 when MiMo is grouped with the lower tier).The SQL agent and Codex also beat the best published agent by 14 to 24 points on every model they ran on.Source: B Additional results and significance · p. 10

Investigation orientation

• From Luna to Sol, the SQL agent uses 25 instead of 30 actions and first queries the root cause at step 14 instead of 17; Codex first reaches it at step 3 instead of 5.More effort moves investigation clouds right without moving them down: Luna’s SQL agent uses 52 actions at max rather than 30 at medium and first reaches the cause at step 22.Source: B Additional results and significance · p. 10

By source

• The simple harnesses lead on seven of eight sources, while FoA leads on RCA-100 (72.2); on OpenRCA 1.0, NSigma reaches 57.1 versus 56.3 for one-shot.AIOps 2022 is the hardest source for every method, with a best result of 52.8.Source: B Additional results and significance · p. 10

Cost

• One-shot sends 235k–293k prompt tokens in one call, while the SQL agent on Luna costs USD 0.011 per case against USD 0.052 for one-shot.Agents’ total token counts range from 90k to over two million, with 76–95% served from the provider’s cache.Source: B Additional results and significance · p. 11

Reasoning effort

• Across effort levels and harnesses, raising effort above medium multiplies wall time by 1.5 to 7 and changes RR by −2 to +7 points; replacing Luna with Sol at medium raises RR by 13 to 15 points.Figure 8 compares RR with wall time per case, excluding network waits.Source: B Additional results and significance · p. 11, Figure 8 · p. 11

C Benchmark and method details

• The benchmark converts cases to one UTC format with entity and relation tables, metrics, traces, logs, and events; of 542 screened cases, 348 pass the exclusion rules.Frequent exclusions are an unclean pre-fault period (69), no evidence (56), incomplete data (38), and a fault late in the window (37).Source: C Benchmark and method details · p. 11

C.1 Case construction

• Case construction checks label consistency, culprit coverage, fault timing, completeness, shortcuts, and observable evidence, with Claude Sonnet 5.5 reviewing most cases and ITBench-SRE using automatic checks only.The sample diversifies scenario, fault class, root-cause level, data kinds, volume, and clue coverage, using fixed tie-breaking and seed.Source: §C.1 · p. 11

C.2 One-shot evidence package

• The one-shot prompt includes selected metrics, traces, logs, and events with an answer template; it caps series and event groups and omits full raw traces, full logs, injection traces, and answer material.The case study contains 136 items (about 75k tokens), and no one-shot prompt was truncated.Source: §C.2 · p. 11

C.3 Method settings and hard groups

• Published-agent runs use shared raw-data tools and method-specific interface fixes, while settings also alter such details as RCAgent’s expert access and mABC’s alert.LLM methods run without generation parameters except effort in Setting 3; runs have a one-hour limit and score the first five answers.Source: §C.3 · p. 11
• On node-level faults, agents mention the true node in 85–90% of runs but mostly blame a pod or service on it.Three AIOps 2022 cases fail because the faulty node has no clear metric, the fault leaves almost no trace, or another node has an unlabelled burst.Source: §C.3 · p. 11
• Table 7 reports source-dataset results as percentages and relative PR@1 to each column’s best; Table 8 reports five cost metrics, including wall time, calls, tokens, cache share, and billed cost.Table 8 uses the first pass of each run, and its wall time excludes network waits; Haiku and DeepSeek were not run in the marked settings.Source: Table 7 · p. 12, Table 8 · p. 12, §C.3 · p. 12