PaperlayerView exact revision on arXiv

Source-linked AI summary

Does Graph Structure Earn Its Place in Microservice Root-Cause Analysis? A Controlled Study on RCAEval, and What the Benchmark Was Really Measuring

Imad Buljić

arXiv:2609.27069v1cs.SEcs.LG

TL;DR

Pipeline-level comparisons leave unclear whether graph structure helps microservice root-cause analysis or is redundant given the features. This paper controls graph access in matched RCAEval arms and audits the benchmark; it finds no reliable graph-specific effect and identifies benchmark properties that condition its scores.

  • Problem

    Pipeline-level comparisons of graph and flat models cannot isolate whether graph structure contributes or is redundant given the features.

    Source: §1 · p. 2

  • Method

    The study compares matched learned arms differing in graph access and audits RCAEval’s benchmark pipeline.

    Source: §1 · pp. 2–3, §3 · p. 4

  • Results

    0.003 Avg@5 is the graph model’s in-distribution lead over the flat model, with no reliable graph-specific effect (p = 0.844, n = 6).

    Source: §4 · p. 5

  • Takeaways & Limitations

    A telemetry-blind ranker scores Avg@5 0.488 in-distribution, making that prior rather than uniform-random 0.137 the honest benchmark floor.

    Source: §5.1 · p. 7

    The RE1 schema silently zero-fills telemetry for 250 of 375 cases, and PSC-GRCA is not ported to the RE2 loader.

    Source: §3 · p. 4, §6 · p. 11

Abstract

from arXiv · show

Graph neural networks dominate recent work on microservice root-cause analysis, yet recent results question whether the graph contributes. Those results compare whole pipelines, so when a flat model wins one cannot tell whether structure is useless or redundant. We run the comparison they imply on RCAEval: three learned arms with identical features, optimiser, validation split, early-stopping rule and scoring head, in which a single term separates the graph arms. Across two RCAEval benchmarks, two topology sources and four regimes we find no reliable graph-specific effect: in-distribution the graph model leads the conventional flat model by 0.003 Avg@5 (p = 0.844, n = 6 disjoint folds). Auditing the pipeline surfaced two benchmark properties that condition any result on it. RCAEval injects faults into only five services per system while exposing 12 to 70 in telemetry, and the headline metric is Avg@5: a ranker reading no telemetry at all places the true culprit in the top five on 99.7 percent of held-out incidents, scoring Avg@5 0.488. That prior, not the uniform-random 0.137, is the honest in-distribution floor, and it collapses to 0.192 across systems. The second property is a non-uniform column schema that silently zeroes telemetry for most RE1 cases. We reproduce a published baseline, BARO, RCAEval's own reference implementation; on the one system with a clean schema it reaches similar aggregate accuracy to our heuristic, within 0.004, under a different scoring rule. The audit motivated a new model. PSC-GRCA separates a candidate score into a system prior, telemetry evidence and a centred graph residual, and reaches mean Avg@5 0.915 against 0.864 for the flat baseline, while its ablations locate most of the gain in the prior term rather than the graph. We close with a twelve-item checklist for graph-versus-flat ablation studies, distilled from sixty-two recorded defects.

1 Introduction

Graph-based modelling is standard in microservice root-cause analysis, but recent pipeline-level comparisons cannot isolate whether graph structure itself helps. This paper controls the comparison by holding the learned pipeline fixed while varying graph access, then audits RCAEval and presents a checklist and prior-separated model.

1 Introduction

  • Although microservice RCA has converged on graph-based modelling [7], a topology-agnostic MLP matched GNN-based multimodal diagnosis across five datasets, graph-free RCAEval scored 68% Top-1 on 735 failures, and propagation-aware re-evaluation reduced eleven state-of-the-art models to 0.21 average Top@1 [1] [2] [3].Source: §1 · p. 2
  • Recent work questions graph-based modelling, but pipeline-level comparisons cannot distinguish useless structure from structure redundant given the features.Prior results include an MLP matching GNN-based diagnosis, a graph-free RCAEval method, and a propagation-aware re-evaluation [1] [2] [3].Source: §1 · p. 2
  • The graph arms share features, hidden width, dropout, scoring head, optimiser, validation fraction, early stopping and listwise ranking loss.The gnn and gnn_no_neigh arms differ only by the neighbour term; the conventional MLP differs in depth and capacity too.Source: §1 · p. 2
  • The paper isolates graph access with three learned arms across two benchmarks, two topology sources and four regimes, reporting no reliable graph-specific effect.Its contributions also include an audit of RCAEval and a twelve-item checklist motivated by recorded defects.Source: §1 · p. 3

2 Related Work

Related work establishes both the prevalence of graph-based microservice RCA and the challenge of attributing reported gains to graph structure. The paper frames its controlled comparison against that work and draws on evaluation-methodology research for its testing and reporting practices.

2 Related Work

  • Graph-based RCA spans graph encoders for localization, multimodal dependency models, and datasets, with RCAEval also supplying the BARO reference implementation.Examples include MicroEGRCL, CHASE [7], Eadro, and LEMMA-RCA [9].Source: §2 · p. 3
  • The paper asks whether graph access matters when the rest of the pipeline is fixed, in response to graph-sceptical comparisons [1] [2] [3].Those comparisons report MLP parity, a graph-free method above RCAEval baselines, and a re-evaluation that lowers eleven models to 0.21 average Top@1.Source: §2 · p. 3
  • The study follows evaluation work advocating paired non-parametric tests, attention to benchmark variance, and stronger reporting norms [10–15].Its twelve-item checklist is described as the practical residue of this evaluation-methodology line.Source: §2 · pp. 3–4

3 Experimental Setup

The experiments use RCAEval RE1 and RE2, rank services by likelihood of being the injected culprit, and compare matched models over disjoint folds. The setup also identifies a non-uniform RE1 schema that can silently replace telemetry with zeros.

3 Experimental Setup

  • RE1 contains 375 fault injections across three systems, while RE2 adds 90 Sock Shop cases with actual Kubernetes pod placement.The RE2 placement supports using co-location rather than inferred topology.Source: §3 · p. 4
  • The task ranks system services from a fault-window telemetry window, using AC@k and Avg@5 as benchmark metrics.Avg@5 is defined from AC@k in the supplied metric passage.Source: §3 · p. 4
  • A literal schema lookup silently returns nothing for 250 of 375 RE1 cases, producing zero-filled telemetry and empty graphs.RE1 uses different column conventions across Online Boutique, delay and loss cases, Sock Shop, and Train Ticket.Source: §3 · p. 4
  • Both encoders use matched width, dropout, head, optimizer, validation split and early stopping, as depicted in Figure 1.The figure isolates the single neighbour-access term between the graph arms.Source: §3 · p. 4, Figure 1 · p. 5
  • In-distribution evaluation uses six disjoint folds, while cross-system evaluation holds out three systems; paired testing treats each fold or held-out system as the unit.At n = 3, the smallest reachable p-value is 0.25, and at n = 6 the exact two-sided Wilcoxon minimum is 0.03125.Source: §3 · pp. 4–5

4 Localization: Structure Has No Reliable Effect

On RE1, controlled comparisons find no reliable effect of graph structure in either in-distribution or cross-system evaluation. The decomposition and fold audit further qualify pipeline-level contrasts and the apparent significance of overlapping resamples.

4 Localization: Structure Has No Reliable Effect

  • 0.003 Avg@5 is the graph model’s in-distribution lead, while it trails by 0.058 cross-system; neither difference is significant (p = 0.844 at n = 6; p = 0.25 at n = 3).Table 1 reports RE1 localization Avg@5 with fold-resampled 95% bootstrap intervals.Source: §4 · p. 5, Table 1 · p. 6
  • The in-distribution pipeline-level term varies in sign across folds, with values [+0.022, +0.013, +0.001, −0.003, −0.012, −0.001].The decomposition reports no significant or directionally stable effect.Source: §4 · pp. 5–6
  • Independent 70/30 resamples per seed appeared to show a clean, significant result attributed to encoder capacity, but because they overlapped, the effect did not survive the reported six-fold partition.Source: §4 · p. 6
  • Cross-system, the structure term is +0.101 and the capacity term −0.159 (both p = 0.25), but the plain MLP still beats the full graph model.The authors decline to claim a benefit because the structure term partly recovers a deficit introduced by the deeper graph architecture.Source: §4 · p. 6

5 What the Benchmark Was Really Measuring

A ranker using only injected-service names reaches Avg@5 0.488 in-distribution, making that prior the benchmark’s honest floor rather than uniform-random performance. Across systems the prior falls to Avg@5 0.192; the authors suggest learned models partly absorb it, but state this as a hypothesis, not a measurement.

5.1 A telemetry-blind ranker scores 0.488

  • Avg@5 0.488 and AC@5 0.997: a ranker using only training-time injected-service names puts the culprit in the top five on 99.7% of held-out incidents.It never reads an incident’s feature vector.Source: §5.1 · pp. 6–7
  • The prior, not uniform-random 0.137, is the honest in-distribution floor; the best learned arm’s margin over it is 0.379.Faults target five services per system, despite 12–70 services emitting telemetry on RE1 and five of fifteen on RE2.Source: §5.1 · pp. 6–7

5.2 A candidate mechanism for the transfer collapse

  • Avg@5 0.192 cross-system, the prior loses 0.296 under transfer, more than either learned model; service names do not transfer.The authors hypothesize that learned models partly rely on which services the benchmark injects faults into, but say the decomposition bounds rather than measures memorization.Source: §5.2 · pp. 7–8
  • The service list derived from column names includes node-exporter targets alongside real services on Sock Shop and Train Ticket, inflating apparent RE1 ranking difficulty.The authors flag this for interpreting AC@k.Source: §5.2 · p. 8

6 A Prior-Separated Decomposition: Where the Score Comes From

PSC-GRCA separates candidate scores into a system prior, telemetry evidence, and a centred graph residual. Its in-distribution gain is concentrated in the prior, while transfer and implementation limits constrain the interpretation.

Decomposition

  • The score adds a system prior, telemetry evidence, and a graph residual constrained to have zero mean across candidate services.The centring prevents the residual from absorbing the constant offset supplied by the system prior, leaving only the graph signal's shape.Source: §6 · p. 8

In-distribution results

  • Mean Avg@5 is 0.915 for PSC-GRCA, versus 0.864 for the flat MLP and 0.862 for the capacity-matched no-neighbour control.All six paired fold differences against both controls are positive (p = 0.03125).Source: Table 2 · p. 9, §6 · p. 9

Protocol

  • A prior-swap penalty is exactly zero in this architecture, so the no-swap arm reproduces the same-seed run and serves only as a check on penalty dependence.Ablation arms score terms from the same model trained on the full score, rather than separately trained models.Source: §6 · p. 9

Ablation results

  • The prior-only arm scores 0.488, while the graph residual scores 0.800 and adds nothing over the evidence arm's 0.853.The reported gain over the flat baseline is attributed to the prior; PSC-GRCA is presented as an attribution instrument, not a performance contribution.Source: §6 · p. 10

Cross-system results

  • Under cross-system transfer, the full model and prior-free arm both reach 0.747, while the prior-only arm scores 0.079.On 374 of 375 held-out cases, every candidate has a zero prior count, making the prior a constant offset; the prior-only score reflects tie order rather than transferable prior knowledge.Source: §6 · p. 10

Scope and limitations

  • The study is exploratory and underpowered for transfer and ablation attribution, and PSC-GRCA has not been ported to RE2.An earlier RE2-labelled run used RE1 folds; the authors withdraw it and make no PSC claim on RE2 until the model is ported.Source: §6 · pp. 10–11

7 What a Published Baseline Reveals About the Schema

On Online Boutique, BARO and the max-deviation heuristic have similar aggregate accuracy, but they use different scoring rules. Elsewhere, schema and column-to-service mapping affect how the comparison should be read.

Online Boutique comparison

  • On Online Boutique, aggregate accuracy is 0.880 for BARO versus 0.877 for the heuristic, a difference within 0.004.This is a published-baseline check, not a leaderboard; the two methods score differently and can disagree on individual incidents.Source: §7 · p. 11

Scoring rules

  • BARO ranks raw columns, while the heuristic aggregates curated metric families to services; their agreement on one system does not make the scoring rules equivalent.BARO uses a signed transformed sample after RobustScaler; the heuristic uses clipped absolute standardized post-fault means over four curated metric families.Source: §7 · p. 11

Schema effects

  • The divergence on cAdvisor systems reflects schema and column-to-service mapping, not a demonstrated quality difference between methods.BARO ranks hundreds to over a thousand raw counters, while the heuristic uses an alias table to aggregate services into four curated metric families.Source: §7 · pp. 11–12

8 RE2: A Real Topology Does Not Change the Answer

The RE2 comparison finds no evidence that a real co-location topology improves localization, including when telemetry is masked. Aggregate blind-spot results reverse the hypothesized ordering, but cannot support significance claims.

8 RE2: A Real Topology Does Not Change the Answer

  • RE2 uses actual Kubernetes co-location, a mechanistically appropriate topology because four of six fault types involve resource contention through a shared machine.Source: §8 · p. 12
  • Fully observed, all arms are near ceiling on AC@1 and no pairwise difference approaches significance at n = 3.Scores are MLP 0.978, no-neighbour control 0.967, induced graph 0.970, and real co-location 0.963.Source: §8 · p. 12
  • At 100% masking, the no-neighbour control scores 0.104, above the induced graph at 0.056 and real co-location graph at 0.000.The manifest retains only aggregate scores at each masking rate, so these are point estimates without significance claims.Source: §8 · p. 12
  • Across aggregate masking estimates, learned arms exceed the heuristic under non-total masking, but both the heuristic and real-topology arm fall to zero at 100%.The retained evidence does not show whether this pattern holds fold by fold.Source: §8 · p. 12

9 A Checklist for Graph-versus-Flat Ablation Studies

The twelve-item checklist distills sixty-two recorded defects into checks for graph-versus-flat ablation studies. It emphasizes that negative results require verified implementations, independent folds, valid tests, and non-degenerate controls.

9 A Checklist for Graph-versus-Flat Ablation Studies

  • Twelve checks distilled from sixty-two defects form the practical residue of the study, and roughly half the failures could have survived ordinary code review.Each check is motivated by a specific failure it would have caught.Source: §9 · p. 12
  • Check task-level leakage, message-passing reach to scored nodes, and matched training protocols; correcting one control reversed the headline result.Source: §9 · p. 13
  • Test significance over independent folds, not repeated measurements or overlapping resamples, which can manufacture apparently significant effects.One resampling error produced folds sharing 32% of test cases and a reported p = 0.031 for an effect absent under a true partition.Source: §9 · pp. 13–14
  • Verify tooling with negative controls, require real between-arm tests, generate figures deterministically, and state the decision rule before seeing results.Source: §9 · p. 14
  • Check that every baseline and control can vary its outputs; degenerate controls can be reported as independent despite identical predictions.The authors conclude that null findings require verified implementations and tooling, tested comparisons, independent folds, and non-degenerate controls.Source: §9 · p. 14

10 Limitations

The study’s conclusions are bounded by limited architecture and benchmark coverage, simulated telemetry loss, and few independent transfer systems. PSC-GRCA is exploratory, with its ablations unable to isolate the prior contribution from the chosen parameterization.

10 Limitations

  • The graph comparison covers one architecture family, so confirming that the null is not architecture-specific requires another family.Source: §10 · p. 14
  • RE2 covers one system and 90 cases, while the blind-spot regime simulates telemetry loss that may be correlated with faults in practice.Source: §10 · p. 14
  • PSC-GRCA is exploratory, and its single prior feature and architecture cannot separate the prior’s contribution from its particular parameterization.Its transfer estimate uses the same three held-out systems, and the model has not been ported to the RE2 loader.Source: §10 · pp. 14–15
  • Three held-out systems make transfer significance testing underpowered; pairing folds with single initializations also adds noise and reduces detection power.Source: §10 · p. 15

11 Conclusion

Across the tested RCAEval settings, matched comparisons find no reliable graph-specific effect, while the benchmark audit identifies properties that condition its scores. An exploratory prior-separated model performs strongly in-distribution, with most of its gain attributed to the prior rather than the graph.

11 Conclusion

  • Across tested families, benchmarks, topologies, and regimes, graph structure has no reliable effect on RCA localization under matched protocols.A deviation heuristic beats learned models under cross-system transfer and loses decisively under blind spots; the latter comparison is aggregate-only and has no significance claim.Source: §11 · p. 15
  • RCAEval’s telemetry-blind culprit prior reaches Avg@5 0.488 in-distribution but collapses across systems, and its non-uniform schema silently zeroes telemetry for most RE1 cases.The reproduced published baseline has similar aggregate accuracy to the heuristic on the one system with a clean schema, under a different scoring rule.Source: §11 · p. 15
  • PSC-GRCA reaches mean Avg@5 0.915 in-distribution and 0.747 under transfer, while ablations locate most in-distribution gain in the prior term.The result is exploratory, and the authors read the separation as making attribution visible rather than vindicating graph structure.Source: §11 · p. 15
  • Next steps are a second architecture family, a second prior feature, and more held-out systems to distinguish separation effects and test transfer.Source: §11 · p. 16
  • The research artifact includes code, seed manifests, a sixty-two-entry defect register, and scripts that generate tables and figures from manifests.Source: §11 · p. 16
Loading 2609.27069v1…