PaperlayerView exact revision on arXiv

Source-linked AI summary

The Abstention Protocol: RCA for Clos Fabrics

Madhava Gaikwad, Deepak Pandey

arXiv:2608.21412v1cs.AIcs.DCcs.NI

TL;DR

Root cause analysis in large Clos networks must handle noisy, partial, asynchronous telemetry and failure-specific agent reliability. CoreSec composes heterogeneous agents with PAM-style abstention and topology-aware heuristics, reducing false positives below 1% across more than 700,000 production incidents.

  • Problem

    Root cause analysis is difficult in large Clos networks because many entities show faults while only a subset relate to an incident, limiting weighted agent fusion under failure-specific reliability.

  • Method

    CoreSec composes heterogeneous agents with PAM-style abstention and topology-aware fan-out rules for hierarchical Clos fabrics.

  • Results

    False positives fell from 18-22 percent to below 1 percent across more than 700,000 incidents in a three-year production deployment.

  • Takeaways & Limitations

    Three years across more than 60 Azure regions support explicit abstention and flag-based composition as a strong default for fusing heterogeneous, asynchronous, and partial signals.

  • Takeaways & Limitations

    The evaluation used postmortem engineer-assigned RCAs as ground truth and did not run blinded evaluations.

Abstract

from arXiv · show

Root cause analysis (RCA) in large datacenter networks is challenging because telemetry is noisy, partial, and asynchronous. Score-based approaches degrade under these conditions, often yielding unstable or incorrect attributions. We present \textsc{CoreSec}, a production RCA system that replaces weighted fusion with a PAM-style abstention algebra. Telemetry agents are composed using control flags that yield deterministic decisions and explicit abstention when evidence is ambiguous. CoreSec combines this algebra with topology-aware configurations that capture failure surfaces across Clos fabrics and converge monotonically as evidence accumulates. Deployed at hyperscale, CoreSec provides stable and explainable RCA behavior across diverse environments without retuning. Our experience shows that structured composition with abstention forms a practical foundation for automated RCA in real-world cloud networks.

1 Introduction · 2 Background and Motivation

CoreSec addresses noisy, asynchronous RCA in hyperscale Clos fabrics by combining topology-aware configurations with a PAM-style algebra that produces deterministic attributions or explicit abstentions. The system was designed for gray failures and heterogeneous telemetry, and its three-year deployment reduced false positives to below 1 percent across more than 700,000 incidents.

  • 1 Introduction: CoreSec targets gray failures such as intermittent packet loss, path-specific corruption, and transient reboots whose symptoms blend into the normal fault background.These partial and probabilistic malfunctions are not fail-stop failures and complicate attribution.
  • 1 Introduction: CoreSec replaces weighted telemetry aggregation with a PAM-style algebra whose requisite, required, sufficient, and optional flags yield deterministic decisions or abstention under missing or conflicting evidence.The algebra is specified as a three-valued lattice with associativity, absorption, and consensus properties supporting order-invariance under asynchronous telemetry.
  • 1 Introduction: Five parallel configurations target distinct Clos failure surfaces, while topology heuristics combine their votes across server-to-TOR, TOR-to-T1, and cluster levels.The configurations target server-to-TOR cables, TORs, switch-to-switch cables, T1s, and T2s; thresholds generalize across Azure deployments without recalibration.
  • 1 Introduction: Three-year deployment processed over 700,000 incidents, reducing false positives from 18–22 percent to below 1 percent and abstentions from 10 percent initially to 1.5 percent.CoreSec eliminated three full-time engineers worth of manual RCA work.
  • 2 Background and Motivation: Clos fabrics provide many equal-cost paths and rapid rerouting, but continuous background faults make customer-visible incidents coincide with unrelated link, device, and upgrade problems.A TOR failure usually affects one rack, whereas a T1 failure creates correlated symptoms across multiple TORs.
  • 2 Background and Motivation: Telemetry agents provide complementary but incomplete evidence: active probes offer rapid path detection, counters expose hardware signals with noise and irregularity, and traffic signals reflect customer impact with sparse coverage.No single agent is reliable across all failure modes, so multi-agent fusion is essential.
  • 2 Background and Motivation: Operational RCA must control false positives, explain decisions from agent outputs, incorporate new agents, and account for correlations across the Clos hierarchy.CoreSec emerged after reliability proved failure-specific rather than agent-specific and after infrastructure signals introduced feedback loops.

3 PAM-Style Composable RCA Algebra

CoreSec uses a PAM-inspired abstention algebra to combine heterogeneous, asynchronous, and partial telemetry into deterministic healthy, unhealthy, or indeterminate states for topology-aware RCA. Control flags, freshness windows, and abstention regions prioritize reliable evidence while improving controllability, explainability, and extensibility.

  • PAM-Style Composable RCA Algebra: CoreSec maps heterogeneous telemetry into healthy, unhealthy, or indeterminate states that feed topology-aware heuristics for identifying root causes.The model suits Clos fabrics because telemetry differs in granularity, latency, reliability, and scope.
  • Control Flags: Control flags make RCA decisions deterministic: requisite failures abstain immediately, required failures abstain at the end, optional evidence supports, and sufficient passes vote immediately.These flags encode empirical differences among telemetry agents, including reliability, evidential strength, and dependence on combination with other signals.
  • Evaluation Logic and Freshness: Freshness windows ignore stale data, treating absent fresh requisite evidence as failure and absent fresh required or optional evidence as abstention during partial outages or control-plane churn.Each window is determined by the agent’s reporting latency.
  • Per-Agent Thresholds and Abstention Region: Per-agent thresholds create an abstention region that absorbs benign fluctuations, making abstention safer than forced classification amid noisy, frequent, non-fatal events.Counter spikes may reflect traffic bursts, while probe loss may reflect transient congestion rather than real failures.
  • Why PAM-Style Composition Provides Operational Strength: The algebra improves controllability by gating decisions and bounding sensitivity, while decision traces record agent verdicts and flag effects for auditing and postmortems.These properties address limitations of weighted or probabilistic RCA under missing data or distribution shift.
  • Why PAM-Style Composition Provides Operational Strength: New telemetry agents can be integrated by assigning a control flag, allowing instrumentation to evolve without changing existing configurations.This simplifies maintenance as instrumentation evolves.

4 Composable RCA Configurations

CoreSec applies PAM-style abstention through five parallel RCA configurations, each tied to a Clos failure surface and defined by its agents, control flags, and evaluation order. The same configurations generalize across heterogeneous Azure environments without per-cluster recalibration.

  • Configuration framework: Five independent configurations run in parallel, each encoding authoritative telemetry for one Clos failure surface and voting or abstaining based on available evidence.Parallel execution supports incidents spanning multiple topology layers.
  • Configuration framework: Each configuration specifies its telemetry agents, each agent’s control flag, and the agents’ evaluation order.
  • Stability Across Deployments: The same five configurations and flag assignments operate across heterogeneous Azure environments without environment-specific tuning or per-cluster recalibration.They remained reliable across differing vendors, traffic compositions, telemetry pipelines, and probing strategies because physical failure signatures are consistent and flag composition absorbs telemetry variability.
  • Failure-surface configurations: Server–TOR and switch–switch configurations require direct or corroborating link evidence, abstaining when essential signals or probe coverage are missing to avoid transient-noise misattribution.Server–TOR uses direct link-health indicators; switch–switch uses active path probes confirmed by TOR or T1 counter anomalies.
  • Failure-surface configurations: TOR, T1, and T2 configurations require increasingly strong multi-agent and multi-path agreement because higher-layer evidence is noisier, sparser, and more indirect.Optional control-plane signals strengthen evidence but cannot independently trigger attribution; T2 votes only on high-confidence evidence across independent probe groups.

5 Parallel Execution and Hierarchy Heuristics

CoreSec executes five RCA configurations in parallel, allowing evidence from multiple Clos layers to be evaluated without premature commitment. PAM-style per-configuration decisions are combined with topology-aware hierarchy heuristics that select responsible layers while preserving independent failures.

  • Parallel Execution: Five configurations run concurrently, each producing votes or abstentions before candidate entities are combined and hierarchy heuristics determine root causes.This preserves candidate explanations until sufficient evidence accumulates rather than using weighted aggregation or static priority.
  • Hierarchy Heuristics: PAM algebra fuses heterogeneous evidence within each failure surface, while cross-configuration heuristics encode topology rules for failures propagating up the Clos hierarchy.The split lets algebra handle per-entity evidence and heuristics resolve simultaneous votes across layers.
  • Server to TOR Attribution: TOR attribution requires both the P2.15 outlier threshold and at least 20 percent server impact, identifying concentrated failures while filtering incidental noise.P2.15 is the 97.85th percentile of impacted-server counts across TORs; historical incidents showed the combined criteria detected TOR failures accurately while filtering background noise.
  • TOR to T1 Attribution: The two-thirds T1 fan-out rule identifies upstream failures when at least two thirds of connected TORs are unhealthy, stabilizing attribution despite asynchronous telemetry.One half produced false positives, three quarters delayed attribution, and two thirds matched T1 failures with minimal false positives.
  • Cluster Level Attribution: At cluster level, an aggregate probe drop rate above four percent declares a cluster incident, providing an additional signal for broad failures without usually suppressing lower-layer RCA.The threshold balances sensitivity and noise filtering because customer-visible impact becomes noticeable around three to five percent packet loss.
  • Cross-layer Combination: In a representative incident, 35 of 48 TORs showed correlated probe loss, triggering T1 attribution and suppressing individual TOR candidates; the final RCA was the T1 switch.This demonstrates how parallel configurations and hierarchy heuristics route evidence to the correct layer and prevent premature convergence on the wrong layer.

6 Composition Logic and Convergence

CoreSec merges votes and abstentions from five topology-aware configurations using hierarchy-based dominance rules, producing stable RCA under asynchronous telemetry arrival. Its bounded reruns converge monotonically, finalize after sixteen minutes, and abstain when no layer becomes sufficient.

  • Composition Across Configurations: CoreSec aggregates votes and abstentions from five configurations in a fixed hierarchy-aware order, preserving lower-layer attributions unless a higher layer meets its sufficiency condition.Cable, TOR, T1/T2, and cluster-level evidence are combined across the configurations.
  • Dominance Conditions: Higher-layer attributions suppress lower-layer candidates when correlated failures meet explicit thresholds: two-thirds unhealthy TORs for T1, multi-pod correlation for T2, or probe drop above 4% for clusters.These dominance rules are intended to reduce false positives during large-scale failures.
  • Monotonic Convergence: During a sixteen-minute post-detection window, CoreSec repeatedly incorporates asynchronous telemetry and converges monotonically: higher-layer replacements are permanent once their sufficiency conditions are met.The window covers the slowest signal latency plus a margin for late evidence, while fresh-data filtering prevents stale observations from reversing decisions.
  • Finalization: After sixteen minutes, CoreSec finalizes the RCA or abstains if no layer is sufficient, using a five-minute rerun cadence chosen to balance responsiveness against intermediate-attribution noise.The window is bounded below by the roughly thirteen-minute latency of the slowest agent; the design avoids oscillations seen in statistical voting and ML classifiers.

7 Intentional Abstention

CoreSec treats abstention as a safety mechanism for noisy or ambiguous RCA, preventing harmful misattribution and routing unresolved incidents to human operators. It abstains when all five configurations are inconclusive, may resolve as evidence accumulates, and provides structured context for operators and automation.

  • When CoreSec Abstains: CoreSec abstains when all five configurations are inconclusive, especially under telemetry gaps, conflicting signals, or multi-cluster events.Telemetry gaps account for roughly 60% of abstentions, ambiguous evidence for roughly 39%, and multi-cluster or datacenter-wide events for roughly 1%; the first two prevent misattribution.
  • Convergence and Resolution: During a sixteen-minute convergence window, early abstention can resolve to a valid attribution once sufficient data arrives, while persistent abstention pauses automated workflows.CoreSec repeatedly runs RCA during the convergence window; persistent abstention signals operators to pause automated workflows.
  • Convergence and Resolution: In a representative trace, T1 dominates after three reruns, suppressing TOR candidates while other configurations abstain or provide partial votes.The trace records final attribution T1-XXX and shows T1 receiving 33/47 unhealthy TOR votes; resolution is “T1 dominates; TOR candidates suppressed.”
  • Operator Response: When CoreSec abstains, it suppresses automated mitigation and routes incidents to on-call engineers with a structured summary of evidence gaps, partial signals, and update times.The summary identifies agents lacking fresh telemetry, abstaining configurations, entities with partial signals, and timestamps of last updates.
  • Design Principle: CoreSec makes abstention a design principle for high-stakes incidents, automating common cases while routing rare or ambiguous cases to humans to preserve operator trust.Incorrect diagnoses can degrade operator trust, and automated mitigations increase the cost of incorrect attribution; automated fault isolation and failover orchestration further raise false-positive costs [23].

8 Evaluation

CoreSec was evaluated across more than three years of hyperscale deployments, measuring accuracy, stability, convergence, generalization, and operational impact against the prior weighted-scoring system. Results show substantial improvement and declining abstention, while highlighting limitations from non-blinded postmortem labels and conservative coverage thresholds.

  • Deployment and Workload: CoreSec operated continuously across more than 60 Azure regions and 400 datacenters for over three years, processing more than 700,000 network incidents.The incidents covered link failures, switch reboots, configuration churn, and device maintenance across diverse hardware, traffic, and topology configurations.
  • Threats to Validity: Evaluation relies on postmortem engineer RCAs that were not blinded after CoreSec outputs became visible, and follow-up backtesting and matched prior-system evaluation remain planned.The authors identify labeler bias as the largest validity threat despite independent baseline and operational evidence supporting the improvement.
  • Accuracy and False Positive / False Negative Rates: A twenty-fold improvement over the pre-CoreSec 18–22% baseline was accompanied by an 80% reduction in mis-triggered mitigations.The baseline was measured before CoreSec existed on the same network by the same operations team; downstream operational outcomes provided independent supporting evidence.
  • Why Composition, Not Just Abstention: The PAM algebra makes abstention meaningful by distinguishing absent or conflicting evidence, whereas weighted-score thresholds collapse heterogeneous evidence and can preserve unstable attributions.Its explicit disagreement state prevents guessing, while composition supports deterministic resolution as evidence accumulates.
  • Accuracy and False Positive / False Negative Rates: Abstentions fell from 10% after deployment to 1.5% in the most recent six months, primarily as added agents closed coverage gaps.Abstentions are counted as false negatives, and misattributions occurred roughly once per quarter; Figure 4 plots the abstention and false-positive trajectories.
  • Case Study: A T1 Optical Degradation: False negatives mainly occur when fewer than two thirds of a failed T1’s TORs report unhealthy states within the convergence window, reflecting a conservative false-positive tradeoff.In a representative optical-degradation case, the T1 attribution dominated and suppressed individual TOR candidates, with the trace making the resolution auditable; Pingmesh supplied degraded-latency evidence without identifying the device.

9 RCA Quality Metrics and KPI Framework

CoreSec measures RCA quality hierarchically from infrastructure domain to networking layer and specific root cause, using longitudinal KPI distributions informed by AIR methodology [31]. Across three years, abstention, convergence, deployment stability, and threshold behavior remained strong without retuning.

  • RCA Quality Metrics and KPI Framework: The KPI framework evaluates infrastructure, networking-layer, and specific-root-cause attribution, with aggregated Level 1–3 distributions monitored longitudinally.Persistent Level 2 or Level 3 degradation indicates systemic problems, while reductions in those frequencies validate CoreSec improvements; the framework follows AIR principles [31].
  • RCA Quality Metrics and KPI Framework: Abstention fell from 10 percent to 1.5 percent while thresholds remained constant throughout the three-year deployment.AIR analysis identified clustered abstentions, motivating new telemetry agents integrated into existing PAM configurations.
  • Convergence Behavior and Stability: In roughly 99% of cases, RCA stabilized within the first two reruns, with monotonic outcomes and no observed oscillations.CoreSec reruns RCA every five minutes for up to sixteen minutes; higher-layer attributions suppress lower-layer candidates permanently.
  • Generalization Across Deployments: The same configurations, control flags, and heuristics generalized across diverse datacenter environments and hardware transitions without per-deployment tuning.Novel failure modes require new agents, but existing five configurations and flag-assignment rules were retained without redesigning the algebra or heuristics.
  • Sensitivity Analysis of Thresholds: Sensitivity analysis found lowest total error near a 0.66 fan-out threshold, a 4% cluster drop threshold, and P2.15 near the 97th–98th percentile.Lower or higher fan-out and cluster thresholds increased false positives or false negatives, while P2.15 separated impactful TORs from heavy-tailed background noise.

10 Discussion

CoreSec’s thresholds generalize because they encode properties of Clos fabrics, while its PAM-style composition provides deterministic, auditable attribution under partial and heterogeneous telemetry. The discussion also identifies limitations, justifies operational baselines and runtime design choices, and extends the abstention pattern beyond network RCA.

  • Why These Thresholds Generalize: CoreSec’s thresholds generalize across deployments because they reflect Clos-fabric properties rather than individual workloads.The two-thirds fan-out rule captures correlated propagation, P2.15 uses percentile-based outlier detection, and Figure 5 reports knees holding across ten deployments.
  • Comparison with Prior Fault-Localization Systems: CoreSec differs by targeting network RCA in large Clos fabrics and composing heterogeneous agents with PAM-style algebra instead of machine learning or weighted scores.This design exploits hierarchical failure propagation while avoiding score-based fusion.
  • Limitations: CoreSec’s limitations include novel failure modes, telemetry gaps that force abstention, and catastrophic failures that intentionally produce mass abstention.Most manual corrections arise from upstream telemetry issues rather than misclassification.
  • Why We Avoid ML at Runtime: CoreSec uses ML upstream for signal extraction but deterministic composition for final decisions, enabling predictable behavior, explicit abstention, and auditable decision traces.Supervised fusion is unstable under intentional abstention, distribution shift, and evolving telemetry-agent behavior.
  • Lessons for Other Operational Systems: The discussion generalizes explicit-abstention composition to service RCA, control-plane debugging, storage attribution, and security triage where noisy partial signals make forced classification unreliable.The proposed stance treats heterogeneous signal fusion as a composition problem rather than weighted scoring.

11 Related Work · 12 Conclusion

CoreSec distinguishes itself from prior systems by composing heterogeneous telemetry into stable root-cause decisions with explicit abstention. Its three-year deployment across more than 60 Azure regions supports flag-based composition as a strong default for noisy operational RCA.

  • 11 Related Work: CoreSec targets plausible incident root causes amid many unhealthy components and background noise, unlike fault-localization systems that identify unhealthy links or devices [11].It answers which components form a plausible root cause for a specific incident rather than merely which components are unhealthy.
  • 11 Related Work: Telemetry systems obtain measurements, whereas CoreSec fuses heterogeneous signals into stable RCA decisions.Everflow mirrors packets, INT collects in-band measurements, and OmniMon provides flow statistics under loss.
  • 11 Related Work: CoreSec assumes incident detection and identifies responsible network entities, complementing incident-management systems that detect, route, or tag incidents.Fighting the Fog of War routes incidents to owners, while AutoARTS labels them with root-cause tags.
  • 11 Related Work: CoreSec provides a hierarchical structure into which statistical-correlation or LLM-based RCA modules can be plugged [40, 43, 44].RCA Copilot, OpenRCA, and KPIRoot+ use statistical correlation or LLMs for diagnosis.
  • 11 Related Work: CoreSec adapts PAM’s control-flag composition pattern to RCA, specifying how each telemetry agent contributes to a decision.The adaptation transfers a compositional mechanism introduced for authentication checks into telemetry-driven root-cause analysis.
  • 12 Conclusion: CoreSec is a production root-cause analyzer for hyperscale Clos fabrics that treats RCA as a composition problem and explicitly permits abstention when evidence is inconclusive.The system gives operators an explicit option not to attribute an incident under ambiguous evidence.
  • 12 Conclusion: Three years across more than 60 Azure regions support flag-based composition with explicit abstention as a strong default for fusing heterogeneous, asynchronous, and partial signals.The conclusion suggests that other domains with the same problem shape may benefit from this pattern.

A Algebraic View of the CoreSec Merge Rule

CoreSec formalizes multi-agent RCA merging as a three-state abstention algebra with deterministic, associative, order-invariant behavior. Its rules make failure decisive, require unanimous health evidence, and preserve unresolved states under conflict while telemetry converges.

  • Decision Alphabet: Each agent emits healthy H_r(e), unhealthy U_r(e), or abstain A_r(e), which CoreSec reduces to merged states H, U, and indeterminate I.Abstentions are removed before merging, while I represents unresolved evidence.
  • Merge Rule: The merge rule maps any decisive failure to U, unanimous healthy evidence to H, and disagreement to I.Thus, one solid failure matters, health requires agreement, and conflict means wait.
  • Associativity: Associativity makes outcomes independent of agent ordering and supports correct streaming updates when telemetry arrives asynchronously.The proof reduces cases with any U to U, all H to H, and all other combinations to I.
  • Identities and Absorption: Abstention is neutral: it neither enforces H nor overrides U, and it does not suppress I when evidence conflicts.This prevents silence or partial information from distorting RCA decisions.
  • Operational Consequences: As telemetry becomes fresh, states converge monotonically from I to H or U, while the algebra preserves safety and order-invariance across evaluation modes.A single false healthy report cannot mask a real failure.
Loading 2608.21412v1…