PaperlayerView exact revision on arXiv

Source-linked AI summary

Hybrid Monitoring for Early Fault Detection in Cloud-Native 5G Systems

Anton Andersson, Sai Akshara Naineni, Mats Jansborg, Yixing Zhang, Romaric Duvignau

arXiv:2609.12649v1cs.NIcs.DC

TL;DR

Cloud-native 5G monitoring tools leave gaps in detecting partial network degradation and verifying idle paths. NetMon uses distributed agents with local anomaly detection and centralized correlation; it detects subtle latency or packet-loss faults and localizes them within seconds, including under 50% load.

  • Problem

    Standard health checks miss partial degradation, scrape-based systems introduce delays of tens of seconds, and passive tools cannot verify idle network paths.

    Source: §6.1 · p. 15

  • Method

    NetMon uses a monitoring architecture with distributed agents, local anomaly detection, and centralized correlation.

    Source: §2 · p. 3

  • Results

    NetMon detects faults as subtle as 10ms latency or 5% packet loss, localizes them to the affected worker within seconds, and maintains this capability under 50% load.

    Source: §4.5 · p. 13

  • Takeaways & Limitations

    The hybrid approach provides early detection and fault localization for maintaining service quality in cloud-native 5G infrastructure.

    Source: §6.1 · p. 15

    Severe bandwidth throttling can starve metric reports.

    Source: §5.2 · p. 15

Abstract

from arXiv · show

This paper presents the design implementation and evaluation of NetMon a hybrid network monitoring system designed for Kubernetes-based 5G packet core deployments specifically evaluated on Ericssons Access and Mobility Management Function AMF clusters NetMon combines eBPF-based passive kernel-level traffic observation with active TCP probing and centralized correlation to detect and localize network degradation within seconds The evaluation results demonstrate that the system detects faults as subtle as 10ms of added latency or 5 packet loss correctly attributes them to the affected infrastructure component and maintains this capability under application loads up to 50 simulated UE load The total resource overhead of 34 millicores CPU and 45 MiB memory per pod suggests that the approach is promising for further validation without impacting the monitored workload The hybrid approach addresses a gap in existing monitoring tools standard health checks cannot detect partial degradation scrape-based systems introduce detection delays measured in tens of seconds and purely passive tools cannot verify idle network paths By combining these complementary techniques and centralizing the analysis the system provides the early detection and fault localization capabilities required for maintaining service quality in cloud-native 5G infrastructure.

1 Introduction

Cloud-native 5G networks make subtle degradation difficult to localize, motivating NetMon’s lightweight hybrid monitoring and cluster-wide correlation approach. The paper asks how to detect and localize early degradation and contributes an architecture, correlation engine, visualization pipeline, and live AMF evaluation.

1 Introduction

  • Small increases in jitter, tail latency, and packet drops can precede visible faults in distributed systems, where network problems are difficult to distinguish from application or shared-resource issues.These challenges grow as cloud-native 5G core networks use dynamic microservice communication paths.Source: §1 · pp. 1–2

1.1 Related Work

  • Existing monitoring leaves gaps: liveness checks miss early inter-service degradation, scrape-based metrics can miss transients, and passive telemetry may not distinguish degradation from normal variation.NetMon combines passive eBPF observation with active TCP probes and centralized correlation to identify and localize failure patterns.Source: §1.1 · p. 2

1.2 Research Questions

  • The research asks how lightweight passive kernel monitoring and active probing can detect and localize early degradation, and how anomalies can distinguish pod faults, worker issues, and partitions.Source: §1.2 · p. 3

1.3 Contributions

  • The contributions are a hybrid monitoring architecture, cluster-wide correlation engine, Prometheus-integrated visualization pipeline, and live evaluation on a multi-worker AMF cluster.The evaluation targets detection timeliness, accuracy, and resource overhead, including internal communication paths that were previously unmonitored.Source: §1.3 · p. 3

2 System Design

NetMon is designed for early, lightweight, transparent monitoring with cluster-wide fault correlation. Its sidecar agents combine peer probing and eBPF observation, while a central server aggregates reports and tracks cluster state.

2 System Design

  • NetMon’s design targets detection within seconds, low resource use, cluster-wide fault isolation, and deployment without changing monitored application code.Source: §2 · pp. 3–4

2.1 Architecture Overview

  • Sidecar agents on monitored pods probe peers, monitor traffic with eBPF, detect anomalies locally, and report findings every 5 seconds to a central server.The server correlates reports, maintains per-pair baselines using CUSUM change-point detection, maps topology, and tracks pod and worker health.Source: §2.1 · p. 4
  • Kubernetes headless-service discovery lets agents resolve current peer pod IPs as pods are added, removed, or rescheduled.Source: §2.1 · p. 4

2.2 Distributed Monitoring Agent

Each NetMon agent combines active peer probes with passive kernel telemetry and local anomaly detection. Its probe protocol checks peer connectivity and topology, while eBPF programs measure traffic and latency and support threshold-based anomaly reporting.

Active Probing:

  • Every 5 seconds, agents probe each discovered peer over TCP with a 3-second timeout, exchange worker names, and track repeated failures per peer.A 60-second grace period keeps terminated pods in the monitoring set so probe failures can detect their disappearance.Source: Active Probing · p. 5

eBPF Passive Monitoring:

  • Three custom eBPF programs observe pod traffic directly from the Linux kernel without userspace scheduling overhead.Source: eBPF Passive Monitoring · p. 5
  • The XDP program tracks per-peer packet, byte, TCP-flag, and retransmission counters; it forwards reset and retransmission events for userspace processing.Potential retransmissions are detected with an LRU hash keyed by source IP, source port, and TCP sequence number, with a 50ms minimum age floor.Source: eBPF Passive Monitoring · p. 5
  • TC ingress and egress programs measure probe round-trip latency from outgoing SYN to the first data-bearing reply, including network, handshake, and remote processing time.Kernel-layer timestamps avoid local userspace scheduling effects; the egress program also tracks per-flow traffic counters.Source: eBPF Passive Monitoring · p. 5
  • Latency anomalies use an EMA baseline and trigger when the z-score exceeds 5.0 with at least 2ms absolute deviation; detection waits for 20 samples.RST and retransmission rates are flagged above 5× their EMA baseline or absolute thresholds of 5% and 10%, respectively.Source: eBPF Passive Monitoring · pp. 5–6
  • The agent reports probe failures and sustained failures, high resets or retransmissions, reset or retransmission spikes, packet-rate spikes, and traffic silence lasting over 30 seconds.Source: eBPF Passive Monitoring · p. 6

2.3 Central Server Design

The central server provides cluster-wide analysis by combining agent reports with topology and health-state tracking.

2.3 Central Server Design

  • The central server aggregates monitoring reports from all agents.Source: §2.3 · p. 6
  • It performs cluster-wide correlation and maintains topology information.Source: §2.3 · p. 6
  • It tracks pod and worker health states.Source: §2.3 · p. 6

Pod State and Health Classification:

The central server classifies each pod using failures and anomalies reported by other pods within a 30-second window. Sequential precedence determines which single health state is displayed.

Pod State and Health Classification:

  • Health classification counts probe failures and anomalies targeting each pod within a 30-second window.The state is based on reports from other pods, allowing detection of problems invisible to the target pod itself.Source: Pod State and Health Classification · p. 6
  • A pod is DOWN after no report for > 60s, FAILING at ≥9 probe failures, WARNING at ≥3 failures or ≥30 anomalies, or DEGRADED at ≥1 failure or ≥20 anomalies.HEALTHY applies when none of these thresholds is met.Source: Pod State and Health Classification · p. 6
  • When multiple thresholds apply, sequential precedence displays only the highest-priority state, such as FAILING rather than WARNING.The stated order is DOWN, FAILING, WARNING, DEGRADED, then HEALTHY.Source: Pod State and Health Classification · p. 6

Correlation Engine:

The correlation engine recomputes patterns from anomaly history in a 30-second sliding window on each incoming report. Its nine patterns span pod anomalies, connectivity, and worker health.

Correlation Engine:

  • The engine reevaluates anomaly history in a 30-second sliding window whenever a report arrives, identifying nine patterns in three categories.Source: Correlation Engine · p. 7
  • Pod-level patterns flag ≥9 probe failures to one peer, ≥15 RTT/jitter anomalies targeting a pod, or ≥30 anomalies targeting a pod.Source: Correlation Engine · p. 7
  • Connectivity patterns flag failure to reach ≥3 same-worker peers, an inter-worker link below 50% success while other links remain, or a pod below 10% success while other same-worker pods remain.Source: Correlation Engine · p. 7
  • Worker patterns include external reports of anomalies with health ≥10 below average, >50% of inter-worker links above 10ms RTT, or ≥40 anomalies about ≥3 other workers.Source: Correlation Engine · p. 7

Online Change-Point Detection (CUSUM):

NetMon uses per-pair CUSUM to detect shifts in network measurements, combines probe success and anomaly rates for worker health, and exposes monitoring metrics through Prometheus for CNOM visualization.

Online Change-Point Detection (CUSUM):

  • CUSUM detects shifts in per-pair RTT, jitter, and retransmission measurements by accumulating standardized deviations from a learned mean.With k = 0.5 and h = 4.0, it requires 30 warmup samples and, for RTT only, a minimum 5ms absolute deviation; accumulators reset after detection.Source: Online Change-Point Detection (CUSUM) · p. 7

Topology Awareness and Worker Health:

  • Worker health scores combine probe success rate (weight 0.6) and anomaly rate (weight 0.4) on a 0–100 scale within a 30-second window.Probe handshakes provide worker assignments for topology-aware analysis; worker-pair traffic drops below 40% or spikes above 3× baseline are also detected.Source: Topology Awareness and Worker Health · p. 7

2.4 Reporting and Visualization

  • A Prometheus endpoint exposes pod states, peer RTT and jitter, throughput, worker scores, CUSUM statistics, and active correlations for visualization in Ericsson’s CNOM dashboard.Source: §2.4 · p. 7

3 Implementation and Deployment

NetMon is implemented with eBPF and full-mesh probing, then evaluated through controlled fault injections on a live Kubernetes AMF cluster. The evaluation tracks detection and localization latency, resource overhead, and network traffic.

3 Implementation and Deployment

  • The C implementation uses libbpf-loaded eBPF, push-based reporting, full-mesh probing, and statistical methods; the stated tradeoffs include Linux 5.8+ and scalability limits.The paper describes eBPF as lower-overhead than packet capture, push reporting as lower-latency but a single point of failure, and full-mesh probing as complete coverage with O(N^2) scalability limits.Source: §3 · p. 8

3.1 Evaluation Methodology

  • Evaluation ran on a live Kubernetes cluster with a virtual 5G AMF spanning four workers and ten pods across four microservice types.The topology had 2–3 pods per worker.Source: §3.1 · p. 8, Figure 2 · p. 8
  • Faults were injected on one target worker with the central server on another, across 0%, 20%, and 50% simulated UE loads; Table 1 summarizes scenario configurations.Scenarios were repeated at least five times at 0% load and twice at 20% and 50%; injections lasted two minutes with five-minute recovery periods.Source: §3.1 · p. 8, Table 1 · p. 9
  • The evaluation tracks detection latency, correctly localized correlation latency, CPU and memory overhead, and probe and report network traffic.Source: §3.1 · pp. 8–9

4 Evaluation

NetMon detected faults down to 10ms latency or 5% packet loss and localized them to the affected worker within seconds, with equal or improved results at 20% and 50% load.

4.1 CNOM Dashboard Visualization

  • A 20ms worker-2 latency injection produced anomalies, WORKER_HIGH_LATENCY and WORKER_HOTSPOT correlations, and falling health states, all returning to baseline after removal.The two-level display presents raw anomalies alongside correlations for operator fault identification.Source: §4.1 · p. 9, Figure 3 · p. 10

4.2 Resource Overhead

  • 3.4 millicores CPU and 4.5 MiB memory per agent; 10-pod overhead was <35 millicores and 47.6 MiB.Every 5 seconds, the system sends 90 probes under 200 bytes each and 10 reports of approximately 4–5 KB each.Source: §4.2 · p. 9

4.3 Baseline Behavior

  • 18-hour normal operation produced ∼2900 anomalies and 423 correlations per hour; PEER_DOWN and WORKER_ISOLATED never activated.The authors note that reducing anomaly volume would require raising thresholds and increasing detection latency.Source: §4.3 · p. 10, Table 2 · p. 11

4.4 Experiment Results

  • Latency-injection detection took 0–10s, while jitter detection took 0–5s and correlation 6–19s, identifying the affected worker in every run.For latency injection, correlation was 6–10s at 50% load versus 13–20s at 0% load.Source: §4.4 · p. 11, Table 3 · p. 11, Table 4 · p. 12
  • 3–17s detection at 5–20% packet loss rose to 6– 58s at 50% loss as faults dropped metric reports; retransmission-based correlation was probabilistic.Source: §4.4 · p. 12, Table 5 · p. 12

Bandwidth Throttle, Link Failure, Pod Termination, and Gradual

4.5 Summary

  • 10ms latency or 5% packet loss was detectable, with affected-worker localization within seconds and equal or improved performance at 20% and 50% load.Source: §4.5 · p. 13

5 Discussion and Limitations

NetMon detects latency and jitter faults within seconds and localizes them to the correct worker, with timing comparable to or faster than typical Kubernetes liveness probes. The discussion reports low measured overhead and identifies constraints involving threshold portability, probe scaling, centralized correlation, bandwidth, permissions, and visualization.

5.1 Discussion

  • Latency and jitter faults are detected within 2–10 seconds and localized to the correct worker within 6–20 seconds, comparable to or faster than 10–30-second Kubernetes liveness probes.Standard liveness probes cannot detect gray failures in which components remain alive but perform poorly.Source: §5.1 · pp. 13–14
  • NetMon probes every path every 5 seconds, whereas scrape-based alerting depends on scrape and evaluation intervals and passive eBPF cannot verify idle paths.The cited common Prometheus scrape interval is 15–60 seconds.Source: §5.1 · p. 14
  • Per pod, measured overhead was 3.4 millicores CPU and 4.5 MiB memory, less than 1% of application resources at 50% load.Overhead remained constant regardless of load or fault state.Source: §5.1 · p. 14
  • PEER_DOWN and WORKER_ISOLATED produced no false positives during the 18-hour baseline, and a full partition was identified within 10 seconds from the non-partitioned side.Softer WORKER_HOTSPOT and WORKER_SOURCE_HOTSPOT correlations still require operator judgment.Source: §5.1 · p. 14
  • Thresholds were tuned on preliminary baseline runs rather than dedicated validation splits, creating overfitting risk and likely requiring recalibration for different infrastructure.CUSUM’s online baseline learning partially mitigates this limitation.Source: §5.1 · p. 14

5.2 Limitations

  • In the virtualized environment, 2–5 ms RTT variance generates 2,900 anomalies per hour, while O(N^2) full-mesh probing bounds scalability beyond the evaluated 10-pod cluster.Bare-metal deployments would probably exhibit a lower noise floor; larger deployments may require probe sampling or hierarchical aggregation.Source: §5.2 · p. 14
  • The central correlator is a single point of failure, and severe bandwidth throttling can starve metric reports and reduce visibility when monitoring is most needed.Agents continue local detection if the correlator is lost, but cluster-wide correlation stops until it is rescheduled.Source: §5.2 · pp. 14–15
  • eBPF requires CAP_BPF and CAP_NET_ADMIN, which may conflict with hardened security policies; CNOM also limits each widget to 20 time-series lines.The detection engine is independent of the visualization layer and could use a more capable frontend.Source: §5.2 · p. 15

6 Conclusion and Future Work

On a live AMF deployment, NetMon detected subtle latency and packet-loss faults within seconds and localized them within tens of seconds. Future work proposes broader detection, adaptive thresholds, learning-based analysis, more scalable probing, and improved controller fault tolerance.

6.1 Conclusion

  • 10ms latency or 5% packet loss faults were detected within 2–10 seconds and localized within 6–20 seconds on a live AMF deployment.Full network partitions were detected and correlated within 9–11 seconds; high-confidence correlations had zero false positives across 18 hours, with overhead of 3.4 millicores CPU and 4.5 MiB per pod.Source: §6.1 · p. 15
  • The hybrid approach targets health-check, scrape-delay, and idle-path gaps to support early detection and fault localization for cloud-native 5G service quality.Source: §6.1 · p. 15

6.2 Future Work

  • Future work spans richer detection and adaptive thresholds, ML/AI-based classification and root-cause summaries, hierarchical probe aggregation, and agent-based controller fault tolerance.Source: §6.2 · p. 15
Loading 2609.12649v1…