Source-linked AI summary
Characterizing the eBPF-Based Data Plane for Multi-ClusterKubernetes: A Systematic Evaluation of Cilium Cluster Meshand KVStoreMesh
Simhadri Podala Narasimha
TL;DR
The paper addresses the lack of reproducible evaluation of Cilium’s multi-cluster data plane and proposes a testbed and measurement methodology across five research questions. Without live cluster access, it uses a calibrated simulation to generate falsifiable predictions, not empirical findings.
Problem
The multi-cluster eBPF data plane lacks independently reproducible benchmarks, including evidence about latency and jitter for GPU/RDMA-sensitive cross-cluster traffic.
Source: §1 · p. 2Method
The paper proposes a multi-regime Kubernetes testbed and paired architecture experiments, and reports an analytical/discrete-event simulation while live testbed access is unavailable.
Source: §5.1 · p. 4, §5.3 · p. 5, §6 · p. 5Results
The calibrated model predicts KVStoreMesh will retain lower propagation delay as scale and churn increase, while cross-cluster forwarding will have higher mean latency and jitter than intra-cluster forwarding.
Source: §6.5 · p. 8Takeaways & Limitations
The paper’s primary contribution is a testbed, instrumentation, and experimental design for future empirical evaluation; its simulation predictions require real-infrastructure validation.
Source: §8 · p. 9No live Kubernetes/Cilium measurements were available; the KVStoreMesh 6× capacity multiplier is assumed, and the RQ5 latency model is illustrative.
Source: §7 · p. 8
Abstract
from arXiv · showhide
Kubernetes deployments increasingly span multiple clusters for reasons of scale, fault isolation, regulatory boundaries, and heterogeneous hardware placement, including GPU-dense clusters dedicated to AI and HPC workloads. Cilium Cluster Mesh, built on extended Berkeley Packet Filter (eBPF) technology, has emerged as a widely deployed mechanism for connecting such clusters into a single logical network without a dedicated multi-cluster gateway. Despite its adoption, the academic literature contains no systematic, reproducible evaluation of the eBPF-based multi-cluster data plane itself: existing work either benchmarks eBPF against iptables within a single cluster, or applies eBPF to cross-cluster monitoring rather than forwarding and identity propagation. The most detailed scale data available - a report of a clustermesh-apiserver failing under load at approximately 45,000 nodes across 256 clusters - comes from an industry engineering blog rather than a peer-reviewed source. This paper proposes a methodology to close that gap. We define five research questions covering latency, throughput, control-plane limits, churn, and GPU/RDMA sensitivity, and describe a testbed and eBPF-level instrumentation design. Lacking a live multi-cluster testbed for this submission, we report results from a calibrated analytical simulation of control-plane architectures fit to the production data point. These results are presented as falsifiable predictions intended for validation once the testbed is complete. This paper provides the motivation, experimental design, and preliminary simulation results to serve as a template for a full empirical study.
1 Introduction
Kubernetes federation needs to connect clusters while accommodating regulatory, availability, and hardware-placement constraints. The paper presents Cluster Mesh and motivates reproducible evaluation of its data plane and evolving control plane.
1 Introduction
- Cluster Mesh synchronizes service, endpoint, and identity state across clusters, enabling pod connectivity, global service discovery, and consistent network policies without a dedicated gateway or application-code changes.Its purpose is to make multiple clusters behave like one connected platform from an application’s perspective.Source: §1 · pp. 1–2
- KVStoreMesh replaces remote watches on raw etcd stores with local caching and watches between KVStoreMesh instances, aiming to reduce failure domains and smooth load distribution.The earlier clustermesh-apiserver design was reported to fail near tens of thousands of nodes across hundreds of clusters.Source: §1 · p. 2
- Practitioners lack independently reproducible benchmarks for choosing among multi-cluster networking options, while GPU workloads raise unanswered questions about cross-cluster latency and jitter.Existing evidence is largely vendor-reported, and generic microservice benchmarks do not address RDMA-sensitive traffic such as gradient synchronization or model-parallel activations.Source: §1 · p. 2
- The paper proceeds from related work and architecture through research questions and methodology to simulation, validity threats, and conclusions.The simulation is explicitly preliminary and substitutes for unavailable live testbed access.Source: §1 · p. 2
2 Related Work
Prior work covers single-cluster eBPF performance, cross-cluster monitoring, and multi-cluster comparisons, but does not directly evaluate the multi-cluster eBPF data plane.
2.1 Single-Cluster eBPF Performance Studies
- A single-cluster comparison found iptables had higher throughput and lower latency than eBPF under unloaded conditions, challenging assumptions that eBPF is always faster.The study measured throughput, latency, CPU, and memory under loaded and unloaded conditions.Source: §2.1 · p. 2
2.2 Cross-Cluster Monitoring and Security via eBPF
- FedMon applies eBPF to cross-cluster monitoring and security, reporting 94% precision, 91% recall, F1 0.92, and 60% lower bandwidth than centralized baselines across three clusters [7].It shares model updates rather than raw telemetry, addressing observability rather than cross-cluster data-plane forwarding.Source: §2.2 · p. 2
2.3 Comparative Evaluations of Multi-Cluster Networking
- An earlier thesis compared multi-cluster networking implementations but predates KVStoreMesh and current global services; ONCache demonstrates targeted single-cluster eBPF instrumentation in 524 lines [6].ONCache eliminates redundant container-overlay-network overhead, illustrating lightweight instrumentation relevant to the proposed methodology.Source: §2.3 · p. 3
2.4 Industry and Community Documentation
- The most detailed public scale report describes clustermesh-apiserver failing near 45,000 nodes across 256 clusters because etcd could not keep up with watch load and lease handling.KVStoreMesh’s local caching design is documented in Cilium and community materials, but neither architecture has independent controlled academic measurement.Source: §2.4 · p. 3
3 Background: Cluster Mesh and KVStoreMesh Architecture
Cluster Mesh connects Cilium agents across member clusters to discover and reach remote services and pods. KVStoreMesh changes how state is watched and propagated, while global services provide cross-cluster service-level load balancing.
3 Background: Cluster Mesh and KVStoreMesh Architecture
- In clustermesh-apiserver, remote agents watch each cluster’s etcd directly, so watch counts scale with the product of cluster count and node count.The paper links this pattern to the reported failure near 45,000 nodes across 256 clusters.Source: §3 · p. 3
- KVStoreMesh caches remote state locally and limits local agents to watching their own cluster’s instance, making watcher counts proportional to local nodes.The design reduces direct watchers receiving remote churn and smooths load spikes, at the cost of an additional caching hop.Source: §3 · p. 3
- Global services synchronize endpoints for annotated Kubernetes Services across connected clusters, enabling cross-cluster load balancing without an external load balancer or service-mesh sidecar.Source: §3 · p. 3
4 Research Questions
The paper frames five questions about forwarding performance, identity propagation, scaling limits, churn robustness, and sensitivity of GPU/RDMA traffic. They define the measurements needed to compare architectures and cross-cluster with intra-cluster communication.
4 Research Questions
- RQ1 asks how cross-cluster eBPF forwarding changes latency and throughput relative to intra-cluster traffic, and how costs scale with cluster and node counts.Source: §4 · p. 4
- RQ2 asks how quickly new pod identity and endpoint information becomes globally routable under steady-state and high-churn conditions, comparing both architectures.Source: §4 · p. 4
- RQ3 asks which cluster-count, node-count, and churn combinations cause degradation or failure, including whether controlled tests reproduce the reported failure near 45,000 nodes across 256 clusters.Source: §4 · p. 4
- RQ4 tests whether KVStoreMesh’s claimed load smoothing holds under adversarial churn, including simultaneous mass pod restarts or coordinated rolling upgrades.Source: §4 · p. 4
- RQ5 asks whether cross-cluster forwarding adds latency jitter significant to RDMA-sensitive GPU workloads such as distributed-training gradient synchronization, compared with intra-cluster GPU communication.Source: §4 · p. 4
5 Methodology
The proposed methodology combines a scalable multi-cluster testbed with eBPF-level and control-plane instrumentation. Paired architecture comparisons, churn and scale sweeps, GPU-collective tests, and planned baselines address the five research questions.
5.1 Testbed Design
- The testbed scales from 2–4 clusters to a possible 100+ clusters, with medium-regime sweeps used for extrapolation if full-scale reproduction is infeasible.The small regime targets latency and correctness; the medium regime targets early degradation trends.Source: §5.1 · p. 4
5.2 Instrumentation
- Hubble and lightweight eBPF probes measure forwarding, while Kubernetes timestamps track remote identity propagation and Prometheus metrics capture control-plane resource use.The plan also measures endpoint propagation and readiness and synchronization indicators.Source: §5.2 · p. 4
5.3 Experimental Conditions
- Paired architecture tests hold cluster and node counts fixed; separate sweeps test scale, adversarial churn and GPU collective tail latency and jitter.Scale tests record threshold crossings, while churn tests compare load distribution and recovery time.Source: §5.3 · p. 5
5.4 Baselines
- Where feasible, Istio multi-cluster and Liqo provide cross-approach baselines, while Calico eBPF mode provides a same-technology comparison.The planned comparisons situate Cluster Mesh against sidecar-based and virtual-node-based alternatives.Source: §5.4 · p. 5
6 Simulation-Based Evaluation
Because no live multi-cluster testbed was available, the paper uses a calibrated analytical simulation to generate falsifiable predictions, not empirical findings. It predicts lower propagation delay for KVStoreMesh as scale and churn rise, and higher mean latency and jitter for cross-cluster forwarding; real-infrastructure tests are needed to validate these patterns.
6 Simulation-Based Evaluation
- Without a live multi-cluster testbed, the reported quantitative results are model-derived hypotheses for confirmation or falsification on real infrastructure.The simulation assumptions, calibration point, code, parameters, and output data are disclosed for inspection and re-execution.Source: §6 · p. 5
6.1 Simulation Methodology and Disclosure
- The simulation models watch fan-out and uses an M/M/1-style queueing approximation; its assumed 6× KVStoreMesh capacity multiplier is the largest uncertainty in the comparative magnitude.The apiserver capacity is calibrated to one reported failure near 45,000 nodes across 256 clusters; the RQ5 distributions are synthetic and qualitatively parameterized.Source: §6.1 · pp. 5–6
6.2 RQ1/RQ3: Simulated Scaling Behavior
- Removing the multiplicative clusters × nodes scaling term is predicted to raise the practical ceiling, while the specific crossover and delay magnitudes remain model artifacts.Table 1 reports simulated watch counts and propagation delay by mesh scale; Figure 1 marks the calibration point on a logarithmic delay axis.Source: §6.2 · p. 6, Table 1 · p. 6, Figure 1 · p. 7
6.3 RQ2/RQ4: Simulated Behavior Under Churn
- At 32 clusters and 60 nodes per cluster, the clustermesh-apiserver model destabilizes above roughly 25× steady-state churn, while the KVStoreMesh model remains flat across the sweep.The models cannot predict actual churn multipliers or recovery times; real churn-injection tests are needed.Source: §6.3 · p. 6, Figure 2 · p. 7
6.4 RQ5: Illustrative RDMA/GPU Jitter Analysis
- In the synthetic model, cross-cluster mean latency is 14.8 µs versus 8.0 µs intra-cluster, and 99th-percentile latency is 27.6 µs versus 10.8 µs.The modeled jitter (standard deviation) is 3.9 µs cross-cluster versus 1.2 µs intra-cluster; these are not GPU- or RDMA-traffic measurements.Source: §6.4 · pp. 6, 8, Figure 3 · p. 8
6.5 Summary and Status
- The simulation predicts lower propagation delay for KVStoreMesh as scale and churn increase, and higher mean latency and jitter for cross-cluster forwarding, but these are hypotheses, not findings.The cross-cluster effect would matter for RDMA-sensitive GPU traffic if its effect size holds at measured scale.Source: §6.5 · p. 8
7 Threats to Validity
The simulated results are not live measurements, and several assumptions, calibration limits, scale constraints, environment differences, version changes, and instrumentation overhead bound their interpretation.
7 Threats to Validity
- The results are from calibrated simulation rather than live Kubernetes/Cilium clusters; KVStoreMesh’s 6× capacity multiplier is assumed, and the RQ5 latency model is illustrative.These are the central scope limits on interpreting the reported results.Source: §7 · p. 8
- The clustermesh-apiserver capacity fit uses one publicly reported failure observation at one cluster/node ratio and has no second independent validation point.The observation is reported in.Source: §7 · p. 8
- Reproducing the 256-cluster/50,000-node regime may require significant compute resources; smaller-scale testbed results must be framed as extrapolations, not production-scale confirmation.Source: §7 · pp. 8–9
- Shared-infrastructure kind/Talos clusters may not capture production network topology, inter-region latency, or physical NIC and kernel variation, especially for RQ5 jitter.Source: §7 · p. 9
- Because Cluster Mesh and KVStoreMesh are actively developing, results must identify specific Cilium versions and be revalidated after material version changes.Source: §7 · p. 9
- Future custom eBPF probes must be shown to add negligible overhead relative to the measured phenomena, following ONCache’s low-instruction-count precedent [6].Source: §7 · p. 9
8 Conclusion and Future Work
The paper offers a measurement methodology and calibrated simulation predictions because no live testbed was available. Those predictions are explicitly provisional, with empirical validation planned as future work.
8 Conclusion and Future Work
- The paper’s primary contribution is a testbed, instrumentation, and experimental design intended to address the lack of peer-reviewed scale data on widely deployed Cluster Mesh and KVStoreMesh.Source: §8 · p. 9
- Without a live testbed, the calibrated simulation reproduces one reported failure point and provides falsifiable quantitative predictions for all five research questions, not a substitute for measurement.Source: §8 · p. 9
- Future work will begin with small- and medium-scale testbed regimes to confirm, revise, or falsify the predictions and report empirical results in a follow-on submission.Source: §8 · p. 9