Source-linked AI summary
D-Pingmesh: Dynamic Probing and Fault Localization for Reconfigurable Datacenter Networks
Yunze Wei, Kaicheng Yang, Xiaohui Xie, Dunkai Mao, Peiying Tao, Yikai Zhao, Bolong Cui, Chen Sun, Bingyang Liu, Kun Tan, Kai Chen, Yong Cui
TL;DR
Reconfigurable datacenter networks make uniform probing inefficient, and control-plane changes complicate fault localization at update boundaries. D-Pingmesh co-designs probing and localization with the TPE/TE control plane, achieving fewer tasks, faster pinglist generation, and improved localization in evaluation.
Problem
Prior monitoring faces low-hit interfaces, changing snapshot coverage, and observations near update boundaries that may reflect paths from either side.
Source: §2.2 · pp. 3–4, §2.3 · p. 5Method
Results
At a 0.99 coverage target, D-Pingmesh reduces task count by 17.7× over Pingmesh and generation time by an estimated 5.1× over RD-Probe; ns-3 F1 improves from 0.871 with deTector PLL to 0.953.
Source: §1 · p. 2Takeaways & Limitations
D-Pingmesh’s abstractions do not depend on OCS-specific properties and apply to networks whose forwarding paths change faster than the measurement window, including LEO satellite constellations.
Source: §8 · p. 13Transient faults may disappear before the next stable window, while a pinglist efficient for one snapshot may lose coverage after TPE or TE changes.
Source: §2.2 · pp. 3–4
Abstract
from arXiv · showhide
Pingmesh monitors datacenter networks through periodic pings between servers. In reconfigurable datacenter networks, topology engineering (TPE) and traffic engineering (TE) make uniform server-pair probing inefficient: a pinglist wastes budget retesting common paths while missing low-probability interfaces. Frequent control-plane updates further require rapid pinglist refresh, and cause path ambiguity at control-plane boundaries for fault localization. We present D-Pingmesh, a probing and fault-localization system co-designed with the TPE/TE control plane. Its control-plane-aware generator builds task-efficient pinglists by greedily admitting probes with high interface coverage under the current TPE/TE snapshot. An incremental replay-and-repair mechanism updates pinglists across reconfigurations rather than recomputing from scratch. At control-plane boundaries, a Rho-EM localizer jointly infers per-task old/new-path mixture weights and the fault set through EM-style iteration. We evaluate D-Pingmesh on an 8-pod reconfigurable topology under dynamic TPE/TE. At a 0.99 coverage target, D-Pingmesh uses 17.7x fewer tasks than Pingmesh and generates each pinglist about 5.1x faster than RD-Probe. In ns-3 simulation, boundary-aware localization improves micro F1 from 0.871 to 0.953 over the deTector PLL baseline. A testbed deployment demonstrates second-scale fault localization with low host overhead. These results demonstrate that D-Pingmesh is practical for reconfigurable datacenter networks with control-plane-aware monitoring.
1 Introduction
TPE and TE make stable, uniform server-pair probing inefficient and complicate diagnosis across reconfiguration boundaries. D-Pingmesh co-designs probe generation, pinglist updates, and fault localization with the control plane, improving probing efficiency and boundary-aware localization in evaluation.
Motivation
- At cluster scale, even small component failure rates make online network testing and accurate fault localization necessary.Reported average daily failure rates are 0.005% for interconnect cables and 0.04% for OCSes.Source: §1 · p. 1
Approach
- D-Pingmesh uses current TPE/TE snapshots to determine observable interfaces, greedily builds an interface multicover pinglist, and incrementally replays, repairs, and prunes it after reconfiguration.The incremental update preserves still-valid tasks while repairing coverage deficits without reducing achieved coverage.Source: §1 · p. 2
- Rho-EM jointly refines per-task old/new-path mixture weights and a fault set through EM-style iteration, covering complete link failures and gray packet-loss faults.It uses aggregate probe results around a TE transition and estimates per-interface loss rates.Source: §1 · p. 2
Evaluation
- 17.7× fewer tasks than Pingmesh at 0.99 coverage, an estimated 5.1× faster generation than RD-Probe, and F1 improvement from 0.871 with deTector PLL to 0.953.The evaluation also reports second-scale fault localization with low host overhead on a testbed.Source: §1 · p. 2
2 Background and Motivation
OCS-based networks combine optical inter-pod connectivity with TPE and TE decisions that shape forwarding paths. These changes challenge prior monitoring systems through skewed interface coverage, stale pinglists, and mixed-path observations at reconfiguration boundaries.
2.1 Reconfigurable Datacenter Networks
- TPE establishes optical circuits while TE splits traffic over the resulting connectivity, so the same server pair can traverse different interfaces or direct and detour paths.OCS cross-connects change inter-pod connectivity; TE uses weighted next-hop groups and WCMP forwarding.Source: §2.1 · p. 3
2.2 Limitations of Prior Monitoring Systems
- Prior monitoring systems are not designed around TPE/TE awareness, limiting efficient pinglist generation and updates and accurate fault localization in dynamic networks.The stated comparison dimensions include coverage under current TPE/TE state, update cost after reconfiguration, and diagnosis at boundaries.Source: §2.2 · pp. 3–4
2.3 Challenges
- Topology and forwarding asymmetry skew interface hit probabilities, causing uniform probing to repeat observations on likely paths while low-probability interfaces may miss their diagnosis coverage target.In the example, low-probability ToR and Agg interfaces reach only 82.4% and 54.8% coverage after 20 probes.Source: §2.3 · p. 4
- TPE/TE updates can invalidate a pinglist optimized for the prior snapshot, while rebuilding from scratch is expensive and reusing it can leave newly exposed interfaces uncovered.The stated challenge is to preserve valid tasks, repair deficits, and prune redundancy with low overhead.Source: §2.3 · p. 4
- At TE boundaries, probe-window aggregates can mix forwarding states; 15.25–28.29% of tasks change paths on average, and localization F1 falls from 95.3% to 88.9% versus steady state.Among 1,463 recorded link flaps, 96.4% recovered within 30 seconds, the example probe-window duration.Source: §2.3 · p. 5
3 System Overview
D-Pingmesh coordinates probing and fault localization with the TPE/TE control plane. It generates slot-specific pinglists, aligns probe windows with updates, and localizes faults from aggregate probe results.
3 System Overview
- The controller uses current TPE/TE state and upcoming update schedules to generate slot-specific pinglists and synchronize probe windows with reconfigurations.Ping agents execute the lists in synchronized windows and return aggregate attempt and failure counts; the system requires no switch modifications.Source: §3 · p. 5
- Hash-aware path resolution reconstructs each probe’s expected path from its concrete five-tuple and the current control-plane snapshot.The generator uses resolved paths to assess coverage, while the localizer uses them to associate outcomes with candidate faulty interfaces.Source: §3 · p. 5
- The generator selects tasks for interface coverage, and incremental update re-resolves the prior list, repairs coverage deficits, and removes redundancy.The controller distributes the updated pinglist to agents and synchronizes probe windows with TPE/TE updates.Source: §3 · p. 5
- Rho-EM resolves both pre- and post-update path hypotheses and jointly refines the fault set and per-task path mixtures using aggregate probe counts.It returns a predicted fault set and ranked suspicion list.Source: §3 · pp. 5–6
4 Probe Generation and Update with Control-Plane Awareness
D-Pingmesh resolves probe paths against the current TPE/TE snapshot and uses them to generate and incrementally update interface-covering pinglists. Its design combines hash-aware path resolution, multicover formulation, probability-guided greedy selection, and replay-and-repair across reconfigurations.
4.1 Hash-Aware Path Resolution
- ResolvePath maps a concrete probe five-tuple through the current topology and TPE/TE state to reconstruct its expected path of interfaces.The path calculation depends on supplied control-plane state and switch hash behavior matching active forwarding; otherwise coverage accounting or fault attribution may be wrong.Source: §4.1 · p. 6
4.2 Probe Task Generation Problem
- The task-generation objective is a minimum-cardinality pinglist that covers every reachable target interface at least k times under the current snapshot.Exact minimization is NP-hard, and full candidate enumeration is impractical at datacenter scale, motivating a probability-guided greedy approximation.Source: §4.2 · p. 6
4.3 Asymmetry-Aware Probe Generation
- The generator estimates interface hit probabilities from pod-pair paths and forwarding weights, then prioritizes pairs by expected contribution to interfaces below the k-coverage target.It samples up to B candidate tasks for a selected pair and accepts the one with the largest marginal gain; B=3 is the default because larger budgets provide modest additional task reduction at higher computation cost.Source: §4.3 · pp. 6–7
4.4 Dynamic Incremental Probe Update
- D-Pingmesh re-resolves prior tasks under each new snapshot, repairs remaining coverage deficits, and prunes redundant tasks without reducing achieved coverage.This incremental update reuses useful tasks instead of discarding the pinglist and regenerating from scratch.Source: §4 · p. 6, §4.4 · pp. 7–8
- Incremental update has re-resolution, repair, and pruning phases: retain tasks that still cover target interfaces, fill residual deficits, then remove redundancy while preserving coverage.If greedy pruning cannot meet all capped coverage targets, D-Pingmesh keeps the unpruned merged set, so pruning does not reduce coverage achieved by re-resolution and repair.Source: §4.4 · pp. 7–8
5 Fault Localization Across Reconfiguration Boundaries
TPE/TE updates can make a probe task’s path ambiguous at a transition, complicating fault localization from aggregate observations. Rho-EM models old- and new-path hypotheses and jointly refines per-task mixture weights and candidate faults through EM-style iteration.
5 Fault Localization Across Reconfiguration Boundaries
- At update boundaries, aggregate observations may combine attempts over pre- and post-update paths because TE updates deploy at pod granularity rather than atomically.Rho-EM treats the per-attempt path as hidden and returns a predicted fault set with ranked suspicion scores.Source: §5 · p. 8
5.1 Observation Model
- Each changed task has a mixture weight for the probability that an attempt followed the complete post-update path, and aggregate counts summarize attempts over a probe window.The model excludes hybrid paths in which different hops use old and new forwarding states.Source: §5.1 · p. 8
- The likelihood considers candidate interfaces traversed under either snapshot and estimates per-interface loss rates, with independent faulty interfaces contributing to path failure probability.This supports paths that traverse multiple faults.Source: §5.1 · pp. 8–9
5.2 Rho-EM Fault Localization Algorithm
- Rho-EM alternates greedy fault selection by marginal log-likelihood gain with updates to changed tasks’ mixture weights under the selected fault set.The mixture-weight update uses a beta prior to prevent degenerate estimates when evidence is weak, and is damped for numerical stability.Source: §5.2 · p. 9
- Iterations stop when the maximum mixture-weight change falls below a threshold or the iteration cap is reached; a zero cap gives the fixed-mixture fallback.Source: §5.2 · p. 9
6 Evaluation
The evaluation tests probe generation, fault localization, and testbed practicality under dynamic TPE/TE, using an 8-pod topology and production-derived traffic traces. D-Pingmesh reduces probing and generation costs, improves boundary localization, and runs on a physical testbed with low host overhead.
6.1 Experimental Setup
- The main experiments use an 8-pod, 2,048-server OCS topology and 96 fifteen-minute TE windows driven by Gemini-optimized snapshots from production-derived traffic traces.The setup compares five generators targeting k=4 coverage per interface per slot, including D-Pingmesh’s incremental and Cold Start variants.Source: §6.1 · pp. 9–10
6.2 Main Generator Comparison
- In the controlled F20 same-coverage experiment, with each generator reaching at least 0.99 end-to-end coverage, ratios normalized to D-Pingmesh show that Pingmesh required 17.7× more tasks.Source: §6.2 · p. 10, Figure 8 · p. 10
- Incremental update is 2.37× faster than Cold Start, while remaining 1.32× faster even when 90% of TE groups change weight.Coverage remains near 100% across the controlled TE-change settings.Source: Figure 9 · p. 11, §6.2 · p. 11
- Across 4–64 pods, D-Pingmesh exceeds 99% coverage at every measured scale, reaching 79.9K tasks and 43.78 s/slot at 64 pods.At 64 pods, Pingmesh expands to 30.5M tasks/slot and deTector* takes 42,510 s/slot.Source: Figure 10 · p. 11, §6.2 · p. 11
6.3 Fault Localization
- Across 120 cases, boundary-aware Rho-EM achieves 0.953 micro F1, versus 0.871 for deTector PLL and 0.907 for New Path.The advantage is more pronounced at low loss rates, where weak observations amplify boundary ambiguity.Source: Figure 11 · p. 12, Figure 12 · p. 12, §6.3 · p. 12
6.4 Physical Testbed Validation
- On the four-pod testbed, D-Pingmesh localizes persistent loss and transient link flaps at the transition-window boundary, with results available within 2 seconds.Average CPU utilization is 0.43%, and average per-server TX+RX traffic is 0.10 Mbps.Source: Figure 13 · p. 12, §6.4 · p. 12
7 Related Work
Prior work covers datacenter probing, diagnosis, runtime forwarding verification, traffic engineering, and reconfigurable fabrics. D-Pingmesh differs by targeting path changes across TPE/TE reconfigurations.
7 Related Work
- Earlier probing and diagnosis systems largely assume stable forwarding paths, whereas D-Pingmesh targets path changes across TPE/TE reconfigurations.Related work also describes runtime forwarding verification as complementary, and uses TE output as forwarding state for probe generation and localization.Source: §7 · p. 13
8 Discussion
The discussion describes D-Pingmesh’s abstractions as applicable beyond OCS-based datacenter fabrics, while identifying limits in its fault model and evaluation scope.
8 Discussion
- The approach operates over topology, TE weights, and WCMP hash behavior rather than OCS-specific properties, and is described as applicable to reconfiguring LEO networks.The stated scope is networks whose forwarding paths change faster than the measurement window.Source: §8 · p. 13
- D-Pingmesh’s likelihood model assumes independent per-interface loss with a stable rate within each probe window, excluding shared-risk failures and time-varying loss rates.The evaluation uses 15-minute TE windows; handling observations during optical switching requires further evaluation.Source: §8 · p. 13
9 Conclusion
D-Pingmesh combines control-plane-aware probe generation, incremental pinglist updates, and Rho-EM localization for reconfigurable datacenter networks.
9 Conclusion
- The paper concludes that D-Pingmesh achieves equivalent coverage with fewer tasks and lower generation time while improving localization under continuous reconfiguration.Rho-EM jointly refines per-task old/new-path mixtures and the fault set to address boundary ambiguity.Source: §9 · p. 13