Exploiting the Interplay of Compute- and Memory-Bound kernels in MPI Applications

I. INTRODUCTION AND RELATED WORK

• Prior desynchronization studies largely focus on communication-heavy workloads, leaving kernel interaction in communication-light heterogeneous applications comparatively unexamined.The paper frames this as a missing case for applications with little inter-process communication.Source: §I · pp. 1–2
• The study examines the Parallel Optical Flow Solver and a tunable Pisolver-Triad benchmark, each pairing compute-bound and memory-bound kernels with negligible inter-process communication.The application pairs Ray Tracing with Optical Flow; the benchmark pairs Pi Solver with STREAM Triad.Source: §I · p. 2
• The authors quantify natural desynchronization against forced lock-step execution and identify workload fraction and problem size as factors for tuning its performance impact.The contribution list specifies memory-to-compute fraction and L3 bandwidth utilization as tuning targets.Source: §I · p. 2
• Removing MPI communication stalls through asynchronous progress can degrade performance by eliminating incidental relief from memory-bandwidth contention.The paper highlights this as a counterexample to treating every communication stall as a hazard to remove.Source: §I · pp. 1–2
• The work validates real-hardware traces against DisCostiC simulated traces to characterize where and why idealized simulation diverges from hardware behavior.The paper also provides raw results, runtime logs, scripts, inputs, and environment documentation for reproduction.Source: §I · p. 2
• Performance is maximized when concurrent memory-bound ranks remain near the bandwidth-saturation threshold, whether through workload balance, incidental stalls, or deliberate noise injection.The threshold is presented as the design rule for sustaining desynchronization.Source: §I · p. 2

II. METHODOLOGY AND APPLICATIONS

• The methodology derives a theoretical upper bound on desynchronization speedup and describes the experiments used to isolate kernel-driven effects.The section covers the theoretical bound, experimental applications, and methodology for separating kernel-driven desynchronization from other performance effects.Source: §II · p. 2

A. Theoretical Upper Bound on Desynchronized Speedup

• With negligible communication, each iteration combines compute-bound and memory-bound time; overlap reduces effective memory-bound time, and ideal iteration time is minimized when it matches compute-bound time.The memory-to-compute fraction ρ is defined from tMB and tCB as ρ = tMB/(tMB + tCB).Source: A. Theoretical Upper Bound on… · p. 2
• Bandwidth saturation limits the reduction in memory-bound time: within the saturation region, aggregate bandwidth is roughly constant and the memory-time floor occurs at Csat active processes.The saturation-limited floor changes the ideal bound at ρ* = N/(N + Csat).Source: A. Theoretical Upper Bound on… · p. 2
• For the platform with N = 18 and Csat = 10, the threshold is ρ*≈64%; the measured best case, ρ = 55%, lies below it and matches the predicted peak region.The supplied passage says the theoretical limit is consistent with speedup peaking near the best-case region and declining toward compute- and memory-heavy extremes.Source: A. Theoretical Upper Bound on… · p. 2

B. Parallel Optical Flow Solver

• Each MPI process alternates between compute-bound Ray Tracing and memory-bound Optical Flow kernels, with minimal communication between processes.The application synthesizes images by ray tracing, then computes their optical flow fields.Source: B. Parallel Optical Flow Solver · p. 3
• The multigrid level sets the Optical Flow image size, while supersampling scales the Ray Tracing input and tunes its compute-bound workload.Sweeping these parameters moves the execution profile from compute-bound to memory-bound.Source: B. Parallel Optical Flow Solver · p. 3
• Ranks start on evenly spaced frames, but the hourglass-shaped sphere distribution creates frame-to-frame workload differences that can cause secondary load imbalance.The study separates this effect from kernel-driven desynchronization.Source: B. Parallel Optical Flow Solver · p. 3
• Table I lists the tuned Pisolver-Triad benchmark parameters for single-node runs.Source: Table I · p. 3
• Setting sinPrefactor = 0 and dt = 10−5 removes spatial imbalance and yields near-identical per-process frame data, isolating later performance changes as kernel-driven.The authors report lock-step startup after mitigating the initial imbalance.Source: B. Parallel Optical Flow Solver · p. 3

C. Pisolver-Triad Benchmark

• Pisolver-Triad pairs a compute-bound π-calculation kernel with a memory-bound STREAM Triad kernel and lightweight periodic communication.Source: C. Pisolver-Triad Benchmark · p. 3
• Each iteration interleaves a Pisolver step, a non-blocking send, three further Pisolver–Triad sub-steps, a blocking receive, and a final Triad step.Source: C. Pisolver-Triad Benchmark · p. 3
• Kernel weights, array sizes, and artificial noise can be injected at specified ranks, timesteps, and ccNUMA domains, unlike in the full application.The benchmark’s communication time is matched to the application’s best-case configuration.Source: C. Pisolver-Triad Benchmark · p. 3

III. EXPERIMENTAL SETUP

• The section describes the hardware environment used for the evaluation.Source: §III · p. 3
• The section describes the software environment used for the evaluation.Source: §III · p. 3
• The section presents the measurement methodology and simulation framework used throughout the evaluation.Source: §III · p. 3

A. Testbed and Experimental Methodology

• Fritz runs use five nodes and 360 MPI processes for multi-node experiments, while Helma runs use one node with 384 MPI processes.Fritz multi-node experiments use a fixed 2.4 GHz clock with Hyper-Threading disabled.Source: A. Testbed and Experimental Methodology · pp. 3–4
• The application, benchmark, and simulator are built with Intel oneAPI and Intel MPI, using different compiler flags for Cluster A and Cluster B.Execution traces use ITAC and Vampir; bandwidth and floating-point rates use LIKWID’s likwid-perfctr.Source: A. Testbed and Experimental Methodology · p. 4
• Kernel timings use std::chrono after warm-up iterations, and benchmark experiments run for 10,000 iterations.Variability is characterized over 100 repetitions, and reported speedups subtract measured barrier overhead.Source: A. Testbed and Experimental Methodology · p. 4
• Process timeline plots sample memory bandwidth and floating-point activity on all node cores every 200 ms.Source: A. Testbed and Experimental Methodology · p. 4

B. DisCostiC Simulation Framework

• DisCostiC combines ECM-based per-kernel runtime estimates with a model of concurrent memory-bound ranks to simulate contention-driven progress.It represents the MPI program’s kernel and communication sequence without re-executing the application code.Source: B. DisCostiC Simulation Framework · p. 4

IV. RESULTS: OPTICAL FLOW SOLVER

• OF bandwidth saturates near 80 GB/s beyond seven active cores per ccNUMA domain, while RT performance scales linearly with cores.As desynchronization reduces concurrent OF execution, per-process performance rises and stabilizes.Source: §IV · p. 4, Figure 1 · p. 5
• ≈46% peak normalized desynchronization speedup occurs near ρ = 55% across the swept solver configurations.The sweep varies multigrid level and supersampling factor.Source: Figure 2 · p. 5
• 278.5 GF/s versus 221.7 GF/s for forced-synchronous execution, a ≈25.5% improvement with natural desynchronization.The comparison uses mean performance over time and the forced-synchronous run as baseline.Source: §IV · p. 5

A. Effect of the memory-to-compute fraction

• ≈46% peak speedup occurs at the best-case ρ = 55%, where processes stagger execution and use memory bandwidth more efficiently.The best-case timeline has the widest per-process spread and largest net gain among the compared configurations.Source: A. Effect of the memory-to-compute… · pp. 5–6, Figure 3 · p. 6
• In the memory-heavy case, midpoint activity reaches 16–18 active cores per ccNUMA domain, more than twice the 7-core saturation threshold, causing sustained over-contention; counts fall only in the final iterations.Source: A. Effect of the memory-to-compute… · p. 6
• Imperfect process staggering, boundary-rank effects, and residual serialization account for the gap below the theoretical ceiling.These factors qualify the best-case result.Source: A. Effect of the memory-to-compute… · p. 6

B. The Cost of Removing Communication Stalls

• Enabling asynchronous progress reduces longer-run performance by nearly 80% by removing MPI stalls and allowing too many ranks to contend for memory bandwidth.With stalls present, only a subset of processes execute OF at a time and benefit from higher per-core bandwidth.Source: B. The Cost of Removing Communication… · p. 6, Figure 4 · p. 6

C. Cross-Architecture Validation on AMD CPU

• 88% performance improvement over forced lock-step execution occurs on Helma at ρ = 55%, with the same qualitative bandwidth-pressure mechanism observed on Intel.The comparison uses an AMD EPYC node; the absolute speedup differs with core count, NUMA organization, and bandwidth characteristics.Source: C. Cross-Architecture Validation on… · pp. 6–7, Figure 5 · p. 7

V. RESULTS: PISOLVER-TRIAD BENCHMARK

• The Pisolver-Triad benchmark peaks at ≈5.7% speedup near ρ ≈55%, much less than the flow solver’s 46% gain.The benchmark shows at most 5.7% spontaneous desynchronization benefit across its tested configurations.Source: §V · p. 7
• At least one ccNUMA domain remains synchronized throughout each benchmark configuration, limiting the throughput benefit despite workload-balance tuning.Which domain remains synchronized varies across runs.Source: §V · p. 7

A. Periodic, Spatially Distributed Noise Injection

• Per-ccNUMA-domain perturbations are needed for full spatial desynchronization; injection every fourth and second timestep yields 20% and 25% gains, respectively.A single localized delay is insufficient, while coarser group- or node-level noise fails to remove persistent domain synchrony.Source: A. Periodic, Spatially Distributed… · p. 7, Figure 8 · p. 8

B. Simulation Validation with DisCostiC

• DisCostiC reproduces the unperturbed benchmark's key execution pattern and message exchanges, predicting runtime with 0.5% error.Source: B. Simulation Validation with DisCostiC · pp. 7–8, Figure 9 · p. 8

C. Best Case: Rapid Desynchronization When Data Fits in L3

• ≈70% speedup occurs when the Triad working set fits shared L3 (6 × 10^6 elements), falling to ≈5% as it grows into main memory.L2-resident data yields little benefit; main-memory contention brings back a persistently synchronized ccNUMA domain.Source: C. Best Case: Rapid Desynchronization… · p. 8, Figure 10 · p. 9

• The Parallel Optical Flow Solver gains 25.5% over a forced-synchronous baseline, rising to 46% at a near-balanced 55% memory-to-compute fraction.Source: §VI · p. 9
• MPI asynchronous progress reduces performance by 80% because removing communication stalls permits excessive simultaneous memory-bandwidth contention.Source: §VI · p. 9
• Pisolver-Triad needs periodic ccNUMA-targeted idle-wave injection for up to 25% speedup in main memory, but naturally desynchronizes with substantially higher speedup in shared L3.Source: §VI · p. 9
• DisCostiC reproduces qualitative trends for the benchmark and full application, while larger workloads, arbitrary multi-kernel support, and improved trace fidelity remain future work.Source: §VI · p. 9
• Keep the number of ranks concurrently executing memory-bound phases near the platform's bandwidth-saturation threshold.The paper proposes testing a mechanism that enforces this condition against accidental and injected desynchronization.Source: §VI · p. 9