Source-linked AI summary
Energy-Aware Performance Characterization of Kokkos Kernels Across Precision Levels
Maria Patrou, Andrey Prokopenko, Aditya Kashi, Oscar Hernandez, Jean-Luc Fattebert
TL;DR
Kernel performance models traditionally omit the energy required to complete GPU work, despite energy being a major HPC concern. This paper combines energy-aware visualization models with Kokkos/PAPI measurements of applications and kernels. Precision effects vary across kernels and workloads, with FP64 generally consuming more energy and FP32 showing the most favorable measured performance–energy trade-off.
Problem
Conventional Roofline data relate throughput to arithmetic intensity but do not quantify kernel energy, although HPC systems face strict power budgets.
Source: §1 · p. 1Method
The paper presents energy-aware Roofline and thermal inter-kernel visualizations and combines application-level measurements with dedicated repeated-kernel benchmarking for short regions.
Source: §1 · p. 1, §4.1 · p. 5Results
Precision effects vary by kernel and workload; FP64 generally has lower arithmetic intensity and throughput while consuming more energy, and FP32 has the most favorable measured performance–energy trade-off.
Source: §5.5 · p. 10Takeaways & Limitations
The ArborX DBSCAN::clusters::query region's dominant measured energy share establishes an optimization priority independently of temperature.
Source: §6 · p. 11Future work must extend application coverage and test control-point value under controlled thermal conditions, measuring intervention effects on performance, GPU energy, and temperature.
Source: §6 · p. 11
Abstract
from arXiv · showhide
Kokkos is a performance-portable programming model used in GPU-accelerated HPC applications comprising kernels with diverse computational and memory behaviors. We characterize kernels from two applications to evaluate how FP32, FP64, and mixed FP32/FP64 affect arithmetic intensity, performance throughput, and GPU energy consumption. We introduce an Energy-Aware Roofline Model that incorporates energy into performance analysis and a Thermal-Aware Inter-Kernel Energy Model that visualizes energy consumption alongside duration, GPU temperature, and execution order. We study both applications with: (1) fine-grained, application-level instrumentation through the Kokkos Performance Tools PAPI Connector and (2) coarse-grained PAPI instrumentation applied to a dedicated-kernel benchmark using vendor-specific power management components. Our analysis shows that precision effects vary by application, kernel, and workload. Mixed FP32/FP64 kernels generally achieve arithmetic intensity comparable to FP32, with modestly lower throughput and, in most cases, a modest increase in energy consumption. FP64 kernels generally exhibit the highest energy consumption among the evaluated precision configurations.
1 Introduction
HPC developers must weigh energy alongside runtime, but conventional Roofline data show performance limits without quantifying kernel energy. The paper introduces two GPU visualization models and studies precision, workload, and instrumentation across Kokkos applications.
1 Introduction
- Energy efficiency is a first-order HPC concern because modern supercomputers face strict power budgets, including a cited 30 MW cap for exascale systems.The Green500 ranks supercomputers by FLOPS per Watt.Source: §1 · p. 1
- Conventional Roofline analysis relates throughput to arithmetic intensity and hardware ceilings, but does not quantify the energy required to complete a kernel.Developers need to compare throughput with completion time and energy and locate costly regions in application execution.Source: §1 · p. 1
- The Energy-Aware Roofline Model adds measured GPU energy to arithmetic intensity and achieved throughput, while the Thermal-Aware Inter-Kernel Energy Model relates region energy and duration to temperature and execution order.The models address kernel behavior and the execution context of costly application regions.Source: §1 · p. 1
- The study analyzes LOBPCG and ArborX kernels across FP32 and FP64 data and operations to examine precision effects on time-to-solution, energy, power, and temperature.The authors note interest in low- and mixed-precision algorithms, and identify LOBPCG-type eigensolvers as important to Density Functional Theory workloads.Source: §1 · p. 2
- The paper compares region-level Kokkos-tools/PAPI instrumentation in ArborX with a dedicated-kernel benchmark for LOBPCG, discussing advantages and limitations of both.The contributions address research questions about energy-aware Roofline analysis, region energy and temperature, and precision- and workload-dependent kernel behavior.Source: §1 · p. 2
2 Background
The background establishes the Roofline model and the monitoring interfaces used to characterize Kokkos GPU kernels. It covers performance ceilings, instrumentation trade-offs, PAPI/NVML counters, and Kokkos profiling support.
2.1 Roofline Model
- Roofline analysis plots kernel throughput against arithmetic intensity and hardware limits; the roofline is min{mα, f}, where m is peak memory bandwidth, f is peak flop throughput, and α is arithmetic intensity.Moving toward the roofline and increasing arithmetic intensity are traditional optimization aims for fixed problem size.Source: §2.1 · p. 2
2.2 Performance Monitoring Methodologies
- Runtime monitoring combines hardware-assisted counters with software instrumentation, which delimits code regions and trades runtime overhead for portable, region-level attribution.Hardware-assisted approaches can be costly, platform-specific, or limited to summarized data; software approaches range from emulation to direct code instrumentation.Source: §2.2 · p. 2
2.3 Performance Application Programming Interface (PAPI) - NVIDIA Management Library (NVML)
- PAPI provides low- and high-level APIs for performance counters, and its CUDA/NVML integration exposes GPU operation counts as well as power, energy, and temperature measurements.NVML queries include device power and temperature, while total energy consumption is reported in millijoules since the driver was last reloaded.Source: §2.3 · pp. 2–3
2.4 Kokkos
- Kokkos abstracts parallel patterns such as loops, reductions, and scans from hardware, and its profiling interface can load tools to collect kernel-level performance and debugging data without build changes.The space-time stack reports calls, execution time, application percentage, and memory information; the PAPI connector records configured events for parallel constructs and profiling sections.Source: §2.4 · p. 3
3 Related Work
Related work extends Roofline analysis toward energy, memory hierarchy, and instruction behavior, and develops tools for profiling applications. This paper positions its measurements and models alongside those approaches and application-aware thermal management.
3.1 Roofline and Energy Models
- Prior energy-aware Roofline research includes an energy-balance model, instruction-throughput modeling, modeled precision energy, GPU energy-per-operation plots [16], and cache-aware CPU power and energy models.Source: §3.1 · pp. 3–4
3.2 Profiling Tools
- Related profiling systems include a Kokkos Score-P adapter, PowerPack's distributed-system profiling framework, and Verificarlo's precision-analysis methodology.The paper's methodology uses PAPI through Kokkos Performance Tools for application-level instrumentation and a dedicated-kernel benchmark.Source: §3.2 · p. 4
3.3 Application-Aware Thermal Management
- Application-aware thermal management studies motivate distinguishing phases because they can respond differently to frequency changes, while control decisions must account for performance and monitoring or redistribution costs.The paper's application-level GPU measurements pair named regions and execution order with energy-aware Roofline and thermal inter-kernel views, supporting examination of temperature across regions with different computational behavior.Source: §3.3 · pp. 4–5
4 Methodology
The methodology combines application-level and repeated-kernel measurements with two complementary energy-aware visualizations. It accounts for measurement constraints and uses the models to relate kernel behavior to energy and thermal execution context.
4.1 Energy-focused Instrumentation Techniques
- Application-level measurements use Kokkos profiling and PAPI to identify dominant kernels, collect operation counts and memory traffic, record temperature, and retain region execution order.Direct measurements are restricted to regions lasting at least 101 ms because the energy counter updates at approximately 10 Hz; instrumentation costs 6.31 ms per region, about 6.25% of that duration in this setup.Source: §4.1 · p. 5
- Direct region energy is computed as the difference between cumulative NVML energy-counter readings at region boundaries, rather than by integrating separately sampled power.The counter is collected through PAPI and measures GPU energy; its readings remain subject to counter update behavior.Source: §4.1 · p. 5
- For shorter regions, a dedicated repeated-kernel benchmark amortizes boundary-read overhead and provides average per-kernel metrics, but omits region ordering and interactions during a full application run.The evaluated configurations are FP32, FP32 storage with FP64 computation, and FP64.Source: §4.1 · p. 5
- Temperature is an instantaneous NVML reading, whereas energy is derived from cumulative-counter differences; temperature is reported as the mean and sample standard deviation across runs.Energy measurements cover the GPU device.Source: §4.1 · p. 5
Energy-Aware Roofline Model
- The Energy-Aware Roofline Model adds GPU energy to arithmetic intensity and achieved throughput, allowing kernels to be compared by compute- or memory-bound behavior and energy consumption.The unified view is intended to identify high-energy kernels and prioritize optimization opportunities.Source: §4.2 · p. 5, Energy-Aware Roofline Model · p. 5
Thermal-Aware Inter-Kernel Energy Model
- The Thermal-Aware Inter-Kernel Energy Model plots region energy against duration, encodes GPU temperature by color, and uses idle-power and TDP reference lines to indicate average power.It includes regions without floating-point operations; a region's energy is accumulated consumption, while its distance from a reference line reflects average power rather than an energy bound or efficiency definition.Source: Thermal-Aware Inter-Kernel Energy Model · pp. 5–6, Figure 1 · p. 6
- Solid and dashed arrows distinguish consecutive from nonconsecutive kernel execution, while the pink area marks durations excluded from direct energy comparisons.Execution-order arrows can reveal potential measurement dependencies and similarities, but do not represent heat transfer, a continuous timeline, or demonstrated causal influence.Source: Thermal-Aware Inter-Kernel Energy Model · p. 6, Figure 1 · p. 6
5 Evaluation
Across LOBPCG and ArborX, precision effects vary by kernel and workload; FP64 often costs more energy, while execution-order and temperature measurements add context to region energy costs.
5.1 Applications
- LOBPCG uses kernels including sparse matrix-vector and matrix-matrix operations, preconditioning, and orthogonalization; ArborX evaluates DBSCAN over cosmology datasets using tree traversal.LOBPCG mixed precision varies data and computation precision, and the study excludes effects on algorithm-wide stability and accuracy.Source: Block Iterative Eigensolvers · p. 6, ArborX · pp. 6–7
5.2 Environment Setup
- The experiments use two NVIDIA H100 NVL GPUs, seven repetitions per benchmark, and separate PAPI event groups; ArborX region order follows the Kokkos Tools observations.The ArborX repetitions include a one-second pause but do not establish a common starting temperature.Source: §5.2 · p. 7
5.3 Application Kernel Analysis
- LOBPCG has about 40 mostly iterative short-lived kernels, while ArborX has 25 kernels, mostly invoked once; analysis focuses on kernels exceeding 5% of computational time.Selected kernels account for most execution time in each application's principal computational region.Source: §5.3 · p. 7
5.4 Application Kernel Characterization
- LOBPCG FP64 kernels have substantially lower arithmetic intensity and throughput, but higher energy use than FP32 and mixed-precision kernels; the analyzed kernels are memory-bandwidth-limited.The reported FP64 changes include intensity decreases of 50% for BLAS::dot and up to 64.2% for Preconditioner::apply, and energy increases of 60.3% for BLAS::dot and about 2× for BLAS::gemm.Source: Figure 2 · p. 8, §5.4 · p. 8
- For large ArborX workloads, FP64 lowers intensity and raises energy in BVH::BVH and DBSCAN::clusters::query, but raises intensity and throughput in DBSCAN::clusters::num neigh while also raising energy.The latter kernel's intensity and throughput increase by 19.8% and 29.6%, respectively, alongside a 39.5% energy increase.Source: Figure 3 · p. 8, §5.4 · pp. 8–9
- In ArborX, the longest, most energy-intensive regions run consecutively and are hotter; DBSCAN::clusters::query also uses 43.5% more FP64 energy than FP32 despite a slightly lower mean temperature.The corresponding query measurements are 705.9 J at 62.4 °C for FP32 and 1,012.6 J at 62.0 °C for FP64.Source: Figure 4 · p. 9, §5.4 · p. 9
- For small workloads, FP64 raises energy by up to 1.2% for two ArborX kernels, while LOBPCG FP64 generally runs longer, uses more energy, and has comparable or slightly lower temperatures.Small-workload DBSCAN::postprocess consumes 32.3% more mean energy in FP64 despite a lower mean temperature; short-kernel energy readings can be affected by sensor-update timing.Source: §5.4 · p. 10
5.5 Discussion
- Across LOBPCG, FP32 and mixed precision generally have comparable intensity, while mixed precision often uses modestly more energy; ArborX precision effects depend on kernel and workload.The discussion reports FP32 as the most favorable measured performance–energy trade-off for LOBPCG, while FP32's runtime and energy advantages in ArborX are more pronounced for larger workloads.Source: §5.5 · p. 10
6 Conclusion and Future Work
The paper presents two visualization models for analyzing Kokkos GPU applications and characterizes LOBPCG and ArborX across precisions and workloads. LOBPCG shows more consistent precision trends, while future work will broaden application coverage and test whether identified regions can guide interventions.
6 Conclusion and Future Work
- The study characterizes primary non-overlapping LOBPCG and ArborX kernels across numerical precisions and workloads using the two proposed visualization models.Source: §6 · p. 11
- The Energy-Aware Roofline and Thermal-Aware Inter-Kernel Energy models connect kernel behavior with energy and retain duration, execution order, and temperature, including for regions without floating-point operations.In ArborX DBSCAN::clusters::query, the large workload has substantially different energy costs at similar temperatures, making energy a distinct optimization priority.Source: §6 · p. 11
- Compared with ArborX, LOBPCG shows more consistent workload trends and larger, more systematic FP32–FP64 differences in arithmetic intensity, throughput, and energy.The LOBPCG benchmark characterizes repeated execution, whereas ArborX instrumentation retains application-region order; both temperature measurements can depend on execution history and instrumentation.Source: §6 · p. 11
- Future work will extend application coverage and assess whether identified regions are useful control points for DVFS, power capping, or thermal scheduling.That evaluation requires controlled thermal conditions and measurement of intervention effects on performance, GPU energy, and temperature.Source: §6 · p. 11