PaperlayerView exact revision on arXiv

Source-linked AI summary

Kernel Performance Engineering under Physical Non-Idealities

Supun Kuruppu, Pranav Dangi, Wei Li, Zhenyu Bai, Tulika Mitra

arXiv:2610.37466v1cs.AR

TL;DR

Physical non-idealities concealed by conventional abstractions can constrain kernel performance and obscure optimization opportunities. The paper analyzes floorsweeping and power constraints through DSMEM on H100 GPUs and evaluates wave-aware clustering. It reports throughput gains for FlashAttention 4 and shows that TDP-related frequency effects also matter on GPUs and Wormhole.

  • Problem

    Physical non-idealities are concealed by conventional hardware abstractions, though they can limit kernel performance and obscure optimization opportunities.

    Source: §1 · p. 1

  • Method

    The paper analyzes DSMEM on H100 GPUs, characterizes floorsweeping’s effects on clustered execution, and develops wave-aware clustering while examining TDP-related frequency effects.

    Source: §1 · p. 2

  • Results

    Wave-aware clustering improves Hopper FlashAttention 4 throughput by up to 5.30% over its cluster-size-1 baseline, while the evaluations also show resource and power constraints affecting performance across GPUs and Wormhole.

    Source: §1 · p. 2, §8 · pp. 12–13

  • Takeaways & Limitations

    Accounting for floorsweeping and power constraints helps explain kernel performance behavior and identify optimization opportunities within hardware’s physical constraints.

    Source: §8 · p. 13

Abstract

from arXiv · show

Physical non-idealities in integrated circuit (IC) design and fabrication are concealed by conventional hardware abstractions, yet they can limit kernel performance and obscure optimization opportunities. Using distributed shared memory (DSMEM) on NVIDIA H100 GPUs as a case study, we show how floorsweeping restricts hardware resources available for clustered execution, potentially offsetting the benefits of reduced data movement enabled by DSMEM. Guided by these insights, we develop a wave-aware clustering strategy that accounts for these resource constraints to improve kernel performance. Evaluations using a global-load-and-broadcast microbenchmark, GEMM, and the Hopper version of FlashAttention 4 demonstrate the performance implications of these constraints, with wave-aware clustering improving FlashAttention 4 throughput by up to 5.30% over its baseline. We further show that, under thermal design power (TDP) limits, reduced data movement enables higher Streaming Multiprocessor (SM) frequencies through Dynamic Voltage and Frequency Scaling (DVFS), explaining otherwise unexpected throughput improvements. Extending this analysis beyond GPUs, we show that power constraints also limit throughput scaling on Tenstorrent's Wormhole accelerator, highlighting the broader importance of accounting for physical non-idealities when analyzing and optimizing kernel performance.

1 INTRODUCTION

The introduction argues that physical non-idealities hidden by conventional abstractions affect kernel performance and can conceal optimization opportunities. Using DSMEM on H100 GPUs, it connects floorsweeping and power limits to clustered execution and presents wave-aware clustering as a response.

1 INTRODUCTION

  • Physical non-idealities such as floorsweeping and TDP limits remain hidden by conventional hardware abstractions, yet can limit achievable kernel performance.The paper uses DSMEM on NVIDIA H100 GPUs as its case study of how these effects inform kernel engineering and explain performance variation.Source: §1 · p. 1
  • DSMEM lets thread blocks in a cluster access one another’s shared memory, enabling a block to load data and broadcast it to blocks needing the same data.Despite this opportunity, the introduction notes that modern FlashAttention kernels do not use the feature and that expected gains do not materialize at GPC-level cluster sizes.Source: §1 · p. 2
  • Floorsweeping leaves only 120 of the H100’s 132 SMs concurrently usable for GPC-level clustering, potentially adding execution waves that offset DSMEM’s reduced redundant loads.The paper calls 120 the active SM limit and attributes the constraint to functional-resource organization into virtual GPCs.Source: §1 · p. 2
  • Wave-aware clustering improves Hopper FlashAttention 4 throughput by up to 5.30% over the cluster-size-1 baseline.The strategy selects between TPC and GPC clustering based on the kernel’s wave behavior.Source: §1 · p. 2
  • Under GPU power limits, reduced data movement lowers transfer power and allows higher sustained SM frequency through DVFS; Wormhole results also show power constraints limiting throughput scaling as more cores activate.The paper presents both effects as evidence that physical non-idealities matter beyond GPUs.Source: §1 · p. 2

2 BACKGROUND AND RELATED WORK

This section describes Hopper DSMEM’s cluster and SM organization, then situates the paper’s analysis of fabrication and power constraints within prior work. It identifies floorsweeping-related resource limits and TDP-induced frequency effects as the paper’s focus.

2.1 Distributed shared memory in NVIDIA Hopper GPU architecture

  • Hopper DSMEM connects shared memories across SMs within a GPC; clusters of size 2 stay within a TPC, while sizes 4 and 8 span GPC-level groups.The GPU has eight GPCs, each formed from nine TPCs; clustered execution can reduce data movement but adds synchronization and scheduling constraints.Source: §2.1 · pp. 2–3

2.2 Impact of physical non-idealities on kernel performance

  • Prior work reports declining DSMEM throughput at larger GPC-level cluster sizes, fewer SMs available for clustered execution, and floorsweeping as a potential cause of resource restrictions.The section also identifies prior work on physical SM topology and cluster scheduling, especially with MIG partitioning and Green Contexts.Source: §2.2 · p. 3
  • The paper characterizes floorsweeping’s resource restrictions, derives wave-equalization conditions, and investigates how TDP-induced frequency scaling combines with those restrictions.Prior studies covered DSMEM behavior and power-induced frequency changes separately, but had not isolated how DSMEM data-movement reductions affect operating frequency.Source: §2.2 · p. 3

3 METHODOLOGY

The methodology evaluates physical non-idealities on four H100 SXM5 GPUs using several CUDA kernels and profiling. It also modifies Hopper FlashAttention to use DSMEM for global loading and broadcasting shared key and value tiles.

3 METHODOLOGY

  • Experiments use four H100 SXM5 GPUs, four CUDA kernels, CUDA 13.0 with the sm 90a target, and NVIDIA Nsight Compute profiling.The GPUs are geographically distinct to reduce the likelihood that observations reflect a single manufacturing batch.Source: §3 · p. 3
  • The GEMM experiment uses CUTLASS 4.8.0’s Hopper warp-specialized kernel, while FlashAttention uses the Hopper forward kernel.GEMM uses three pipeline stages and non-persistent execution; FlashAttention processes query tiles while iteratively loading key and value tiles.Source: §3 · p. 3
  • The FlashAttention code is modified to use DSMEM global-load-and-broadcast for shared key and value tiles in non-causal multi-head attention.Different query-tile blocks need the same key and value tiles, creating an opportunity to reduce redundant loads.Source: §3 · p. 4

4 DEMYSTIFYING FLOORSWEEPING EFFECTS ON CLUSTERED EXECUTION

The section shows that floorsweeping and the H100 skyline constrain which SMs can participate in clustered execution, sometimes offsetting DSMEM’s reduced data movement. It connects these constraints to execution waves and details how the skyline and active-SM limits were measured.

4.1 Microbenchmarking global load and broadcast

  • For the microbenchmark, single-block broadcast achieves lower latency than all-blocks broadcast because smaller TMA transfers can reduce effective memory bandwidth.At cluster sizes above 1, the all-blocks pattern issues the same number of transfers as cluster size 1, but with smaller transfer sizes.Source: §4.1 · p. 4
  • GPC-level clustering reduces data transferred from L2 to shared memory, but its expected latency improvement does not materialize because fewer SMs are active.The microbenchmark measures 256 blocks loading 1 MB in 128 KB chunks; its results show the same unexpected trend described for the kernel in Fig. 1.Source: §4.1 · pp. 4–5, Figure 4 · p. 4

4.2 Active SM limits at different cluster sizes

  • At cluster sizes 1 and 2, all 132 H100 SMs are used, while GPC-level sizes enable 124 SMs and permit at most 120 concurrently.Table 1 reports unique active SMs within iterations and across all iterations for each cluster size.Source: §4.2 · p. 5, Table 1 · p. 5

4.4 Reverse engineering the skyline

  • A kernel that records scheduled SM IDs across increasing cluster sizes and 1,000 iterations identifies 12 H100 vGPCs, despite the GPU having eight physical GPCs.The inferred grouping merges sets of SM IDs shared across observed clusters.Source: §4.4 · p. 6

4.3 Effect of floorsweeping on active SM limit

  • The skyline maps differing physical fault patterns to a common virtual resource profile, but can fragment resources and make otherwise functional TPCs unavailable for multi-TPC clustered execution.The H100 skyline includes single-TPC vGPCs; such singletons cannot participate in GPC-level clusters.Source: §4.3 · p. 5, §4.4 · p. 6

4.5 Impact of the skyline on the active SM limit

  • At GPC-level cluster sizes, floorsweeping leaves at most 120 SMs concurrently active, down from 132 exposed SMs, because some SMs cannot form supported clusters or remain unassigned.The section reports 124 SMs enabled for clustered execution before partitioning constraints reduce the concurrent limit to 120.Source: §4.2 · p. 5, Table 1 · p. 5, §4.5 · p. 6
  • SMs with neighboring logical IDs are frequently grouped into clusters, and the resulting 120-SM concurrent limit can add execution waves and reduce kernel performance.The section says the following analysis characterizes when this additional-wave penalty occurs and when wave counts remain unchanged.Source: §4.5 · p. 6, Figure 6 · p. 7

5 NAVIGATING ACTIVE SM LIMIT PENALTIES

Floorsweeping reduces the active SM limit, which can add execution waves and offset clustering’s data-movement benefits. The section evaluates wave-equalized launches and a wave-aware clustering strategy across broadcast, GEMM, and FlashAttention workloads.

5 NAVIGATING ACTIVE SM LIMIT PENALTIES

  • A wave-equalized launch requires the same number of execution waves under the full 132-SM and reduced 120-SM limits, avoiding the additional-wave penalty from scheduling the same blocks across fewer SMs.Kernels limited to one thread block per SM are especially affected because they have fewer blocks in each wave.Source: §5 · p. 7

5.1 Wave-equalized global load and broadcast within clusters

  • In the global-load-and-broadcast microbenchmark, clustering reduces latency for a wave-equalized 240-block launch but increases it for a non-wave-equalized 256-block launch at GPC-level cluster sizes.The comparison illustrates how extra execution waves can offset latency benefits from reduced data movement.Source: §5.1 · p. 7, Figure 7 · p. 7

5.2 Wave-equalized GEMM kernel launches

  • For the wave-equalized 7680 × 4096 × 512 GEMM configuration, which supports all evaluated cluster sizes, GPC-level clustering improves throughput relative to cluster size 1.Source: §5.2 · p. 8, Figure 9 · p. 8
  • For GEMM workloads where GPC-level clustering requires 12 additional waves, its throughput is significantly lower than with cluster sizes 1 and 2.The section relates the relative clustering benefit to workload execution-wave counts; larger GEMM operands can also increase the benefit of GPC clustering by generating more HBM reads that broadcasting can reduce.Source: §5.2 · pp. 7–8, Figure 8 · p. 8

5.3 Wave-equalized FlashAttention kernel launches

  • The FlashAttention evaluation compares wave-equalized 15-head configurations with non-wave-equalized 16-head configurations across sequence lengths; the 128 × 80 tile shape has the highest throughput among those evaluated.Figure 10 presents throughput increases over cluster size 1 across sequence lengths for wave-equalized and non-wave-equalized launches.Source: §5.3 · p. 8, Figure 10 · p. 8

5.4 Wave-aware clustering on FlashAttention

  • Wave-aware clustering improves FlashAttention 4 throughput over the cluster-size-1 baseline by up to 5.30% and beats fixed TPC or GPC clustering at every evaluated instance.The reported throughput gains are not driven by increased tensor-core utilization from reduced data movement; the authors attribute them to TDP limits and their impact on operating frequency.Source: Figure 11 · p. 9, §5.4 · p. 9

6 DEMYSTIFYING THE IMPACT OF THERMAL DESIGN POWER

Under TDP limits, reduced data movement can raise SM frequency through DVFS and improve throughput, while floorsweeping and synchronization constrain clustered execution. Power limits also constrain scaling on Wormhole.

6.1 Effect of TDP on kernel performance

  • TPC-level clustering achieves approximately 1.5% and 3% higher throughput than cluster size 1 for head dimensions 128 and 256, respectively.L2 reads are approximately halved with each doubling of cluster size.Source: §6.1 · p. 10, Figure 13 · p. 10
  • Reduced transfer power allows higher SM frequency through DVFS; throughput and frequency increases have Pearson ρ >0.9 across evaluated cluster sizes and head dimensions.The FlashAttention evaluation attributes the throughput gains to TDP limits and frequency scaling, not increased tensor core utilization.Source: §6.1 · pp. 10–11, §6 · p. 10, Figure 14 · p. 11
  • Increasing GPC cluster size from 4 to 8 improves throughput at head dimension 256 but reduces it at 128, where tensor core utilization cannot mask synchronization overhead.At dimension 256, tensor core utilization approaches its maximum, helping mask that overhead.Source: Table 4 · p. 10, §6.1 · p. 11

6.2 Effect of TDP on other AI accelerators

  • On Wormhole, HiFi4 throughput plateaus at roughly 31 TFLOPS, while LoFi reaches around 60 TFLOPS at 64 cores under the default power limit.At 64 cores, HiFi4 clock falls to about 57% of nominal; LoFi is throttled to about 86%.Source: §6.2 · pp. 11–12, Figure 15 · p. 11

7 DISCUSSION

Wave equalization has a limited set of eligible thread-block counts on the H100, but its condition can be adapted to other GPUs. The discussion also cautions that physical non-idealities challenge kernel-development assumptions.

7.1 Stringency and portability of wave-equalization conditions

  • On H100, wave-equalized launches require 132(n −1) < B ≤ 120n, with positive integer n < 11; the eligible thread-block-count set is limited.B is the thread-block count and n the common number of execution waves.Source: §7.1 · p. 12
  • For another GPU, replace 132 and 120 with the complete and reduced active-SM limits measured using the unclustered and GPC-clustered microbenchmark.The adapted limits yield that GPU’s wave-equalization condition.Source: §7.1 · p. 12

7.2 Implications for kernel development systems

  • Physical non-idealities challenge kernel-development assumptions: amortizing memory accesses or increasing compute utilization to saturation can adversely affect performance.The paper also notes possible numerical nondeterminism from changed floating-point evaluation order and power spikes; future work will extend the analysis to other physical non-idealities.Source: §7.2 · p. 12

8 CONCLUSION

The conclusion argues that accounting for physical non-idealities helps explain kernel performance and identify optimization opportunities. Its evaluations connect floorsweeping and power constraints to clustered execution and throughput.

8 CONCLUSION

  • Wave-aware clustering improves Hopper FlashAttention 4 throughput by up to 5.30% over its baseline across sequence lengths from 2K to 64K.The strategy chooses between TPC and GPC clustering for a kernel.Source: §8 · p. 12
  • Floorsweeping limits GPC-level clustered execution to at most 120 concurrently active SMs despite 132 exposed, potentially adding waves that offset reduced data movement.The GPU exposes 124 SMs as eligible for GPC-level clustering, with at most 120 active concurrently.Source: §8 · p. 12
  • Reduced transfer power can provide TDP headroom for higher SM frequencies through DVFS, and Wormhole results show power constraints also affect frequency and achievable throughput beyond GPUs.The paper frames these findings as evidence that physical constraints matter when explaining kernel performance.Source: §8 · pp. 12–13

Related papers

Loading 2610.37466v1…