PaperlayerView exact revision on arXiv

Source-linked AI summary

Validating Memory-Optimal Transformer Kernels on Real Hardware: From Formal Derivation to Measured Performance Across Two HPC Clusters

Lenore M. Mullin, Gaetan Hains

arXiv:2609.33916v1cs.DCcs.AI

TL;DR

Prior work formally derived and verified memory-optimal transformer kernels, but its correctness checks did not measure performance at scale on real hardware. This paper tests those predictions across kernel families, clusters, and CPU/GPU targets, finding broadly diagnosable agreement alongside topology-dependent costs.

  • Problem

    Prior kernel verifications used one small fixed problem size on one CPU core or host fallback, leaving real-hardware performance predictions unmeasured.

    Source: §1 · p. 2

  • Method

    The paper checks derived cost predictions against real DRAM traffic, atomic counts, and thread scaling across CPUs, GPUs, and two clusters.

    Source: §1 · p. 2, §1.1 · p. 3

  • Results

    Across four kernel families, two HPC clusters, and CPU/GPU targets, predictions held within roughly 1.5–2× diagnosable margins; decode cost was 535× on Delta versus at most 2.9× on Anvil.

    Source: §8 · p. 32, §10 · p. 34

  • Takeaways & Limitations

    Topology-specific cost is part of the prediction, and diagnosed mismatches can guide ONF fixes while leaving the verified DNF unchanged.

    Source: §8 · p. 32, §10 · p. 34

    The two-cluster scope limits broad generalization; the authors call for more topologies before generalizing.

    Source: §7 · pp. 30–31

Abstract

from arXiv · show

We validate memory-optimal cost functions for transformer kernels derived via the Mathematics of Arrays (MoA). Companion Papers I-IV formally derive kernels for attention forward, backward, fused forward+backward, decode, and the complete block (RMSNorm, gated MLP) as a hardware-independent specification (DNF) transformed to a machine-specific realization (ONF) via gamma, with verification to machine precision against PyTorch. This paper checks those predictions against measured performance on two HPC clusters (Purdue Anvil, NCSA Delta) across CPU and GPU. Three results stand out. (1) We identify and fix a GPU regression: fusing forward+backward, proven to avoid materializing an O(n^2) intermediate, initially ran slower than naive on GPU due to atomic contention. Profiling confirmed 2.00x more atomic instructions; a targeted ONF rewrite reversed it, yielding up to 2.5x speedup. (2) Identical derivations produce markedly different real costs by topology: 535x NUMA-locality penalty on one cluster vs <3x oversubscription on another, showing optimal deployment is a function of the machine's array structure. (3) We report a partially resolved anomaly: identical denotational computations run faster in C than Fortran on CPU but faster in Fortran than C on GPU, narrowed to one dominant kernel and one memory-latency stall mechanism (3.17x time gap matches 3.35x stall gap). We treat hardware-specific optimization as a routine ONF rewrite with fixed, verified DNF, a candidate methodology for scaling AI onto evolving hardware without re-deriving correctness.

1 Introduction

This paper measures the real-hardware performance of kernel designs formally derived in Papers I–IV, testing whether MoA’s machine-specific cost predictions match measured behavior. It treats the machine’s memory structure as an array so that the derived realization yields checkable memory-access costs.

1 Introduction

  • The paper checks on real hardware whether cost predictions for Papers I–IV’s formally derived transformer kernels match measured performance; it does not introduce new derivations or kernel designs.The tested kernels cover attention, decode, and the complete transformer block.Source: §1 · p. 2

1.1 From array algebra to a real machine: DNF, ONF, and γ

  • γ translates a hardware-independent DNF into an ONF that specifies loop order, tiling, and which intermediates are recomputed or stored.Because the machine’s memory layout and access pattern can be described in the same shape/index vocabulary, the ONF’s cost functions are byte counts that can be checked on hardware.Source: §1.1 · p. 3
  • The empirical test compares predicted machine behavior with measured DRAM traffic, atomic counts, or thread scaling, diagnosing mismatches through specific memory-access or contention mechanisms.The paper’s empirical claim is narrower than the parent papers’ correctness proofs: it tests γ’s machine-as-array predictions across real CPUs, GPUs, and two clusters.Source: §1 · p. 2, §1.1 · p. 3

1.2 Related work

  • FlashAttention independently reaches a similar qualitative conclusion about avoiding DRAM materialization of the n × n attention score matrix, but this paper does not benchmark against it.Instead, the paper measures whether the cost functions for its own derived kernels predict that benefit on real hardware.Source: §1.2 · pp. 3, 5

2 Methodology

The methodology benchmarks verified kernels across CPU and GPU resources on Purdue Anvil and NCSA Delta, while excluding Delta’s AMD GPUs because the toolchain does not target them. It also corrects a GPU timing-harness flaw that obscured decode’s scaling.

2.1 Machines and allocations

  • Delta’s gpuMI100x8 (AMD, ROCm) is present but excluded from this study because every kernel is compiled with nvc/OpenACC, which does not target AMD hardware.Source: §2.1 · p. 5

2.2 Kernels benchmarked

  • Kernels are benchmarked by doubling sequence length and, on CPU, thread count; correctness is checked separately against PyTorch or a serial-CPU reference.The report focuses on timing rather than correctness, except where noted.Source: §2.2 · p. 5

2.3 A real methodological bug: GPU cold-start dominance

  • A per-process GPU timing harness added about 300 ms of context and JIT startup, masking decode’s O(n) runtime across a 1024× sequence-length range.After switching to one process per sequence-length sweep with an untimed warmup, measured decode time rose from 0.00014 s at n = 1024 to 0.72 s at n = 1,048,576; the other five kernels received the same fix.Source: §2.3 · pp. 5–6
  • A naive per-call timing harness can overstate small-sequence GPU costs by roughly 300 ms and obscure a fast kernel’s asymptotic complexity.The paper reports this measurement issue because it affected all six GPU benchmark programs.Source: §2.3 · pp. 5–6

3 Paper I: Forward Pass

Paper I’s real-hardware runs confirm the predicted scaling while exposing deployment and implementation effects that change measured cost. The CPU optimum depends on sequence length, GPU scaling approaches O(n^2), and profiling finds excess DRAM traffic tied to uncoalesced access.

3.1 CPU scaling (Delta, confirmed)

  • At n = 64, using all 128 threads is 14.5× slower than the 4-thread optimum; the crossover to more threads helping lies between n = 1024 and n = 2048.At n = 8192, wall time at 1 thread approximately quadruples with each doubling of n, confirming the O(n^2) prediction; 128 threads gives 60.6× speedup over 1 thread, with 47% parallel efficiency.Source: Table 1 · p. 6, §3.1 · pp. 6–7

3.2 GPU shape comparison

  • At n = 8192, the local exponent reaches 1.97 on A100 and 2.04 on H200, while H200’s advantage shrinks from 1.75× at n = 256 to 1.06× at n = 8192.H200 is faster at every tested size, with its advantage concentrated at smaller, more launch-overhead-sensitive sizes.Source: §3.2 · pp. 7, 10, Table 2 · p. 8

3.3 Validation against Mfwd

  • At N=2048, measured DRAM traffic is 2.01× the naive prediction; writes are 2.06× versus reads at 1.44×, consistent with uncoalesced access.Stores use 9.4 of 32 bytes per transmitted sector (29% efficient), while loads use 18.2 of 32 (57% efficient); the timed kernel reaches 95.4% of its theoretical occupancy ceiling, and 17.4% of wall time is host-side overhead.Source: Table 3 · p. 10, §3.3 · p. 11

3.4 Scope decision and summary

  • Real-hardware validation confirmed GPU scaling near O(n^2), a size-dependent H200 advantage, and roughly 2× excess DRAM traffic at N=2048 from a fixable coalescing issue.The reported results attribute the traffic gap to current code generation, not the memory-optimal derivation.Source: §3.4 · p. 11, §3.3 · p. 11
  • Paper I’s experimental validation is complete; its GPU results are C-only because no Fortran90 GPU kernel was built.The corrected A100 run, H200 shape comparison, and Mfwd profiling were all performed and analyzed.Source: §3.4 · p. 11

4 Paper II: Backward Pass and Fused Forward+Backward

The CPU and GPU measurements initially diverged: fusion helped on CPU but regressed on GPU because of atomic contention. Rewriting the GPU kernel reversed the result, while removing another atomic produced a size-dependent trade-off.

4.1 First real submission: a genuine crash, found and fixed

  • The first CPU submission crashed when an OpenMP array reduction’s 8MB stack allocation matched the cluster’s 8MB stack limit; replacing reductions with heap-allocated intermediates fixed the trigger case.A search found three other Paper IV reduction sites with buffers not scaling with N; they did not crash at their tested parameters.Source: §4.1 · p. 13

4.2 CPU results: the real fusion comparison (job 21226440)

  • At N=4096 on CPU, fused/naive time improved from 1.19× at one thread to 0.72× at 128 threads.Six of eight tested sizes favored fusion at 128 threads; N=128 was a 2.07× slower outlier, and timing was not repeated and averaged.Source: Figure 9 · p. 14, §4.2 · p. 15

4.3 GPU results (jobs 21225789, 21225790): a surprising, opposite finding

  • On GPU before the fix, fusion was slower than naive at every tested size, reaching 1.84× on A100 and staying near 1.5× on H200.The CPU and GPU comparisons use a naive baseline combining forward-alone and backward-alone timings.Source: §4.2 · p. 13, §4.3 · pp. 15–16, Figure 10 · p. 16

4.4 The atomic-contention hypothesis: confirmed

  • At N=2048, fused executed 2.0000× backward’s atomic instructions, while device time was 1.9716×; the atomic-using backward kernel was also its slowest and least occupied.The fused kernel used atomics for both GV and GK, compared with GK alone in backward.Source: §4.4 · p. 17, Figure 11 · p. 17

4.5 The fix, applied and confirmed: complete reversal on both shapes

  • After the GPU fix, fused was faster than naive at every tested size: on H200, fused/naive time fell from 0.920× at N=64 to 0.401× at N=8192.At N=8192, the A100 ratio was 0.517×.Source: §4.5 · p. 18, Figure 12 · p. 18

4.6 Removing GK’s atomic too: a real, nuanced result

  • Removing GK’s atomic improved backward alone to 0.56–0.81× its old time, but fused slowed to 1.32× at N=8192 after improving at smaller sizes.For fused, the crossover was measured at N=4096 (1.04×) and N=8192 (1.32×).Source: §4.6 · p. 19, Figure 13 · p. 19

4.7 Confirming the mechanism: the extra GS re-read, profiled directly

  • At N=4096, removing GK’s atomic raised total DRAM traffic from 45.60GB to 89.83GB; the standalone GK pass read 43.01GB, nearly matching the preceding GS read.The atomic version’s combined GQ+GK kernel read 0.30GB because it used each thread’s GS value without a DRAM round-trip.Source: §4.7 · pp. 19–20

5 Paper III: Inference (Decode)

Decode measurements reveal a severe, topology-dependent CPU thread-count penalty on Delta but ordinary oversubscription behavior on Anvil. After correcting GPU timing, measured decode scaling aligns with its O(n) cost prediction.

5.1 CPU: two real findings, one benign, one genuinely surprising

  • The first Delta decode run’s output was unrecoverable, so the reported validation uses clean resubmission 21249667.The earlier job exited successfully but left no log or CSV available.Source: §5.1 · p. 20
  • At the largest tested N, every thread count from 2 to 128 triggered a 3.00×10^-3–3.05×10^-3 absolute-error warning against the sequential reference.The warning uses a 10^-3 threshold and single-precision arithmetic; its stable magnitude across thread counts is consistent with accumulated summation-order differences, not a logic error.Source: §5.1 · p. 20
  • At N=1024, Delta’s 128-thread decode run is 535× slower than its 64-thread run, consistent with remote-memory and fixed scheduling overhead dominating small workloads.The timing remains roughly 0.04–0.05s across N=1024–32768, then reaches 1.28× slower at N=1,048,576.Source: §5.1 · pp. 20–22, Figure 14 · p. 21

5.2 Cross-cluster comparison: two different penalties, two different causes

  • Anvil’s oversubscription penalty shrinks with N, then reverses into a benefit from N=16384; Delta’s NUMA penalty remains severe within the tested range.The reported contrast is consistent with two distinct mechanisms: latency hiding under oversubscription versus remote-memory latency across NUMA domains.Source: §5.2 · p. 22, Figure 15 · p. 23

5.3 GPU: fix confirmed on real hardware

  • 0.77–1.16 is the decode GPU’s local scaling exponent from n=32768 onward, close to the O(n) target after the timing fix.Time rose from 0.00036s at n=1024 to 0.0809s at n=1,048,576; the fixed timing methodology removed the misleading flat pre-fix result.Source: §5.3 · pp. 22, 25

5.2 Cross-cluster comparison: two different penalties, two different causes

  • 535× is the 128-versus-64-thread penalty on Delta, while Anvil’s 64-versus-32-thread penalty never exceeds 2.9×.Delta’s 128 threads span eight NUMA domains; Anvil’s 64 threads oversubscribe 32 physical cores, and its ratio becomes a benefit as N grows.Source: §5.2 · p. 22, Figure 15 · p. 23

6 Paper IV: Transformer Block

The complete-block measurements show architecture-dependent language performance, expected GPU scaling, and a measured DRAM-traffic gap whose cause remains partial. CPU results also isolate a real scaling jump and a memory-stall association in the dominant gated-MLP backward kernel.

6.1 CPU: real data, and a genuine cross-language reversal

  • RMSNorm shows no consistent C-versus-Fortran pattern across 11 sizes, with ratios from 0.70× to 1.55× in single measurements.The lightweight kernel’s jitter may dominate the signal; this benchmark lacks repeated averaging.Source: §6.1 · p. 25
  • 3.165× slower is C than Fortran for the profiled gated-MLP backward kernel, which accounts for 87.7% of C’s and 65.3% of Fortran’s profiled total.C’s long-scoreboard stall ratio is 4.56 versus Fortran’s 1.36 for this kernel; the 3.35× stall gap is close to the 3.17× time gap.Source: §6.1 · pp. 26–27
  • N=2048 →4096 remains a 12.5× jump after five-repeat averaging, rather than smoothing toward the expected growth.The repeated measurement rules out single-shot noise; the cache-capacity explanation is consistent with the data but has not been directly confirmed.Source: §6.1 · pp. 27–28, Figure 19 · p. 27
  • The complete block’s worst 128-thread penalty is 6.98× at N=64, falling to 1.0–1.3× by N=256 and beyond.The measured penalty is much milder than decode’s 535× on the same Delta topology, consistent with greater per-thread work in the complete block.Source: §6.1 · p. 28

6.2 GPU: real data, clean O(n2) confirmation, and Fortran’s advantage

  • 4.40×, 4.10×, and 3.91× are the complete-block GPU growth ratios across successive doublings from N=64 to 512, approaching the O(n^2) prediction.The tested range is limited to N=512 by the GPU sweep setting.Source: §6.2 · p. 28
  • Fortran is 1.45–2.81× faster than C for the fused norm+MLP GPU kernel, while C is 1.2–1.85× faster than Fortran for the CPU MLP kernel.The computations have identical denotational specifications; the GPU code-generation cause remains unidentified.Source: §6.1 · pp. 25–26, Figure 17 · p. 25, §6.2 · p. 28

6.3 Mblock: real DRAM traffic validation

  • 15.74MB measured versus 10.49MB predicted is 1.50× the predicted Mblock DRAM traffic across the complete-block launches.Coalescing inefficiency is measured but does not fully reconstruct the gap: ideal-coalescing traffic is 5.82MB, and the remaining explanation is unresolved.Source: §6.3 · pp. 28, 30

7 Scope and Limitations

The study tests whether MoA-derived designs match their own predictions, not whether they compete with production kernels. Its measurements cover limited hardware and use varying statistical rigor, so broader generalization requires care.

Scope

  • The paper evaluates MoA-derived kernels against their own cost predictions, not against FlashAttention, vendor libraries, or other optimized implementations.It makes no claim about competitive performance against the state of the art.Source: §7 · p. 30

Hardware coverage

  • Hardware coverage is limited to two clusters, four GPU models, and AMD EPYC CPUs; Intel CPUs, AMD GPUs, and cloud hardware are absent.The cross-cluster NUMA result shows topology affects measured cost, supporting tests across more topologies before broad generalization.Source: §7 · pp. 30–31

Measurement rigor

  • Statistical rigor varies by benchmark: some results use five-repeat averages, while some are single-shot measurements without formal confidence intervals.Single-shot results on shared nodes may be affected by scheduling jitter, which the paper does not quantify.Source: §7 · p. 31

Reproducibility

  • Benchmark source, submission scripts, and analysis code are being prepared for archival release; job identifiers are retained for interim reproducibility.The archival release is not yet reported as complete.Source: §7 · p. 31

8 Discussion

Real-hardware tests add findings unavailable from correctness-only verification, including scheduling effects and GPU timing bugs. The discussion’s clearest diagnosis–prediction–verification cycle links an ONF feature to measured GPU cost, while topology changes the measured cost of the same decode schedule.

Discussion

  • Real-hardware testing exposed scheduling crossovers that correctness-only verification could not establish, including the Paper I CPU thread-count-versus-n crossover.That result concerns scheduling overhead on a specific machine, not a prior theoretical claim about the DNF.Source: §8 · p. 31
  • Profiling found 2.0000× backward’s atomic instructions in fused attention, matching the ONF’s predicted GV-and-GK versus GK-only accounting; restructuring the ONF reversed the GPU result.The rewritten design was tested on both GPU shapes, demonstrating a measurable cost predicted by a specific ONF feature.Source: §8 · p. 31
  • The same decode schedule incurs a 535× penalty on Delta but no more than 2.9× on Anvil, consistent with their different NUMA and oversubscription topologies.The measured cost depends on the target machine’s array structure rather than being a universal value.Source: §8 · pp. 31–32
  • Removing GK’s remaining atomic helped backward uniformly but made fused attention slower at N≥4096, showing that a fix must be checked on each kernel structure.The N=2048 →4096 full-block jump also persisted after repeated averaging, ruling out measurement noise as its explanation.Source: §8 · p. 32
  • The discussion recommends real-hardware validation before external use of cost-function claims and investigation of discrepancies within the ONF’s ρ/ψ/ι structure.It also treats mixed outcomes as part of the evidence, rather than reporting only successful fixes.Source: §8 · p. 32

9 Implications: active rewrites as a scaling strategy for AI systems

The project uses real-hardware results to argue that keeping the DNF fixed while actively rewriting the machine-specific ONF is a practical approach to changing hardware. The evidence supports this argument for the tested kernels and clusters, but broader generalization remains a research direction.

9.1 The problem this project’s own experience illustrates

  • The same fused norm+MLP computation ran faster in C than Fortran on CPU, but faster in Fortran than C on GPU.The reversal occurred across the two tested targets despite the source code implementing the same ONF.Source: §9.1 · pp. 32–33
  • The project also encountered ordinary infrastructure drift, including a Delta nvhpc upgrade that broke scripts hardcoding the previous version and a DCGM profiling conflict.The conflict required a cluster-specific workaround.Source: §9.1 · p. 33

9.2 What MoA’s DNF/ONF separation buys, concretely, in this project’s own evidence

  • The GPU rewrite kept the fused forward+backward DNF unchanged, removed an atomic site through a four-kernel ONF, and reduced wall-clock cost 2–2.5× on both tested GPU shapes.The diagnosis came from the ONF’s ρ/ψ/ι structure; the inherited correctness proof did not need re-verification.Source: §9.2 · p. 33
  • Table 1 and Section 5.2 show that deployment choices depend on machine topology: the measured penalty reached 535× on Delta but was under 3× on Anvil.A deployment default selected for one machine can be actively harmful on another.Source: §9.1 · p. 32, §9.2 · p. 33

9.3 The broader argument, stated as a research direction

  • Across the tested kernels, cost-prediction gaps were diagnosable in DNF/ONF terms, including Paper I’s coalescing gap and Paper II’s atomic contention.The paper argues that the machine-specific ONF can be revised against a checkable target while the hardware-independent DNF remains fixed.Source: §9.3 · pp. 33–34
  • The broader scaling argument remains unestablished beyond the tested kernels and two clusters, despite the observed pattern holding across four papers and CPU and GPU targets.The authors present wider AI-workload and production-hardware generalization as a research direction.Source: §9 · p. 32, §9.3 · p. 34

10 Conclusion

The paper tests whether formally derived memory-optimal cost functions predict real hardware behavior and reports results across four kernel families, two clusters, and CPU and GPU targets.

10 Conclusion

  • Across four kernel families, two clusters, and CPU and GPU targets, memory-cost predictions held within roughly 1.5–2× when discrepancies were investigated, and the GPU regression’s gap pointed to a confirmed fix.The C-versus-Fortran reversal in Paper IV remains only partially explained.Source: §10 · p. 34
  • The paper argues that keeping correctness in a hardware-independent specification while revising machine-specific realizations offers a practical scaling strategy as hardware changes.The proposed strategy uses specific, checkable targets rather than open-ended search.Source: §10 · p. 34
  • The conclusion’s evidence is limited to four kernel families, two clusters, and CPU and GPU targets; it does not establish that the broader strategy generalizes to all AI systems.The paper reports the unresolved C-versus-Fortran reversal alongside its validated results.Source: §10 · p. 34

A Real-Hardware Job Identifiers

The appendix identifies Table 4 as a reproducibility record for the real-hardware runs behind the paper’s claims.

A Real-Hardware Job Identifiers

Related papers

Loading 2609.33916v1…