PaperlayerView exact revision on arXiv

Source-linked AI summary

Memory-Bound but Not Bandwidth-Limited: The Physical AI Inference Gap in Batch-1 LLM Decode

Josef Chen

arXiv:2605.30571v1cs.ARcs.AIcs.DCcs.PFcs.RO

TL;DR

Physical-AI inference commonly relies on batch-1 autoregressive decode, but evidence is limited on whether memory bandwidth alone determines latency. This measurement study spans GPUs, models, and contexts, finding that CPU launch overhead limits fast-GPU decode while kernel choice can dominate performance on cheaper GPUs.

  • Problem

    Physical-AI inference uses batch-1 autoregressive decode, but its latency is insufficiently characterized beyond the usual memory-bandwidth-bound account.

  • Method

    The paper measures batch-1 bf16 decode across three 7–8B GQA transformers, four NVIDIA GPUs, and contexts from 2048 to 16384.

  • Results

    1.259× CUDA Graphs speedup on H100 versus 1.028× on L4 shows launch overhead binds on fast GPUs, while HBM bandwidth binds on L4.

  • Takeaways & Limitations

    On L4, GPTQ+ExLlamaV2 cuts Qwen-2.5-7B step time from 62.32 ms to 17.36 ms, outperforming AutoAWQ+Marlin at 45.24 ms.

  • Takeaways & Limitations

    The evaluation covers only grouped-query attention and does not measure batch sizes of at least 8 on H100 or above 4 on L4.

Abstract

from arXiv · show

Physical AI systems, including robots, autonomous vehicles, embodied agents and edge copilots, often run a different inference workload from cloud LLM serving: single-stream, batch-1 autoregressive decode, where one robot, camera feed or user session waits on the next token. This workload is usually described as memory-bandwidth-bound. Each decode step streams model weights and the active KV cache, so latency should scale with peak HBM bandwidth. We show that this account is true but incomplete. We measure batch-1 decode for three 7 to 8B-class GQA transformers across four NVIDIA GPUs: H100 SXM5, A100-80GB SXM4, L40S and L4. We evaluate context lengths from 2048 to 16384, producing 44 valid cells under a controlled bf16 SDPA setup. The achieved fraction of peak HBM bandwidth falls as peak bandwidth rises. On the headline Qwen-2.5-7B ctx=2048 cell, an L4 reaches roughly 81 percent of its analytic memory floor, while an H100 reaches only 27 percent. Physical-AI decode is memory-dominated, but faster memory does not translate into proportional latency gains. We test the missing term with a CUDA Graphs A/B experiment. On H100 at ctx=2048, CUDA Graphs improves decode latency by 1.259x across N=10 fresh sessions, with a 95 percent bootstrap confidence interval of 1.253 to 1.267. On L4, the same intervention gives only 1.028x. This isolates a launch-side overhead that becomes visible on fast GPUs but remains mostly hidden on slower, bandwidth-bound GPUs. The deployment implication is that memory savings matter only when the runtime realises them. On L4, bf16 decode sits close to the memory floor, but common quantised paths do not recover the expected 4x weight-traffic reduction: bnb-nf4 reaches 59.36 ms/step and AutoAWQ+Marlin reaches 45.24 ms/step from a 62.32 ms bf16 baseline. GPTQ+ExLlamaV2, with Ada-tuned int4 kernels, reaches 17.36 ms/step.

1 Introduction

Physical-AI inference is dominated by single-stream, batch-1 autoregressive decode, but HBM bandwidth alone overstates the speedups available from faster GPUs. Measurements identify launch overhead and kernel implementation as decisive deployment factors, especially for fast H100 and quantised L4 inference.

  • Workload: Single-stream, batch-1 decode makes between-token latency the user-facing metric for robots, autonomous vehicles, embodied agents and edge assistants.Throughput-oriented batching is unavailable because each workload contains one human, robot or camera-feed session.
  • Bandwidth gap: The achieved fraction of peak HBM bandwidth falls from roughly 81% on L4 to 27% on H100 at Qwen-2.5-7B ctx=2048.This shows that the standard memory-bandwidth-bound account is incomplete on faster silicon.
  • Study scope: The study contributes a controlled 44-cell sweep across three 7–8B-class GQA architectures, four NVIDIA GPUs and four context lengths, plus backend and quantisation comparisons.Its claims concern the cross-GPU measurement, N=10 CUDA Graphs test, attention-kernel ranking, L4 quantisation result and inverted deployment ladder—not novelty of launch-bound regimes themselves.
  • Launch overhead: 1.259×: CUDA Graphs improves H100 ctx=2048 decode latency, versus 1.028× on L4 under the same intervention.The H100 result uses N=10 independent sessions and has a 95% bootstrap CI of [1.253, 1.267], supporting a CPU launch-overhead mechanism.
  • Attention backends: 36.05 us/layer: PyTorch’s default SDPA dispatcher beats explicit FLASH ATTENTION context at 44.35 us, FlashInfer at 48.20 us and FlashAttention-3 at 79.25 us on H100.At this batch-1 single-decode shape, kernel choice among explicit attention backends is not the binding constraint.
  • L4 quantisation: 17.36 ms: GPTQ+ExLlamaV2 reduces Qwen-2.5-7B L4 step time from 62.32 ms bf16, while AutoAWQ+Marlin reaches 45.24 ms.The comparison exposes an Ada SM89 kernel-implementation gap rather than a simple weight-traffic reduction.

2 Related work

Prior work frames autoregressive decode through bandwidth and IO-aware attention, while this section emphasizes the missing role of CPU-side kernel launches. The paper combines established launch and quantization techniques with a controlled cross-GPU measurement sweep and CUDA Graphs falsification test.

  • Roofline and arithmetic intensity: Roofline analysis diagnoses memory- versus compute-bound kernels, but omits CPU-side per-kernel launch overhead.Transformer inference work applies roofline reasoning to memory-bound autoregressive decode; this paper tests the omitted launch term.
  • Attention kernels: At batch 1 single-token decode, attention KV traffic is modest at ctx≤8192, while H100 is constrained by launching many small transformer-layer kernels.FlashAttention and FlashAttention-2 primarily target IO-aware attention gains in prefill and long-context training.
  • FlashDecoding++: FlashDecoding++ mitigates launch-sensitive batch-1 decode with asynchronous softmax, flat-GEMM double buffering, and adaptive dataflow, reporting up to 4.86× speedup over Hugging Face baselines and an average 1.37× over Flash-Decoding.It is identified as the closest prior work for batch-1 decode where per-kernel launch overhead matters.
  • CUDA Graphs: CUDA Graphs replace per-kernel CPU launches with a single replay of a captured DAG and serve here as a measurement instrument for bounding launch tax.The experiment asks whether removing launch overhead changes step time across GPUs, rather than presenting CUDA Graphs as a deployment recommendation.
  • Quantisation: Quantization methods promise roughly 4× lower weight HBM traffic, but L4 measurements show far smaller step-time reductions because the deployment chain determines whether bandwidth savings materialize.The related work covers GPTQ, AWQ, Marlin, SmoothQuant, and bitsandbytes nf4 dequantization-plus-bf16-matmul.
  • What is new here: The paper contributes a controlled cross-GPU sweep and CUDA Graphs falsification test for 7–8B-class GQA bf16 batch-1 decode, without claiming the launch-bound observation itself is new.The scope is limited to Modal-hosted NVIDIA silicon.

3 Method

The study measures batch-1 bf16 decode across three GQA models, four NVIDIA GPUs, and four context lengths using a controlled timing protocol, then compares observed latency with an analytic HBM-bandwidth floor. A CUDA Graphs A/B experiment isolates CPU-side kernel-launch overhead while leaving GPU kernels unchanged.

  • Measurement protocol: Each cell uses bf16, fixed-length prefill, 5 warmup steps, and 30 measured single-token decode steps at batch 1, with median per-step wall time reported.PyTorch scaled-dot-product attention is the default implementation; FlashAttention-2 appears only in the controlled software-stack matrix.
  • Experimental matrix: 44 valid cells cover three GQA architectures, four NVIDIA GPUs, and four context lengths after four L4 out-of-memory failures.The four context lengths are 2048, 4096, 8192, and 16384; the failed cells are Qwen-2.5-7B and Llama-3.1-8B at longer L4 contexts.
  • Roofline model: The roofline floor divides per-step weights-plus-active-KV traffic by spec-sheet peak HBM bandwidth, while Rfloor is computed directly from observed median latency and that analytic floor.The active KV traffic is K(M, ctx) = 2 · nlayers · nkv heads · dhead · ctx · 2 bytes, and the weight footprints span 14.5–16.1 GB decimal.
  • CUDA Graphs experiment: CUDA Graphs captures and replays the per-decode-step kernel sequence using 5 warmup replays and 30 measured replays, isolating CPU-side launch overhead without changing kernels.The eager-minus-graphed difference bounds launch tax from above on each tested GPU.
  • Roofline model: 27% of peak is achieved by Qwen-2.5-7B on H100 at ctx=2048, from a 3.93 ms floor versus 14.83 ms observed latency.The example uses approximately 13.16 GB of HBM read per decode step and 3.35 TB/s H100 peak bandwidth.

4 Results: the main matrix

Across 44 valid batch-1 decode cells, the achieved fraction of the analytic HBM-bandwidth floor declines as peak bandwidth increases. L4 remains near bandwidth-bound, whereas H100 achieves only roughly 0.21–0.31 of the floor at short contexts, motivating the separate CUDA Graphs mechanism test.

  • Measurement matrix: 44 valid cells report median decode step time t_obs and the analytic floor t_floor = (W + K)/B_peak.Per-cell values appear in Appendix B, Table 9, with directly measured R_floor = t_floor/t_obs in Table 1.
  • Bandwidth scaling: As B_peak rises from L4 at 300 GB/s to H100 at 3350 GB/s, R_floor falls from roughly 0.7–0.8 to roughly 0.21–0.31 at short contexts.The HBM-bound prediction is within tens of percent on L4 but overshoots achievable speedup by a factor of three to four on H100.
  • GPU distributions: The four distributions are separated and ordered by B_peak: H100, A100-80GB, L40S, then L4, with overlap at long contexts.Mistral L40S at ctx=16384 overlaps A100, and L4 ctx=16384 overlaps L40S.
  • Interpretation and scope: The matrix is consistent with per-kernel CPU launch overhead that does not scale with B_peak, but R_floor is descriptive rather than a predictive model.The CUDA Graphs A/B provides the direct mechanism test and independent bound on the H100 launch tax; the sweep alone cannot provide that bound.

5 The CUDA Graphs A/B test

CUDA Graphs confirms that batch-1 decode on H100 includes a substantial launch-side overhead, while the same intervention has little effect on L4 because its step remains dominated by HBM-bound GPU compute. The measured H100 speedup is 1.259× with a tight bootstrap interval, versus 1.028× on L4.

  • Mechanism: The H100 positive result rules out a pure-HBM explanation on H100-class silicon, while the L4 null rules out a launch-tax explanation on L4-class silicon.The intervention changes launch behavior but not the kernels themselves, isolating the differing contribution of launch overhead across GPUs.
  • Headline measurement: 1.259×: CUDA Graphs reduced H100 ctx=2048 Qwen-2.5-7B batch=1 decode latency across N=10 fresh sessions, with a 95% bootstrap CI of [1.253, 1.267].The within-session A/B used eager followed by graphed execution, with 5 warmup and 30 measured single-token decode steps per arm.
  • Removed launch overhead: 3.05 ms: Graphs removed this per-step launch-side overhead on H100, reducing mean eager time from 14.83 ms to 11.78 ms.Graphs also remove framework dispatch, stream events, allocator variability, and dispatcher work, so the difference is not assigned to a per-kernel launch count.
  • L4 null: 1.028×: The same CUDA Graphs intervention on L4 ctx=2048 produced only a small speedup, with eager p50 64.48 ms and analytic floor 51.17 ms.The L4 cell has Rfloor = 0.81, indicating that removing CPU launch serialization helps little while the GPU waits on HBM.
  • Batching alternative: 1.110× to 1.036×: H100 batch=4 CUDA Graphs speedup shrank monotonically across contexts from 2048 to 16384.At ctx=2048, batching from b=1 to b=4 increased graphed latency from 11.78 ms to 14.75 ms while delivering 3.20× decode throughput.

6 Software stack matrix: SDPA backends, FlashAttention, FlashInfer

On H100 batch-1 decode, attention-kernel choice is not the binding constraint: default SDPA outperforms FlashAttention-2, FlashInfer, FlashAttention-3, and pinned fallbacks, while launch overhead remains substantial. The tested cuDNN backend rejects this GQA shape, and the mechanism behind default SDPA’s advantage remains unresolved.

  • Software-stack matrix: 1.259× at N=10 is the sdpa+eager Graphs speedup, versus 1.43× at N=1, while Rfloor rises from 0.268 to 0.383.The N=10 result supersedes the N=1 host-noise spread as the bounded eager estimate.
  • Backend-pinned SDPA test: CUDNN ATTENTION rejects the head-dim-128 GQA single-decode shape in all three sessions, while the other tested backends accept it.All four SDPA backend flags were enabled by default in PyTorch 2.8+cu126 on H100.
  • Backend-pinned SDPA test: 36.05 us/layer is default SDPA’s p50, beating FLASH ATTENTION at 44.35 us, FlashInfer at 48.20 us, FlashAttention-3 at 79.25 us, and math at 177.55 us.These backend-pinned results use the Llama-3-8B decode shape at ctx=2048 and bf16 on H100.
  • Backend-pinned SDPA test: ∼20% is default SDPA’s advantage over explicit FLASH ATTENTION, but the mechanism is not definitive; plausible explanations include fusion or shape-specialised kernel-selection heuristics.The measured gap is not explained by routing through cuDNN, which rejects the tested shape.
  • Mechanism implication: 3.05 ms of Graphs-removed launch overhead is comparable to the 4.53 ms span between the slowest and fastest attention backends.Attention totals range from 1.15 to 5.68 ms across 32 layers, while full-step decode is 14.83 ms eager and 11.78 ms graphed.

7 Quantisation baseline on L4

On L4, quantisation only delivers its expected speedup when the runtime’s kernels efficiently stream quantised weights on Ada silicon. ExLlamaV2 reaches 17.36 ms/step, while bnb-nf4 and AutoAWQ+Marlin remain much closer to the 62.32 ms bf16 baseline.

  • Quantisation results: 59.36 ms for bnb-nf4 and 45.24 ms for AutoAWQ+Marlin fall short of the expected 4× weight-traffic reduction from quantisation.The corresponding AutoAWQ speedup is 1.38× over bf16.
  • Quantisation results: 17.36 ms/step for ExLlamaV2 delivers a 3.59× speedup from the 62.32 ms bf16 baseline on L4.The EXL2 4.25bpw cell used Ada-tuned int4 GEMM kernels.
  • Deployment implication: Kernel implementation, rather than bit width alone, determines whether quantisation’s bandwidth savings reach the workload.Marlin was tuned for Ampere SM80, whereas ExLlamaV2’s int4 GEMMs were tuned for Ada SM89.
  • Deployment implication: Rfloor = 0.82 for bf16 shows that L4 decode is bandwidth-bound, but quantised execution reaches Rfloor = 0.754 only with ExLlamaV2.The AWQ cell reached Rfloor = 0.289 against the same 4×-reduced floor of 13.09 ms.
  • Cross-GPU implication: 17.36 ms on L4 versus 11.78 ms on H100 makes H100 roughly 1.47× faster, while L4 serves the workload at roughly 6× less $/token.The cost comparison uses published Modal rates as of May 2026 and excludes idle, networking, and storage.

8 Per-kernel evidence with torch.profiler

torch.profiler confirms the cross-GPU pattern but substantially distorts H100 absolute timings and bandwidth-utilisation estimates. Unprofiled results show lower realised HBM bandwidth and greater CPU launch/synchronisation overhead on faster GPUs, while the utilisation figures remain comparative rather than ground truth.

  • Profiler limitations: 3.2× profiler overhead raises H100 step time from 16.97 ms unprofiled to 54.38 ms profiled, while L4 changes only 1.01×, from 63.0 ms to 63.67 ms.Using profiled time as the denominator therefore underreports H100 bandwidth utilisation.
  • Cross-GPU evidence: 26.9% unprofiled bandwidth utilisation on H100 versus 79.4% on L4 leaves a 3× gap, rather than the 10× profiled-denominator gap.The cross-GPU ordering remains unchanged, but the magnitude shrinks after removing profiler distortion.
  • Cross-GPU evidence: 0.75 CPU-to-CUDA time on H100 falls to 0.35 on L40S and L4, indicating that CPU launch and synchronisation occupy more of each step on faster silicon.This column is unaffected by the denominator choice.
  • Interpretation: The utilisation estimates are not claimed as ground truth; they reproduce the cross-GPU ordering of Rfloor and CUDA Graphs A/B results.Their intended use is to support comparative ordering across GPUs, not absolute utilisation claims.

9 Limitations and threats to validity

The study’s conclusions are explicitly limited to a narrow model, attention, precision, decoding, batch-size, hardware, hosting, and software scope. Additional threats include cloud noise, profiler asymmetry, long-context attention compute, failed configurations, and unavailable FlashDecoding++ reproduction.

  • Explicit scope: The claims cover 7–8B GQA transformers with head dim 128, bf16 weights, single-token decode, and primarily batch size 1.The models are Qwen-2.5-7B, Mistral-7B, and Llama-3.1-8B; batch 4 is only a sensitivity check on H100 and L4.
  • Explicit scope: The evaluation uses only NVIDIA GPUs and Modal cloud hosting, with sdpa as the main attention implementation and FA2 only in a controlled H100 matrix.The hardware excludes AMD MI300, Blackwell, Jetson, and Apple, while hosting excludes bare-metal and other clouds.
  • Architecture coverage: Results do not claim cross-GPU generalisation beyond this workload, and may not transfer to MQA, full MHA, different head dimensions, or dissimilar architectures.The approximately 10-kernel-per-layer anchor was observed only on three structurally similar GQA implementations.
  • Batch size: At batch 4, roughly 4× more projection and attention-output work leaves launch count essentially unchanged, shrinking the launch-tax fraction and graphed speedup.This batch-size sensitivity was measured across the full context grid on H100 and L4.
  • Cloud host noise: Cloud noise is mitigated with a common container, 30-step medians, and N=10 H100 headline sessions, but cross-session variation remains wider.Within-session CV is 0.9% eager and 0.2% graphed, while v10, v11, and v14 cross-session times vary more widely.
  • Profiler instrumentation: Profiler overhead inflates H100 step time 3.2× but L4 step time 1.01×, so profiled denominators underreport H100 bandwidth utilisation.The headline unprofiled-denominator utilisation is H100 26.9% versus L4 79.4%, a 3× rather than 10× gap.
  • Attention-compute regime at long context on small GPUs: At ctx≥8192 on L4, attention QK and softmax compute become nontrivial relative to weight streaming, so tfloor under-counts work in long-context cells.The caveat specifically applies to Mistral-7B L4 cells at long context.
  • Failed deployment configurations: CUDA allocator failures excluded torch.compile and FlashAttention-2 plus CUDA Graphs cells, leaving the Section 6 software-stack matrix sparser than intended.FlashDecoding++ and FlashDecoding++Next were also unreproducible because neither has a public source release; FlashAttention-3 and FlashInfer were substituted.

10 Conclusion

For batch-1 physical-AI decode, the binding constraint shifts from CPU launch overhead on H100 to HBM bandwidth on L4. Kernel choice and CUDA Graphs determine achieved step time more directly than peak hardware metrics, with important deployment and generalisation limits.

  • Binding constraints: 1.259× speedup with 95% bootstrap CI [1.253, 1.267] identifies CPU launch overhead as binding on H100, versus a null 1.028× result on L4.Both results are for N=10, ctx=2048, and the same within-session CUDA Graphs intervention.
  • Binding constraints: CUDA Graphs is most valuable for H100 batch-1 short-context decode, while increasing batch size shrinks the launch-tax fraction and graphed speedup.The paper presents Graphs as evidence about the binding constraint, not as a universal optimisation recipe.
  • Deployment implication: 17.36 ms/step with GPTQ+ExLlamaV2 cuts L4 bf16 time from 62.32 ms, delivering 3.59× speedup; AutoAWQ+Marlin reaches 45.24 ms and 1.38×.On L4 Ada SM89, kernel choice matters substantially for this workload.
  • Deployment implication: 17.36 ms/step on L4 with ExLlamaV2 compares with 11.78 ms/step on H100 with Graphs, despite the L4 costing roughly 12× less per GPU-hour.The relevant deployment metric is achieved batch-1 step time after applying kernel and Graphs levers, not peak FLOPs or HBM bandwidth.
  • Limitations: Generalisation beyond batch 1, GQA with head dim=128, bf16, NVIDIA accelerators, current NVIDIA silicon, Modal hosting, and single-stream serving is out of scope.The paper explicitly makes no claim for these settings.

A Reproducibility · A.1 Replication discipline

The paper’s reproducibility framework fixes the software, hardware, measurement, artifact, and rerun paths, while separating independent rigs and validating CUDA Graphs with fresh-container replication. Per-session outputs, medians, and variance are preserved rather than hidden.

  • A Reproducibility: All cells ran on Modal using a Debian slim container with PyTorch 2.4, transformers 4.45.x, flash-attn 2.6.3, and autoawq 0.2.7.post3.The GPU set was H100, A100-80GB, L40S, and L4; results were persisted to the paper-b7-v11-results Modal volume.
  • A Reproducibility: The main matrices contain 14 cells each for Qwen-2.5-7B-Instruct, Mistral-7B-Instruct-v0.3, and Llama-3.1-8B-Instruct, produced by v10 or v11 sweep scripts.The scripts share a Debian slim/PyTorch 2.4 base and the same 5-warmup-plus-30-measured single-token autoregressive protocol.
  • A Reproducibility: Each measurement cell produces a JSON artifact containing architecture, GPU, context, latency, traffic, floor, layer, and source fields.CUDA Graphs artifacts additionally record eager and graphed p50 latency, while OFT and profile artifacts record their respective flags and utilization ratios.
  • A Reproducibility: Sweeps can be rerun with modal run <script>.py from paper-b7/modal-validate/v11/, after which analyze.py reconstructs the cell directory and summary outputs.The scripts write directly to the Modal volume before analysis pulls the contents back.
  • A Reproducibility: 30-step medians absorb host noise despite unpinned random seeds, while bimodal H100 warmup tails are attributed to clock or kernel-selector convergence.Some H100 cells show first-16-step timings near 20–21 ms and last-14-step timings near 17–18 ms.
  • A.1 Replication discipline: Per-session JSONs cover headline H100 ctx=2048 eager and graphed sessions, with cross-session variance reported in Section 5.The published sessions span the v10 main sweep, v11 OFT rig, and v11 CUDA Graphs A/B rig.
  • A.1 Replication discipline: The three H100 rigs are treated as independent sessions because they used identical model and decode code but different scripts, launch days, and warmup-state handling.They otherwise shared the container, 5-warmup-plus-30-measured protocol, SDPA attention, bf16 weights, H100 SKU, and Qwen-2.5-7B-Instruct checkpoint.
  • A.1 Replication discipline: N=3 fresh Modal containers per GPU formed the v12 gold-standard replication using StaticCache, SDPA, bf16, and the A/B rig’s 5-warmup-plus-30-measured protocol.Each container received a fresh GPU allocation and cold-reloaded the model; per-session medians, cross-session means, speedup range, and raw p50 values were retained.

B Per-cell observed step times and bandwidth floors

Across all 44 cells, observed decode step times are compared with analytic HBM floors, revealing that the floor ratio rises with peak bandwidth but falls as context grows. The context trend reflects KV growth outpacing launch overhead.

  • B Per-cell observed step times and bandwidth floors: R_floor = t_floor/t_obs is measured across all 44 cells, comparing observed median step time with the analytic HBM floor.t_obs is the median of 30 single-token decode steps after 5 warmup steps; t_floor = (W + K)/B_peak.
  • B Per-cell observed step times and bandwidth floors: R_floor increases monotonically with B_peak at fixed context length.The ratio is directly measured as t_floor/t_obs.
  • B Per-cell observed step times and bandwidth floors: R_floor decreases with context length at fixed GPU because the KV term grows faster than launch overhead.This is the pattern observed across the per-cell table.

C Failed experiments and missing cells · D Kernel count and replicate terminology

Section C documents four L4 OOM cells, four H100 allocator failures, and the lack of host access for Nsight Compute, while Section D clarifies kernel-count estimates and replicate terminology. The reported matrix therefore contains 44 valid cells, and headline stability is based on explicitly separated within-session and cross-session measurements.

  • C Failed experiments and missing cells: Four L4 cells OOM’d at long contexts, reducing the experiment matrix from 48 to 44 valid cells.The failures affected Mistral-7B and Llama-3.1-8B at ctx=8192 and ctx=16384.
  • C Failed experiments and missing cells: Four H100 software-stack cells failed with CUDA caching allocator errors and were excluded from Table 5.The failed configurations included torch.compile, CUDA Graphs, and FlashAttention-2 combinations.
  • C Failed experiments and missing cells: Two surviving H100 cells, sdpa+eager and sdpa+eager+Graphs, span the comparison of interest despite the excluded cells.The paper did not work around the crashes.
  • C Failed experiments and missing cells: Nsight Compute was unavailable in Modal because it requires host driver access, so torch.profiler and analytic byte counts replaced per-kernel utilisation measurements.The substitution preserves cross-GPU pattern evidence but loses ground-truth per-kernel HBM and SM utilisation.
  • D Kernel count and replicate terminology: ∼10 kernels per decoder block is a launch-count anchor derived from a fixed sequence with fusion, while observed per-block launches range from 8 to 12.The count depends on architecture and SDPA backend, with additional global per-step kernels.
  • D Kernel count and replicate terminology: 30 within-session decode steps follow 5 warmup steps, and their median is the cell’s reported tobs.Their CV measures per-step jitter within one warmed session and is small on the H100 ctx=2048 headline cell.
  • D Kernel count and replicate terminology: N=10 headline replication means ten independent cold-container runs, each yielding paired eager and graphed medians and a bootstrap 95% CI across paired speedups.N=3 denotes cross-session replicates, whereas N=30 denotes within-session steps.
Loading 2605.30571v1…