Hardware-Aware Features for CUTLASS Kernel Selection

1 Introduction

• CUTLASS GEMM offers over sixty thousand implementations whose best choice depends on interacting configuration, problem, datatype, and architecture factors.Empirical compile-and-benchmark autotuning must be repeated across target GPUs, datatypes, epilogues, and shapes, motivating selectors that rank candidates without execution.Source: §1 · p. 1
• Analytical selectors use manually designed rules, while learned autotuners often use raw dimensions and template parameters that leave hardware consequences to be inferred [19] [26] [16].Analytical methods are fast but brittle and architecture-specific; raw parameters are described as a poor representation of hardware effects.Source: §1 · p. 1
• 6.2% mean selection regret compares with 17% for nvMMH and 71.5% for random selection, with relative reductions of 64.2% and 91.3%, respectively.The selector scores the entire valid catalogue at inference after training ranking models on a 4.9M-kernel corpus.Source: §1 · p. 1
• Hardware-aware features expose static, mechanistic estimates of candidate behavior while models learn interactions from measured data rather than hand-written performance rules.The approach aims to retain architectural inductive bias without brittle rules, while avoiding the need for models to infer hardware behavior from raw parameters.Source: §1 · p. 1

2.1 GPU Execution Model

• A GEMM block moves tiles through shared memory into registers, balancing data reuse and arithmetic intensity against memory traffic, resource use, and SM residency.Registers, shared memory, thread count, and architectural limits constrain resident blocks; higher residency can help hide latency, while larger blocks can reduce it.Source: §2.1 · pp. 1–2

2.2 Hopper Execution Mechanisms

• Hopper TMA transfers tensors asynchronously and WGMMA uses four-warps-per-operation groups; warp-specialized pipelines overlap transfers and computation but consume shared memory as stages increase.Thread-block clusters can reduce memory traffic while imposing scheduling and residency constraints, so their value depends on shape, tiling, and resource needs.Source: §2.2 · p. 2

2.3 GEMM Tiling and CUTLASS

• Tile dimensions determine output tiling and reduction iterations while affecting waste, arithmetic intensity, traffic, and register and shared-memory use.CUTLASS composes a mainloop with an epilogue that may add scaling, bias, activation, residual operations, conversion, and stores; these choices interact with tile and pipeline resources.Source: §2.3 · p. 2
• Across 68 evaluated groups, best and worst valid kernels differed by 1,070× in throughput; random selection achieved 28.5% of the in-group oracle on average, and a static best configuration achieved 71%.Fewer than 2.4% of configurations were within 5% of the best on average; evaluating 17 shapes required 3.9 million GPU measurements.Source: §2.3 · p. 3

2.4 A Generated Kernel

• A representative generated kernel fixes BF16 operands with FP32 accumulation and combines a 128×128×64 tile, four pipeline stages, TMA warp-specialized schedules, and persistent scheduling.Generated sources share a structure; their search space varies configuration choices while preserving GEMM semantics.Source: §2.4 · pp. 3–4

2.5 Configuration Space

• The BF16 generator varies tile geometry, cluster shape, pipeline depth, mainloop and epilogue schedules, and tile scheduling; static validity filtering reduces 2,138,400 combinations to 242,971 configurations.Operand and accumulator types, architecture and operator tags, C/D layout, alignments, and epilogue tile sizing are fixed or defaulted; additional static predicates remove invalid or wasteful candidates.Source: §2.5 · p. 4, §2.6 · p. 4

3 From CUTLASS Candidates to Hardware-Aware Features

• Each candidate’s feature map uses only the problem description, configuration parameters, and hardware constants, enabling execution-free selection.The authors first profile kernels to decide which execution effects are worth representing.Source: §3 · p. 4

3.1.1 Profiling setup.

• The analysis profiles 408 BF16 GEMM kernels on GH200 across 17 shapes, pairing hardware counters with benchmark throughput and comparing candidates within fixed shape–layout groups.This within-group framing is intended to separate candidate quality from problem-size effects in counters that scale with GEMM work.Source: §3.1.1 · pp. 4–5

3.1.2 Global correlation is often a problem-size effect.

• Within fixed problem groups, dram_throughput_pct and tensor_active_pct correlate with candidate quality (mean ρg=+0.408 and +0.392); both are positive in 83.7% of computable groups.By contrast, sm_active_pct falls from global ρ=+0.910 to within-group ρg=−0.001, while l2_read_sectors and dram_read_bytes also lose their apparent ranking power.Source: §3.1.2 · p. 5
• Static proxies for data reuse, traffic, cache-relative working sets, feed–compute balance, pipeline fill, and amortization stand in for execution-dependent throughput and tensor-activity measurements.Barrier stalls and long-scoreboard stalls are execution symptoms; long-scoreboard correlation changes from global ρ=−0.679 to within-group ρg=+0.193.Source: §3.1.2 · p. 5, §3.1.3 · p. 5

3.1.3 Measured symptoms versus controllable causes.

• Pipeline depth correlates with fewer long-scoreboard stalls (mean within-group −0.357), but stage count alone correlates weakly with throughput, motivating features for staging consequences.The representation includes bytes per stage, shared-memory demand, residency, pipeline fill, and stage amortization rather than treating stall counters or stage count as complete performance signals.Source: §3.1.3 · pp. 5–6

3.1.4 Static Configuration Effects.

• Stage effects depend on per-stage storage, available K-iterations for startup amortization, and whether resource use reduces residency; T_K correlates with throughput at +0.257 while reduction-iteration count correlates at −0.260.T_K cancels in the tile-level arithmetic-intensity ratio, but changes reduction granularity, mainloop iterations, per-stage storage, and computation between synchronization points.Source: §3.1.4 · p. 6

3.1.5 Regime Dependence.

• T_K’s within-group correlation varies sharply by geometry: +0.549 for wide GEMMs, +0.116 for square GEMMs, and +0.021 for tall GEMMs, motivating features for both candidate choices and problem regime.Across small, medium, and large problems the correlations are +0.217, +0.307, and +0.245; the complete feature map retains pre-execution quantities tied to configuration or regime and a plausible execution mechanism.Source: §3.1.5 · p. 6

3.2 Feature Map

• For each problem, candidate, and target, the feature map produces a vector; the candidate-catalogue feature matrix is the selector’s sole input.Table 6 summarizes the selected feature groups.Source: §3.2 · p. 6, Table 6 · p. 7

3.2.1 Structural Core.

• The structural core identifies the problem and candidate, while architecture descriptors provide capacities and rates for derived features.Derived features expose consequences that raw stage count alone misses, including shared-memory fit and whether the reduction is deep enough to amortize pipeline fill.Source: §3.2.1 · pp. 6–7

3.2.2 Work Decomposition and Wave

• The work-decomposition features track output tiles and masked edge work, then estimate wave quantization from tile count and concurrently resident blocks.Wave efficiency indicates whether the final wave leaves a ragged tail; additional features record Stream-K use and cluster-grid fit.Source: §3.2.2 · p. 7

3.2.3 Memory Behavior.

• Memory features estimate traffic for the inputs and complete epilogue, tile-level reuse, and re-streaming when operand panels cannot remain resident in LLC.The tile-level reuse feature cancels T_K: T_K affects stage storage and pipeline depth, whereas T_M and T_N determine reuse from loaded operand rows or columns.Source: §3.2.3 · p. 7
• Separate register and shared-memory bounds let the features distinguish register-limited residency from residency constrained by shared-memory use.The shared-memory estimate accounts for stage buffers and schedule-dependent epilogue storage, including whether their storage overlaps.Source: §3.2.3 · pp. 7–8

3.2.5 Pipeline Balance.

• Pipeline-balance features compare estimated TMA producer and WGMMA consumer times, and represent whether pipeline depth suits the contraction and amortizes fill cost.They also expose the interaction between stage count and shared-memory pressure.Source: §3.2.5 · p. 8

• For an input configuration, the method lists the valid CUTLASS kernel catalogue and ranks its candidates with a machine-learning model to choose a kernel.Source: §4 · p. 8
• The section explains how the training data are collected.Source: §4 · p. 8
• The section also explains the model design.Source: §4 · p. 8

4.1 Constructing The Training Dataset.

• The training and held-out datasets serve different purposes: training covers many shapes with sampled candidates, while held-out groups sweep valid candidates exhaustively to establish an in-space oracle.The collection pipeline batches compilation, caches executables and buffers, stores results locally, and runs compilation alongside GPU benchmarking with worker recovery.Source: §4.1 · p. 8, §4.1.9 · p. 11

4.1.1 Shape Selection.

• The 593 base shapes combine log-uniform interior and boundary coverage with anchor-neighbour and roofline/cache-transition supplements, yielding 2,372 shape-layout groups.The grid spans dimensions from 32 to 16,384 in multiples of 32; the supplements target changes in shape dimensions, a GH200 roofline point, and the 50 MiB LLC-residency boundary.Source: §4.1.1 · p. 8

4.1.8 Summary.

• Training measures a budgeted subset per shape-layout group: 2,000 candidates by default, 1,000 for the largest shapes, and 3,000 for atypical or transition shapes.The collection uses a static efficiency score to rank candidates, then samples three quarters from the high-ranked region and one quarter across the rest; the final collection contains approximately 4.9 million measured kernels.Source: §4.1.2 · p. 8, §4.1.8 · p. 10

4.1.3 Sparsity of the High-Performance Tail.

• Only 2.4% of valid candidates are within 5% of the oracle on average, and the most selective groups have fewer than 0.33% within that margin.The sparse high-performance tail means even thousands of uniformly sampled candidates can miss configurations near the optimum.Source: §4.1.3 · p. 9

4.1.5 Sampling Efficiency.

• At 2,000 candidates per group, biased-diverse sampling cuts mean regret from 8.5% to 0.9% and raises within-5%-of-oracle coverage from 25.6% to 91.3%.This is an 89.4% regret reduction versus uniform sampling; static top-B reaches 0.6% regret but does not deliberately measure heuristic-poor configurations.Source: Figure 3 · p. 9, §4.1.5 · p. 9, Figure 4 · p. 10

4.1.3 Sparsity of the High-Performance Tail.

• At budgets of 1,000, 2,000, and 3,000, biased-diverse sampling yields 1.5%, 0.9%, and 0.5% mean regret, while uniform sampling yields 11.3%, 8.5%, and 7.1%.Static top-B reaches 0.6% regret at 2,000 candidates, but its deterministic ranking can exclude competitive kernels.Source: §4.1.5 · p. 9

4.1.6 Probability of Observing a Near-Optimal Kernel.

• At 3,000 candidates, biased-diverse sampling finds a kernel within 5% of the oracle in 97.3% of groups, versus 97.1% for static top-B and 35.3% for uniform sampling.At 2,000 candidates, the corresponding rates are 91.3%, 97.1%, and 25.6%.Source: Figure 4 · p. 10, §4.1.6 · p. 10

4.1.7 Effect of the Exploration Fraction.

• At a fixed budget of 2,000, increasing the heuristic-preferred fraction improves both regret and within-5% coverage, but pure exploitation would rarely measure low-throughput kernels.The 75%-exploitation policy gives 0.9% mean regret and 91.3% within-5% coverage while reserving one quarter of measurements for the rest of the catalogue.Source: §4.1.7 · p. 10

4.2 Evaluation Dataset

• The evaluation set contains 68 groups and 3,922,067 distinct kernel-shape benchmark tasks across 17 held-out shapes and four operand layouts.The layouts are TN, TT, NN, and NT.Source: §4.2 · p. 11
• For each held-out shape and layout, evaluation uses the full catalogue after static validity filtering and shape-aware plan guards, with no proxy pruning.Source: §4.2 · p. 11
• Unlike training’s proxy shortlist of approximately 2,000 configurations per problem, evaluation measures the candidates exhaustively.Source: §4.2 · p. 11

4.3.1 Problem and Candidate Sets.

• Each candidate receives features from problem geometry, configuration choices, and analytical hardware proxies; inference ranks candidates within each GEMM problem rather than predicting throughput directly.The evaluation catalogue is a near-exhaustive set of valid CUTLASS configurations for each problem.Source: §4.3.1 · p. 11, Table 9 · p. 11

4.3.3 Relevance grades (ranking supervision).

• Measured throughput is grouped by problem and converted into relevance grades, with listwise tree training concentrating non-zero relevance on the highest-throughput band or bands.Adjacent throughput values are grouped when their gap is smaller than twice pooled measurement uncertainty; the top-focused scale uses two bands by default.Source: §4.3.2 · p. 11, §4.3.3 · p. 11
• Model A is an XGBoost tree ranker trained with listwise or pairwise ranking objectives, with pointwise throughput regression included as an ablation.Only the ranking objectives align their training loss with the argmax selection decision.Source: §4.3.4 · pp. 11–12

4.3.5 Model B (feed-forward ranker).

• Model B is a feed-forward ranker trained on scaled numeric and one-hot categorical features, while the ridge baseline tests globally linear combinations of the same representation.The MLP also uses within-group pairwise ranking information, retaining pairs involving any of the top eight throughput values before subsampling.Source: §4.3.5 · p. 12, §4.3.6 · p. 12

4.3.9 Shape-Grouped Cross-Validation.

• Hyperparameters are selected by cross-validation on training groups, with folds grouped by base shape so all layouts of a shape stay together and held-out evaluation data is excluded.Search uses Optuna’s TPE, and the winning MLP configuration is revalidated offline before its final fit.Source: §4.3.8 · p. 12, §4.3.9 · pp. 12–13

4.3.10 Final Fit and Test Protocol.

• After fitting on all training groups, the models are evaluated through regret on 68 oracle groups, with results broken down by problem regime.Baselines include uniform random selection and a fixed configuration chosen by its mean normalized throughput when present in at least 80% of evaluation groups.Source: §4.3.10 · p. 13, Figure 5 · p. 13

4.3.11 Relation between Training Loss and Validation Metric.

• Training optimizes ranking surrogates, but validation and testing measure selection regret because deployment uses only the identity of the top-scored candidate.The stated objective prioritizes mis-orderings among top candidates over throughput-prediction errors on clearly suboptimal configurations.Source: §4.3.11 · p. 13

5 Evaluation

• Experiments use four GH200 GPUs and compare against nvMMH; other related baselines are unavailable for Hopper.The study compiles CUTLASS from commit 3476ddb7 using CUDA 13.1.Source: §5 · p. 13

5.1 Training

• Hardware-aware features improve held-out selection across objectives, with larger gains for XGBoost; validation NDCG does not perfectly predict held-out regret.MSE has the highest validation NDCG@10 for both model families, but MLP RankNet achieves the lowest held-out regret among MLP variants.Source: §5.1 · pp. 13–14, Figure 7 · p. 14

5.2 Evaluation on the Exhaustive Dataset

• 6.2% mean regret for hardware-aware MLP beats 17.3% top-1 nvMMH and 71.5% random; hardware-aware XGBoost reaches 6.4%.Hardware-aware features reduce regret from 7.4% to 6.2% for MLP and from 10.7% to 6.4% for XGBoost.Source: §5.2 · p. 14, Table 12 · p. 14

5.3 Model Size Ablations

• At the smallest MLP sizes, structural models incur up to 14% higher regret; only the largest MLP matches the hardware-aware model, while XGBoost retains an advantage at every tested depth.The largest MLP has approximately 1.7 million parameters.Source: §5.3 · pp. 15–16, Figure 14 · p. 16

5.4 Generalization and Broad GEMM Evaluation

• 42.9 Tflop/s geometric-mean throughput for hardware-aware MLP (1.14× nvMMH) and 42.4 Tflop/s for XGBoost (1.13×) across 8,000 problems.Both learned selectors outperform nvMMH from memory-bound through compute-bound GEMMs; all methods achieve 100% execution coverage.Source: §5.4 · p. 16, Table 13 · p. 16

5.5 Transfer Learning Experiments

• At 5% target-shape coverage, hardware-aware transfer reaches 1.20×–1.22× speedup on FP32 and 1.12× on FP8 E4M3; FP8 MLP reaches 1.19× with full data.FP8 transfer is noisier, and structural models generally trail hardware-aware models in the low-data regime.Source: §5.5 · pp. 16–17

5.5.1 Transfer Learning Details.

• With only 10% of the fusion corpus, hardware-aware MLP and XGBoost match nvMMH at 1.00× geometric-mean speedup, outperforming the structural ablation (0.93×); full-coverage FP32 transfer reaches 1.24× for both.Source: §5.5 · p. 17, §5.5.1 · p. 18

5.6 DeepBench Case Study

• 12%–13% higher geometric-mean throughput than nvMMH, with 85%–96% wins across 151 DeepBench problems; inference-server speedups reach 1.24×–1.29×.The structural MLP matches the full model at 1.13×; ridge and random baselines remain below nvMMH.Source: §5.6 · p. 18, Table 15 · p. 18, Table 16 · p. 18

6 Related Work

• Learned cost models guide iterative search in tensor-program spaces, while this work scores the entire valid CUTLASS catalogue in one batch for direct selection.The cited frameworks include AutoTVM and Ansor; the model uses mechanism-level hardware features as an inductive bias for transfer.Source: §6 · p. 19
• Predictive models learn GEMM execution cost from configuration descriptors [11], whereas manual analytical models and heuristics encode hardware-mapping rules [9] [15] [19] [27].The proposed model instead learns complex interactions from analytical hardware-behavior features.Source: §6 · p. 19
• Stream-K and FlashAttention-3 illustrate GPU efficiency through work partitioning and pipeline coordination, mechanisms this work models as compile-time features.Source: §6 · p. 19

7 Conclusion

• 6.2% mean regret with hardware-aware MLP and 6.4% with XGBoost compares with 17.3% for top-1 nvMMH; both model families benefit from the representation.Source: §7 · p. 19
• Hardware-aware representations combine architectural mechanisms with data-learned interactions, rather than relying only on raw parameters or brittle analytical rules.Source: §7 · p. 19
• The authors suggest behavioral representations may provide a reusable basis across precisions, operators, and architectures as accelerator configuration spaces grow.Source: §7 · p. 19

AI Acknowledgment

• Generative AI tools assisted with benchmarking and analysis code, intermediate-result interpretation, and manuscript language editing.Source: AI Acknowledgment · p. 19
• The authors state that AI tools did not generate synthetic experimental data, formulate or prove mathematical claims, or fabricate measurements or results.Source: AI Acknowledgment · p. 19
• Authors reviewed AI-assisted work, validated code against compilation, tests, and measured data, and take responsibility for the final content and conclusions.Source: AI Acknowledgment · p. 19