PaperlayerView exact revision on arXiv

Source-linked AI summary

Optimizing CUDA like a Human: Micro-Profiling Tools as Expert Surrogates for LLM-Based GPU Kernel Optimization

Jiading Gai, Shuai Zhang, Kaj Bostrom, Jin Huang, Vihang Patil, Haoyang Fang, Bernie Wang, Huzefa Rangwala, George Karypis

arXiv:2606.26453v2cs.LG

TL;DR

LLM-based GPU kernel optimization lacks systematic interpretation of hardware telemetry, leaving models to both diagnose bottlenecks and generate code. KernelPro separates these roles with semantic micro-profiling, filtered multi-level analysis, domain-adapted MCTS, and autonomous CuTe code search. It achieves state-of-the-art KernelBench results, improves over greedy search and raw-metric feedback, and reports preliminary matched-speed energy savings.

  • Problem

    Existing CUDA agents expose raw or lightly summarized profiling data to LLMs, conflating hardware diagnosis with code generation and limiting systematic, extensible optimization.

    Source: §1 · p. 2

  • Method

    KernelPro uses pluggable semantic micro-profiling tools, roofline-filtered ncu/SASS/nsys analysis, domain-adapted MCTS with search memory, and autonomous CUTLASS/CuTe source search.

    Source: §1 · p. 2, §2 · p. 4

  • Results

    2.42×/4.69×/5.30× geometric mean speedups on KernelBench Levels 1/2/3 surpass KernelBlaster on all levels.

    Source: §1 · p. 2

  • Takeaways & Limitations

    KernelPro makes LLM-based CUDA optimization more grounded, interpretable, and effective by replacing implicit profiler interpretation with explicit diagnostic workflows.

    Source: §6 · p. 19

    The energy evaluation is preliminary, and full energy assessment is left to future work; CUDA optimization also assumes the paper’s correctness criterion.

    Source: §5 · p. 14

Abstract

from arXiv · show

We present KernelPro, a closed-loop multi-agent system that automatically generates, profiles, and iteratively optimizes GPU kernel code by integrating large language model (LLM) code generation with hardware profiler feedback and pluggable bottleneck detection tools. KernelPro introduces four contributions: (1) a semantic feedback operator that encodes expert heuristics as pluggable micro-profiling tools, transforming raw hardware metrics into actionable natural language guidance; (2) a two-stage tool invocation architecture where roofline-based bottleneck classification filters which specialized analysis tools execute, combining kernel-level (ncu), instruction-level (SASS), and system-level (nsys) profiling; (3) a domain-adapted MCTS with progressive widening, asymmetric branching, log-reward calibration, dead-end pruning, and search memory for cross-iteration learning; and (4) direct CuTe source-level code generation via autonomous code search over the CUTLASS/CuTe codebase. On KernelBench, KernelPro achieves geometric mean speedups of 2.42x/4.69x/5.30x on Levels 1/2/3, establishing state-of-the-art performance across all difficulty levels. On VeOmni's expert-optimized MoE training kernels, KernelPro achieves 1.23x over hand-tuned Triton by generating a from-scratch raw-CUDA+CuTe Hopper WGMMA kernel. Ablation studies demonstrate that each design component independently and significantly improves optimization quality: micro-profiling tools (p < 0.0001 vs raw metrics), MCTS search (26% higher geometric mean vs greedy, p = 0.004), and proactive tool orchestration (23% improvement, p = 0.035). Finally, KernelPro is the first CUDA kernel coding agent to optimize energy efficiency beyond the speed-only focus of prior systems, demonstrating an 11.6% measured energy reduction at matched speed.

1 Introduction

KernelPro separates hardware-metric interpretation from CUDA code generation by encoding expert heuristics as pluggable semantic feedback tools and combining them with bottleneck-filtered profiling and domain-adapted search. It achieves state-of-the-art KernelBench speedups across all levels while also introducing energy-aware optimization.

Results

  • 2.42×/4.69×/5.30× geometric mean speedups on KernelBench Levels 1/2/3 surpass KernelBlaster on all levels.The evaluation uses the fixed 42-task KernelBench subset for controlled component ablations.Source: §1 · p. 2

Core contributions

  • KernelPro converts raw hardware metrics into actionable natural-language guidance through pluggable micro-profiling tools that encode expert diagnostic heuristics.Each tool can trigger on a bottleneck signature, analyze its root cause, and recommend a targeted optimization.Source: §1 · p. 2
  • Roofline classification selects relevant specialized tools across kernel-, instruction-, and system-level profiling, reducing prompt noise before code generation.The two-stage design combines kernel-level, instruction-level, and system-level analysis rather than forwarding undigested metrics alone.Source: §1 · pp. 2–3
  • Domain-adapted MCTS uses progressive widening, asymmetric branching, log-reward calibration, dead-end pruning, and search memory to explore multi-step optimization paths.Search memory carries findings across iterations, while the tree search balances exploration against exploitation.Source: §1 · pp. 2–3
  • Direct CuTe source-level generation searches the CUTLASS/CuTe codebase autonomously to compose raw CUDA+CuTe kernels from expert-style building blocks.This operates below high-level template instantiation by searching for layout algebras, copy atoms, and MMA atoms.Source: §1 · p. 3

Energy-aware optimization

  • 11.6% measured energy reduction at identical speed demonstrates a preliminary energy-aware extension beyond speed-only kernel optimization.The full energy evaluation remains future work.Source: §1 · p. 3

2 Related Work

Prior CUDA optimization agents either omit hardware profiling or expose raw metrics that leave the LLM as the sole interpreter. KernelPro instead combines programmatic semantic feedback, source-level CuTe generation, and domain-adapted tree search for complete compiled-kernel optimization.

Profiling feedback

  • Existing systems span timing-only methods, raw or filtered ncu metrics, and cuPilot’s bottleneck labels, but none programmatically translate counters into actionable directives.cuPilot classifies memory-bound versus compute-bound behavior without generating directives from the underlying counters.Source: §2 · p. 3
  • KernelPro’s micro-profiling tools translate hardware counters into natural-language optimization directives before the LLM acts, unlike systems that leave interpretation to the model.The approach outperforms both no-feedback and raw-metrics-only feedback in ablation results.Source: §2 · p. 3

CUTLASS and CuTe generation

  • KernelPro generates raw CUDA+CuTe source by searching CUTLASS/CuTe for layout algebras, copy atoms, and MMA atoms rather than selecting only predefined high-level template configurations.This matches the source-level workflow used by expert kernel engineers.Source: §2 · p. 3

Tree search

  • KernelPro applies MCTS at solution level, with each node representing a complete compiled-and-profiled CUDA kernel rather than a reasoning step.Its domain-specific design includes progressive widening, log-reward calibration, asymmetric branching, dead-end pruning, and search memory.Source: §2 · pp. 3–4

3 The Architecture of KernelPro

KernelPro coordinates benchmarking, programming, profiling, and search agents in a closed loop that normalizes diverse inputs into CUDA and uses bottleneck-filtered semantic feedback to guide candidate kernels. Solution-level MCTS and search memory support exploration across compiled, validated implementations.

3.1 High-Level Pipeline

  • KernelPro’s three-agent pipeline benchmarks once, generates or repairs CUDA candidates, and profiles working kernels before feeding semantic guidance back into search.The Benchmarking Agent classifies bottlenecks, the Programmer Agent operates in initial, debugging, and optimization modes, and the Profiling Agent analyzes candidates.Source: §3.1 · p. 4, §3.2.1 · p. 4, §3.2.2 · p. 5
  • Proactive orchestration deterministically executes all tools relevant to the Stage 1 bottleneck classification, avoiding stochastic omissions by reactive LLM tool selection.The resulting guidance is injected into the prompt before LLM invocation.Source: §3.1 · p. 4

3.2.1 Benchmarking Agent

  • Stage 1 combines roofline analysis with reference profiling to classify kernels as compute-bound, memory-bound, or mixed and guide downstream tool selection.The Benchmarking Agent returns a ReferenceAnalysis object used by the Programmer and Profiling Agents.Source: §3.2.1 · p. 4

3.2 Programmer Agent

  • Inputs from PyTorch, Triton, and CUDA are normalized through a unified interface and routed with language-specific prompts to produce CUDA C++ output.This supports optimization workflows spanning prototyping, existing Triton kernels, and CUDA implementations.Source: §3.2.5 · p. 8

3.3 Search Strategy

  • KernelPro’s solution-level MCTS treats each node as a complete compiled-and-profiled kernel, exploring alternatives such as tiling, vectorization, and algorithmic restructuring.The approach concentrates computation on promising subtrees rather than making only greedy refinements.Source: §3.3 · p. 8
  • MCTS yields a 26% higher geometric mean speedup than greedy search across 42 KernelBench tasks, with the largest gains on multi-step optimization problems.The matched-budget ablation reports p = 0.004.Source: §3.3.4 · p. 10

3.4 Search Memory

  • Search memory addresses repeated errors, lost discoveries, and plateaus by preserving structured findings across iterations, producing an early-convergence trend without changing final speedup significantly.The memory is scoped to a single optimization run and stores extracted iteration findings outside the main loop.Source: Figure 1 · p. 5, §3.4 · p. 10

4 Micro-Profiling Tools

KernelPro encodes expert GPU-diagnostic workflows as pluggable micro-profiling tools that convert profiler data into actionable guidance. A roofline-filtered orchestration pipeline selects relevant analyses across runtime counters, compiled instructions, and system behavior.

Tool Design

  • 15 micro-profiling tools encode expert heuristics as composable analyzers that transform raw metrics into severity, root-cause, and ranked optimization guidance.The tools follow a trigger-analyze-recommend pattern and are organized across ncu, SASS, and nsys sources.Source: §4 · p. 11, §4.1.1 · p. 11

The 5-Tuple Formalization

  • The 5-tuple formalization specifies each tool’s name, required metrics, thresholds, trigger predicate, and analysis function.Its structured output can include severity, root cause, ranked recommendations, expected improvement estimates, and code examples.Source: §4.1.1 · p. 11

Proactive Tool Orchestration

  • Roofline-based bottleneck classification deterministically filters which profiling tools execute before their combined guidance reaches the LLM.This proactive orchestration avoids irrelevant analyses while ensuring bottleneck-relevant coverage.Source: §4.1.2 · p. 11
  • Proactive orchestration achieves 23% higher geometric mean speedup than reactive function calling (p = 0.035).The ablation supports deterministic execution of bottleneck-relevant tools over stochastic LLM tool selection.Source: §4.1.2 · p. 12

Warp Stall Analyzers

  • Warp-stall analysis identifies the dominant stall reason from five ncu metrics, triggering above 40% and escalating severity at 60%; barrier-specific analysis triggers at 30%.The analyzers translate stall percentages into cause-specific fixes such as async copies, warp-level primitives, or increased instruction-level parallelism.Source: §4.2.1 · pp. 12–13

Instruction-Level Analysis

  • SASS analyzers inspect emitted instructions to distinguish absent tensor-core code, generation mismatches, and runtime cuBLAS handling that ncu counters alone cannot resolve.On Hopper, finding HMMA without WGMMA prompts a switch to warpgroup MMA; register-spill detection combines SASS and ncu evidence to avoid flagging harmless spills.Source: §4.2.2 · p. 13

5 Results and Evaluation

KernelPro is evaluated against prior systems, through component ablations, on expert-optimized kernels, and in a preliminary energy study. It achieves broad benchmark gains, with profiling interpretation, search, orchestration, instruction-level diagnosis, and energy-aware objectives each contributing within their tested settings.

Which Tools Drive the Largest Gains?

  • The tensor-core switch averages 6.25× across nine compute-bound tasks, while register-spill guidance is the most reliable per-fire action and memory tools fire broadly but convert less often.The tensor-core action reaches 9.6× maximum gain; memory tools fire on 37–41 of 42 tasks but convert at 4–6%.Source: §5.3.2 · p. 16

Component Ablation

  • 26% higher geometric mean speedup from MCTS over greedy search reflects gains on multi-step landscapes where tree exploration escapes local optima.The largest reported per-task comparison reaches 10.0× the greedy speedup, including 25.0× versus 2.5× on one Level 1 task.Source: §5.3 · p. 15, §5.3.3 · p. 16

Production Validation: VeOmni MoE

  • KernelPro reaches 1.23× over an expert-tuned Triton baseline on H100 by generating a from-scratch raw-CUDA+CuTe Hopper kernel, with 16 other correct kernels also beating the baseline.Instruction-level and occupancy-level evidence identified zero tensor-core candidates, older HMMA regressions, register spills, and a higher-occupancy configuration.Source: §5.4 · pp. 17–18

Energy-Aware Kernel Optimization: A Preliminary Study

  • Energy-aware search targets reductions separable from latency, selecting a cheaper instruction stream when extra instructions remain hidden under bandwidth stalls.The study pairs speed-only and energy-aware arms and frames systematic characterization of fixed-speed energy savings as future work.Source: §5.5 · p. 18

6 Conclusion

KernelPro argues that the central bottleneck in LLM-based GPU kernel optimization is how profiling data reaches the model, not code-generation capability. Its explicit diagnostic workflow combines micro-profiling tools, MCTS, and proactive orchestration for stronger benchmark and production results.

Conclusion

  • KernelPro identifies profiling-data presentation, rather than code-generation capability, as the key bottleneck in LLM-based GPU kernel optimization.Micro-profiling tools convert hardware counters into actionable natural-language guidance instead of requiring implicit expert reasoning.Source: §6 · p. 19
  • KernelPro combines micro-profiling tools, domain-adapted MCTS, and proactive orchestration to improve optimization quality across benchmark and production settings.The reported headline outcomes are 2.42×/4.69×/5.30× on KernelBench and 1.23× over an expert-optimized Triton baseline.Source: §6 · p. 19

A Comparison with Concurrent Work

Comparison with concurrent GPU-kernel agents is informative but not directly uniform. Hardware, evaluated task subsets, and speedup metrics differ across systems, limiting direct interpretation of absolute ratios.

A Comparison with Concurrent Work

  • Cross-system speedup ratios are not directly comparable because GPU architectures, evaluated KernelBench tasks, and reported metrics differ.KernelPro and KernelBlaster use geometric means, whereas CudaForge uses arithmetic means and StitchCUDA reports end-to-end averages; StitchCUDA also evaluates on a trained-held-out split.Source: A Comparison with Concurrent Work · p. 22

B Micro-Profiling Tool Specifications

KernelPro’s micro-profiling tools convert hardware counters into bottleneck-specific diagnoses and recommendations, covering stalls, tensor-core use, register spills, and launch overhead. The tools share a pluggable interface and combine kernel, SASS, and system-level signals to guide optimization.

Warp stalls

  • WarpStallTool identifies the dominant warp-stall reason and dispatches a tailored optimization recommendation.It maps memory-dependency, barrier, long-scoreboard, and branch-resolving stalls to fixes such as caching, warp-level primitives, asynchronous prefetching, and predication.Source: §B.1 · pp. 22–23, §B.2 · pp. 25–26

Tensor-core analysis

  • WGMMAInstructionTool analyzes SASS to distinguish tensor-core instruction generations and guide architecture-specific optimization.It detects WGMMA, HMMA, IMMA, DMMA, and BMMA variants, while a fast symbol lookup checks for cuBLAS usage before full disassembly.Source: §B.1 · p. 23

Register spills

  • RegisterSpillDetector flags harmful spills only when SASS spill signals coincide with latency-bound, memory-bound, or occupancy-limited conditions.The detector counts STL and LDL instructions and uses throughput, local-traffic, register-limit, issue-activity, and eligible-warp thresholds to avoid treating every spill as harmful.Source: §B.1 · p. 24

Tensor-core utilization

  • TensorCoreUnderutilizationTool recommends cuBLAS or CUTLASS when compute-bound kernels use CUDA cores instead of tensor cores.Its prescribed action is to rewrite matrix multiplication as a cuBLAS/CUTLASS GEMM.Source: §B.1 · p. 24

Launch overhead

  • LaunchOverheadDetector identifies kernels dominated by launch overhead and recommends CUDA Graph capture or kernel fusion.It uses launch-overhead percentage, average launch time, and kernel-launch count as required metrics.Source: §B.1 · p. 24

Tool framework

  • The tools are extensible through a shared ProfilingTool interface that returns structured severity, root-cause, and ranked-recommendation outputs.New tools declare required metrics and thresholds, implement analyze(), and register with ToolRegistry so the orchestrator can invoke them automatically.Source: §B.2 · pp. 24–25

C.1 KernelBench Tasks

On a fixed KernelBench ablation, structured profiling guidance outperforms both no feedback and raw metric dumps, improving correctness, speedup, and optimization trajectories. The largest individual gains come from tensor-core switching and register-spill removal, while memory optimizations are common but harder to convert into wins.

Correctness and solve rate

  • KernelPro solves 100% of the 42 ablation tasks, exceeding No Feedback by 31% and Raw ncu Metrics by 17%.It also produces 7.5× versus 1.0×/1.0× on L2 task 40 and 5.8× versus 1.0×/1.0× on L2 task 41.Source: §C.1 · p. 27, Table 14 · p. 27

Profiling feedback

  • KernelPro’s interpretation layer is the key differentiator: the ordering is Ours > No Feedback > Raw ncu, with the full pipeline solving 100% of tasks.The Wilcoxon evaluation scores best per-task speedup, caps outliers, and assigns unsolved tasks 1.0×.Source: §C.1 · p. 28
  • Raw ncu Metrics performs significantly worse than No Feedback (p = 0.0007), showing that unstructured counters can distract the model.The comparison uses approximately 50 raw counters without interpretation, whereas KernelPro adds roofline classification and structured tool analysis.Source: §C.1 · pp. 27–28

Optimization trajectories

  • The tool-guided system avoids correct-but-slow plateaus across 62 trajectories, whereas No Feedback gets stuck at baseline performance in 10% of cases.Structured feedback supplies actionable bottleneck guidance after compilation recovery, enabling further optimization rather than merely preserving correctness.Source: §C.1 · p. 29

Per-tool impact

  • Tensor-core switching averages 6.25× across 9 tasks, while register-spill detection exceeds 1.5× in 18.2% of its 16 firings.The largest tensor-core trajectory rises from 4.11× to 11.49×, and eliminating spills takes L1_1 from 1.0× to 10.6×.Source: §C.2 · p. 29
  • MemoryCoalescing and Vectorization fire on 37–41 of 42 tasks but achieve only 4–6% hit rates.The passage attributes the lower conversion rate to difficult data-layout and alignment changes.Source: §C.2 · p. 29

Generalization beyond KernelBench

  • Across six Triton inference kernels, tools deliver 2.02× versus 1.49× geometric mean speedup in 5.0 versus 7.2 iterations.The fused_moe case fails without tools after 10 iterations but reaches 3.26× with tools in 2 iterations.Source: §C.3 · pp. 29–30, Table 18 · p. 30

D Search Memory Ablation

Search memory does not significantly improve final optimization quality under the tested budget, but it provides a marginal early-convergence advantage by preventing redundant exploration. Its benefit is therefore reaching good solutions faster rather than finding better ultimate solutions.

Final optimization quality

  • Search memory produces no significant difference in final optimization quality: z = 0.911, p = 0.181, n = 36.The geometric mean ON/OFF ratio is 1.06×, with a 90% confidence interval of [0.936, 1.200].Source: §D.1 · p. 30, Table 19 · p. 31

Early convergence

  • At 10 iterations, memory wins on 15/36 tasks versus 10/36 for control, with mean log(T/C) = +0.043 and p = 0.072.The advantage is strongest early and diminishes at deeper search depths.Source: §D.2 · p. 31, Table 20 · p. 31

Interpretation

  • Search memory prevents re-exploration of dead-end strategies during early iterations, improving convergence speed without changing final quality.Longer memoryless searches compensate for the lack of cross-iteration learning, so the benefit is most relevant under fixed iteration or time budgets.Source: D Search Memory Ablation · p. 30, §D.2 · p. 31

E Search Strategy Ablation: MCTS vs Greedy

Under matched compute budgets, MCTS outperforms greedy search overall, especially on tasks requiring exploration beyond local optima, while greedy retains advantages on simpler unimodal landscapes.

Experimental design

  • The ablation attributes KernelPro’s improvement to structured exploration via progressive widening, UCT selection, terminal pruning, and search memory rather than greedy refinement alone.Both conditions use matched compute budgets; the search strategy is the intended experimental variable.Source: E Search Strategy Ablation: MCTS vs… · pp. 31–32

Aggregate comparison

Threshold analysis

Task-level behavior

  • MCTS delivers its largest gains when greedy becomes trapped at correct but suboptimal solutions and alternative subtrees contain better optimization paths.Examples include 10.0×, 8.0×, and 6.0× MCTS-to-greedy ratios on representative Level 1–3 tasks.Source: E Search Strategy Ablation: MCTS vs… · p. 32
  • Task 46 required 43 consecutive failures before a valid solution, illustrating why persistent tree exploration can exceed budget-limited single-path search.Greedy wins on 8 tasks, particularly where the optimization landscape is unimodal and extra exploration wastes budget.Source: E Search Strategy Ablation: MCTS vs… · p. 32

F Tool Invocation Mode Ablation

Deterministic proactive orchestration of relevant profiling tools outperforms reactive tool selection, because it supplies the LLM with broader bottleneck-specific guidance before generation.

Invocation design

  • Proactive orchestration deterministically executes all tools relevant to the roofline-classified bottleneck, whereas reactive invocation leaves tool selection to stochastic LLM function calling.Both modes have access to the same 15-tool suite.Source: F Tool Invocation Mode Ablation · p. 32

Aggregate comparison

Threshold analysis

Mechanism

  • Reactive invocation calls only 3–4 tools per iteration on average, while proactive coverage can expose six ranked directives that compound into a 2.8× Task 41 improvement.The Task 41 trajectory illustrates the practical cost of surfacing only the highest-ranked analyses.Source: F Tool Invocation Mode Ablation · p. 33

Ablation synthesis

  • Together with interpreted tool feedback and MCTS, proactive orchestration independently and significantly improves optimization quality, with p < 0.05 for all three components.The broader ablation synthesis identifies three independently contributing design components.Source: F Tool Invocation Mode Ablation · p. 33

G.2 Level 3 / Task 46: NetVLAD with Ghost Clusters

Task 46 challenges KernelPro with a heterogeneous NetVLAD pipeline, where persistent search and profiling-guided memory-access fixes recover performance from an initially slower valid solution.

Problem

  • The target combines compute-bound matrix multiplications with memory-bound softmax, normalization, transpose, and slicing operations across a multi-stage ghost-cluster pipeline.This heterogeneous structure makes the task a demanding optimization target.Source: §G.2 · p. 36

Trajectory outcome

  • 2.79× final speedup: KernelPro improves the heterogeneous NetVLAD pipeline from a 0.75× first valid solution to 2.79×.The first valid solution appears at iteration 42, and the final result arrives at iteration 45.Source: §G.2 · p. 36

Search resilience

  • 43 failed attempts precede the first valid kernel, demonstrating resilient exploration through incorrect CUTLASS layouts, softmax numerical errors, and compilation failures.The trajectory contains 46 iterations but only 3 valid solutions.Source: §G.2 · p. 36

Profiling-guided recovery

  • Profiling identifies 17.86 sectors/request versus an ideal 4 and a 41.55% L1 hit rate, directing fixes for coalescing, shared-memory conflicts, and launch bounds.These recommendations guide recovery from a correct but slower-than-baseline implementation.Source: §G.2 · pp. 36–37

Implementation

  • The final implementation combines two layout-aware CUTLASS GEMMs with a hand-written fused batch-normalization, softmax, and slicing kernel using warp-shuffle reductions.The two GEMM configurations use 128×128×16 and 64×64×16 tile shapes.Source: §G.2 · p. 37

H VeOmni dW1 Generated Kernel (Full Source)

KernelPro generates a from-scratch CuTe Hopper WGMMA kernel for VeOmni’s ragged grouped GEMM, achieving 1.23× over an expert-tuned Triton baseline under realistic routing imbalance.

Workload

  • The dW1 kernel computes each expert’s first-FFN projection weight gradient from packed routed tokens, with per-expert token counts encoded by cumulative sums.The contraction dimension varies by expert, making the grouped GEMM ragged.Source: H VeOmni dW1 Generated Kernel (Full… · pp. 39–40

Benchmarking

  • Realistic benchmarking models heavily skewed routing using a Dirichlet–Multinomial distribution with α = 0.24, E = 128, and R = 57,768 routed rows.This reproduces hot experts, a warm shoulder, and empty experts rather than a uniform split.Source: H VeOmni dW1 Generated Kernel (Full… · p. 40

Result

  • 1.23× speedup over the expert-tuned Triton baseline: KernelPro’s generated dW1 kernel targets VeOmni’s MoE weight-gradient grouped GEMM.The kernel is generated at the CuTe source level rather than instantiated from a high-level CUTLASS GEMM template or prebuilt library.Source: H VeOmni dW1 Generated Kernel (Full… · p. 40

Kernel construction

  • The generated source composes copy atoms, a Hopper SM90 MMA atom, swizzled shared-memory layouts, asynchronous copies, and a vectorized epilogue.The implementation uses type-specific SM90 64×128×16 WGMMA atoms for half and bfloat16 inputs.Source: H VeOmni dW1 Generated Kernel (Full… · pp. 40–41
  • The kernel maps expert-specific offsets from cumsum_K into tiled CuTe tensors, stages operands in shared memory, performs WGMMA, and writes the accumulated result through a coalesced epilogue.The source explicitly constructs expert-local A, B, and C views before tiling and MMA partitioning.Source: H VeOmni dW1 Generated Kernel (Full… · pp. 40–41

I Energy-Aware Optimization: Methodology

KernelPro extends kernel optimization with energy measurement, a lexicographic speed-first reward, and profiling tools that target energy waste hidden by timing. Its protocol combines corrected NVML measurements for validation with a deterministic counter-based proxy for search, while acknowledging that some energy levers remain outside kernel generation.

Measurement protocol

  • NVML energy measurements correct for sensor duty-cycling and subtract idle power, with back-to-back arms reducing bias from drifting per-process baselines.On A100/H100, naive on-board sensing can differ from an external meter by an average of 35% and up to 65%.Source: §I.1 · pp. 42–43

Search and validation

  • During search, KernelPro ranks candidates with a deterministic ncu-counter energy proxy because device-wide NVML cannot isolate one SM or kernel inside a fused graph.The NVML protocol is reserved for post-hoc validation of winning kernels.Source: §I.1 · p. 43, §I.2 · p. 43

Energy-aware objective

  • Energy is a strict secondary objective: speedup always dominates, and energy distinguishes only candidates with equivalent speed.The extension activates through a single environment switch, while a passive mode records energy without influencing optimization.Source: §I.2 · p. 43

Energy-only profiling

  • Energy-only tools bypass speed-bottleneck filtering and fire when energy-wasting patterns are latency-hidden, routing findings to a subordinate prompt section.This targets energy reductions that timing does not expose.Source: §I.3 · p. 43, Table 25 · p. 43

Energy-aware objective

  • 11.6% measured energy reduction at identical speed was obtained in a preliminary matched-speed A/B study, while full energy evaluation remains future work.The reported reduction came from lower-energy instruction selection rather than the four energy-only tools.Source: §I.3 · p. 43

Energy bottlenecks

  • Reducing off-chip movement is the dominant kernel-level energy lever, whereas frequency/DVFS and occupancy are speed-independent levers outside the kernel-generation scope.The energy-aware objective therefore targets residual fixed-speed opportunities such as barriers, coalescing, and precision choices.Source: §I.4 · pp. 43–44, Table 26 · p. 44

Related papers

Loading 2606.26453v2…