Source-linked AI summary
AsmEvo: Agentic Assembly-Level Optimization of AMD GPU Kernels with Functional Equivalence Verification
Ji Liu, Puyuan Yang, Rongzhang Zheng, Fan Wang, Jinglin Wang, Muhammad A. Awad, Mortis Huang, Andy Chang, Zekai Li, Zeping Li, Zihao An, Yue Liu, Yuchen Yang, Jianghui Wang, Chushi Chen, Ziqiong Liu, Fuwei Yang, Dong Li, Wen Heng Chung, Shengcai Liu, Emad Barsoum
TL;DR
AsmEvo tackles optimization of compiled AMDGPU kernels when source and independent references are unavailable, using the deployed binary as the behavioral oracle. It recovers and edits assembly, rebuilds ABI-preserving objects, and externally verifies candidates before timing; it improves benchmark and production kernels while preserving functional equivalence.
Problem
Deployed AMDGPU code objects may lack editable source and independent references, leaving the original binary as the only behavioral oracle for optimization.
Method
AsmEvo reconstructs reassemblable AMDGCN, uses an agentic search driver for low-level edits, rebuilds ABI-preserving objects, and admits candidates only after differential verification and performance measurement.
Results
29 of 30 selected KernelBench kernels improve on MI308X, with 1.35x geometric-mean and 3.88x maximum speedup, while evaluated production kernels also improve with functional equivalence preserved.
Takeaways & Limitations
Deployed AMDGPU binaries retain recoverable low-level headroom that agentic search can exploit safely under deterministic correctness and timing gates.
Takeaways & Limitations
Equivalence is empirical and limited to evaluated inputs and launch configurations; the current recovery and rebuild implementation is calibrated for CDNA-class AMD GPUs.
Abstract
from arXiv · showhide
High-performance ML systems increasingly rely on GPU kernels whose editable source is unavailable, generated, or too distant from final machine code to expose remaining optimizations. Existing LLM kernel optimizers and autotuners mainly operate on CUDA, Triton, HIP, or tensor-program source and validate against reference implementations. We study a stricter setting: optimizing an already compiled AMDGPU code object, where the deployed binary is the only behavioral oracle. We present AsmEvo, an agentic assembly-level optimizer for AMD GPU kernels. Given an AMDGPU code object K0, AsmEvo reconstructs a reassemblable representation, proposes low-level edits with a long-horizon agent, rebuilds an ABI-preserving optimized object, and accepts candidates only after differential verification against K0 under identical launches. AsmEvo combines code-object recovery, metadata-aware rebuilding, profiling-guided hot-window editing, correctness-gated timing, and conservative in-place patch fallback. We conduct extensive experiments with AsmEvo on various AMD GPU kernels. On MI308X, AsmEvo improves 29 of 30 selected KernelBench kernels, reaching 1.35x geometric-mean and 3.88x maximum speedup. On MI300X production workloads, it improves all evaluated AITer binaries and vLLM/SGLang Triton assembly kernels, reaching 1.09x/1.31x and 1.18x/1.34x geometric-mean/maximum speedups, respectively, while preserving functional equivalence.
Introduction
AsmEvo addresses source-free optimization of deployed AMDGPU code objects, where low-level opportunities remain after compilation but source and independent references may be unavailable. It reconstructs, edits, rebuilds, and differentially verifies candidates before timing, with gains across benchmark and production kernels.
- Motivation: Compiled AMDGPU objects can retain optimization opportunities after high-level compilation, while source artifacts may be unavailable in deployment.Low-level decisions include wait-counter placement, instruction selection, register allocation, descriptors, and ABI-visible metadata.
- Motivation: Existing kernel optimizers generally assume editable CUDA, Triton, HIP, or tensor-program source plus an independent reference implementation.Compiled code-object kernels instead provide the deployed binary as the only behavioral oracle.
- Approach: AsmEvo reconstructs reassemblable AMDGCN, exposes profiled hot instruction windows, preserves the launcher ABI during rebuilding, and verifies candidates against the original binary under identical launches.Candidates are rejected before timing when differential verification fails.
- Approach: A long-horizon agent proposes architecture-specific edits and composes verified improvements, while deterministic gates control correctness, timing, resource checks, and commits.Failed attempts become search memory and stalled searches are redirected.
- Results: 29 of 30 selected KernelBench kernels improve on MI308X, reaching 1.35x geometric-mean and 3.88x maximum speedup.The experiments also cover AITer and vLLM/SGLang Triton assembly kernels, preserving functional equivalence.
Related Work
Prior work optimizes source, IR, or loaded binaries for generation, scheduling, instrumentation, or verification, but generally does not target source-free AMDGPU code objects with the original binary as oracle. AsmEvo positions itself as a post-compilation complement that combines agentic search, ABI- and metadata-preserving rebuilding, and differential verification.
- Source-Level Optimization: LLM kernel systems and tensor compilers refine CUDA, Triton, HIP, or tensor-program artifacts before or during compilation using high-level references.These approaches include execution-guided refinement, reinforcement learning, agentic search, DSLs, and schedule optimization.
- Assembly Optimization: Assembly superoptimizers search instruction or IR spaces, while learning-based systems have primarily demonstrated low-level optimization on CPU assembly.Examples include STOKE, Souper, AlphaDev, and reinforcement-learning approaches to CPU assembly.
- GPU Binary Frameworks: GPU binary frameworks such as NVBit and Luthier primarily support instrumentation and analysis rather than faster, verified drop-in objects.AsmEvo additionally preserves launcher semantics, resource declarations, descriptors, and metadata.
- Correctness: Translation-validation systems such as Alive2 check compiler transformations, but GPU optimizers commonly depend on source-level or framework references unavailable for HSACO-only artifacts.This motivates execution-based verification against the deployed binary.
- Positioning: AsmEvo is positioned as the first system combining agentic search, ABI- and metadata-preserving AMDGPU rebuilding, and differential verification against the original deployed binary.Search drivers are compared inside a fixed recovery, rebuilding, verification, and commit-gate harness.
Method
AsmEvo optimizes compiled AMDGPU objects without source or an independent reference by searching recovered assembly under deterministic correctness and performance gates. It preserves ABI-visible behavior and accepts only candidates that match the original oracle on evaluated launches and improve measured latency.
- Overview: AsmEvo takes a compiled AMDGPU code object K0 and produces an optimized object K′ without source code or an independent reference implementation.Recovered assembly s(K) is edited by a modular long-horizon driver that replans from structured feedback.
- Requirements: K′ must preserve symbols, kernarg layout, launch semantics, and externally visible metadata so the original host launcher can invoke it unchanged.These requirements define ABI preservation in the optimization problem.
- Requirements: For every evaluated input x, K′ is compared with the original-binary oracle O(x)=run(K0,x), including exact integer-buffer behavior and intact out-of-bounds guards.The supplied formulation distinguishes evaluated-input equivalence from unrestricted formal equivalence.
- Evaluation: Reported speedup is ŝ(K′)=T(K0)/T(K′), and commits are judged against the current verified best while performance is measured relative to the original object.Correctness-critical operations remain under deterministic controller ownership.
- Limitations: Equivalence is empirical rather than formal and applies only to evaluated inputs and launch configurations.Additional shapes require new inferred inputs or dispatch captures, and the implementation is calibrated for CDNA-class AMD GPUs.
Code-Object Recovery and Round-Trip Fidelity
AsmEvo reconstructs a rebuildable AMDGPU assembly representation rather than relying on raw instruction disassembly alone. A round-trip gate checks that reassembly and relinking preserve the instruction body, descriptors, and metadata before edits proceed.
- Recovery: AsmEvo recovers the AMDGPU ELF sections, symbols, notes, kernel descriptors, metadata, and AMDGCN instruction body needed for editable assembly.It also reconstructs .amdhsa_kernel declarations omitted by raw disassembly.
- Recovery: PC-relative control flow is re-symbolized so branches remain valid after instruction insertion, deletion, or reordering.This supports subsequent assembly edits without invalidating branch targets.
- Round-Trip Fidelity: A one-time round-trip gate reassembles and relinks the recovered representation, then compares the regenerated instruction body, descriptors, and metadata with K0.Only linker-determined fields are masked, and rare repairs must pass the same byte-level check.
Input Acquisition and Differential Oracle
AsmEvo acquires behavioral oracles for compiled AMDGPU kernels through synthetic inference or real-dispatch capture, then rebuilds candidates while preserving launcher compatibility.
- Differential Oracle: AsmEvo uses synthetic inputs when metadata exposes launch structure and real-dispatch replay for pointer-rich or application-specific kernels.Synthetic inference covers argument roles and representative inputs; real capture records launch parameters and referenced memory state.
- Differential Oracle: Real-dispatch replay restores identical launch parameters, memory contents, and pointer addresses before comparing candidate outputs and state.Floating-point outputs use tolerance-aware checks, while integer buffers, opaque state, and guard regions use byte-exact checks.
- Metadata-Aware Rebuild: AsmEvo rescans edited assembly and recomputes resource metadata while freezing the kernarg layout, symbol interface, and launch semantics.This produces a loadable object compatible with the original launcher.
- Metadata-Aware Rebuild: A conservative in-place byte patch is used only when rebuilding fails and edits are resource-neutral and size-non-increasing.Patched candidates still undergo static consistency, equivalence, and performance gates.
Profiling and Hot-Window Localization
AsmEvo narrows assembly-level search to profiled hot instruction windows, then restores each local edit to the full kernel before acceptance.
- Profiling and Hot-Window Localization: Hardware counters, instruction sampling, and static analysis identify stall-dominant windows and summarize dependencies, memory behavior, occupancy, and resource pressure.The search context includes each hot window’s surrounding dependencies and relevant metadata.
- Profiling and Hot-Window Localization: A local edit is spliced into the complete assembly for whole-kernel rebuilding and verification, reducing context without weakening acceptance criteria.Hot-window localization also discourages unrelated modifications.
Gated Verification Harness
AsmEvo’s evaluation harness validates assembly, static consistency, functional equivalence, and performance in that order, returning structured failures to the search process.
- Gated Evaluation: Each candidate first undergoes setup checks for roundtrip fidelity and oracle acquisition, followed by assembly, resource, ABI, equivalence, and performance checks.Correctness strictly precedes timing, so broken but fast candidates receive no positive optimization signal.
- Gated Evaluation: Timing uses warmup followed by the median of R event-timed launches under identical device and launch conditions.The harness returns structured outcomes for assembly failure, metadata inconsistency, divergence, runtime failure, and insufficient speedup.
- Gated Evaluation: GatedEval rejects candidates that fail assembly or static consistency before comparing them against oracle inputs and measuring speedup.The reported speedup is T(K0)/T(c) after equivalence checks pass.
Long-Horizon Agentic Search
A deterministic controller coordinates a long-horizon LLM search that explores, composes, and redirects assembly edits while retaining only verified lineage states.
- Search Control: The controller owns build validation, resource checks, equivalence, timing, commit thresholds, lineage, and termination, while the LLM proposes edits and replans.Failed attempts become structured feedback for subsequent search decisions.
- Commit Gate: Candidates advance only through a variance-aware threshold with a minimum improvement margin and speedup floor.The threshold uses timing coefficient of variation to reduce noise-driven commits.
- Verified Lineage: Accepted candidates form a tree-structured lineage recording parents, speedups, changed windows, resources, and rationales.Workers can branch from the best state or return to earlier verified versions without violating the no-regress invariant.
- Multi-Start Exploration: Team mode assigns orthogonal directions to parallel workers and selects diverse verified starting points for independent evaluation.This supports exploration across latency hiding, dependency reduction, register pressure, memory access, and instruction simplification.
- Verified Composition: Compositions are rebuilt and verified incrementally, and committed only when they improve over both parent candidates.Otherwise, the best individual candidate is retained.
- Anti-Stall Supervision: When progress stalls, bounded memory records failures and repeated candidates so the controller can redirect, change bottlenecks, or backtrack.Workers reset to verified states after evaluation, preventing unverified edits from accumulating.
Scope and Limitations
AsmEvo’s verified search operates on an original object, a bounded optimization budget, directional edits, and GPU resources. Its equivalence guarantee is empirical and bounded by evaluated configurations, while the implementation currently targets CDNA-class AMD GPUs and may require real-dispatch capture.
- Limitations: AsmEvo provides empirical differential equivalence only for evaluated inputs and launch configurations, rather than formal verification.Additional shapes require new inferred inputs or dispatch captures.
- Limitations: The current recovery and metadata-aware rebuild implementation is calibrated for CDNA-class AMD GPUs, while complex application state may require real-dispatch capture.Achievable gains also depend on optimization headroom remaining in the compiled kernel.
- Verified Search: The algorithm requires the original object K0, a search budget B, edit directions M, and available GPUs G.
- Verified Search: AsmEvo initializes from the original object K0, acquires an oracle, searches within budget B, and evaluates edited candidates through gated verification.The search maintains candidate lineage and memory while integrating verified edits into the current best object.
Experiments
AsmEvo evaluates source-free optimization of compiled AMDGPU kernels using fixed verification and timing procedures across benchmark and production artifacts. It improves most KernelBench and all evaluated production kernels, with larger and more varied gains on some benchmark operators than on fused or production workloads.
- Evaluation protocol: Functional equivalence is checked before performance measurement, with repeated verification and median post-warmup launch timing.Candidates use a fixed verification harness and report speedups against the unmodified binary.
- Experimental setup: The evaluation covers 30 KernelBench kernels, four AITer binaries, and four Triton JIT HSACOs from vLLM and SGLang on MI308X and MI300X.KernelBench runs on MI308X; AITer and Triton artifacts run on MI300X.
- KernelBench results: 29 of 30 KernelBench kernels improve, reaching a 1.35× geometric-mean speedup and a 3.88× maximum speedup.Level 1 improves all 15 kernels, while Level 2 improves 14 of 15; only 5.1% of evaluated candidates become verified commits.
- KernelBench results: Level 1 shows broader gains, with a 1.31× median and six kernels above 1.5× across convolution, normalization, activation, and matrix multiplication.Representative gains include 3.88× for depthwise convolution, 3.82× for MinGPTNewGelu, and 2.63× for RMSNorm.
- KernelBench results: Level 2 is concentrated near parity, with a 1.10× median and only one kernel above 1.5×, although its best case reaches 2.55×.The results are consistent with less local post-compilation headroom in fused kernels while retaining substantial opportunities in individual artifacts.
- Production results: AsmEvo improves all four AITer and all four Triton production kernels, reaching 1.09× and 1.18× geometric-mean speedups, respectively.Maximum speedups are 1.31× for AITer and 1.34× for Triton; production headroom varies by upstream artifact and kernel.
Conclusion
AsmEvo extends GPU kernel optimization to source-free AMDGPU code objects by using the original binary as its behavioral oracle and requiring ABI-preserving functional-equivalence verification. It improves most selected KernelBench kernels and all eight evaluated production kernels.
- AsmEvo uses the original binary as the behavioral oracle and admits candidates only after ABI-preserving rebuild and functional-equivalence verification.
- AsmEvo improves 29 of 30 selected KernelBench kernels, reaching 1.35× geometric-mean and 3.88× maximum speedups.
- AsmEvo improves all eight production kernels from AITer, vLLM, and SGLang.