PaperlayerView exact revision on arXiv

Source-linked AI summary

EdgeKernelBench: Unified LLM-Driven Kernel Generation for Edge Backends

Yuhua Zheng, Xueyi Chen, Xinze Zou, Xin Li, Haoji Hu, Huan Wang

arXiv:2610.24648v1cs.LG

TL;DR

Edge kernel generation must span fragmented backends and tight resource budgets, but its performance across edge workloads has not been studied systematically. EdgeKernelBench combines a unified IR with a hierarchical benchmark, and LEKA organizes optimization across IR levels. LEKA achieves higher Fast@1.0 than direct generation and a budget-matched Claude Code baseline on the agent evaluation subset.

  • Problem

    LLM kernel generation across diverse edge backends has not been studied systematically, despite heterogeneous hardware, fragmented software stacks, and tight resource budgets.

    Source: §1 · p. 1

  • Method

    EdgeKernelBench combines a unified IR for computation, scheduling, and memory choices with a hierarchical edge-workload benchmark, while LEKA organizes optimization across IR levels.

    Source: §1 · pp. 2–3

  • Results

    On the 69-instance agent subset, LEKA achieves 43.5% Fast@1.0 versus 13.0% for direct generation and 26.1% for Claude Code.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Across evaluated backends, LEKA outperforms the compiler baseline on more tasks than direct generation or a budget-matched general coding agent.

    Source: §6 · p. 9

    The evaluation covers CPU/LLVM, GPU/CUDA, and GPU/OpenCL, but not specialized accelerators such as NPUs or cooperative CPU–GPU execution.

    Source: §G.1 · p. 24

Abstract

from arXiv · show

Recent works show that LLMs can generate kernels with strong performance on server GPUs. Edge deployment presents distinct challenges because the hardware is heterogeneous, the software stacks are fragmented, and the resource budgets are tight. LLM kernel generation across diverse edge backends has yet to be studied systematically. We introduce EdgeKernelBench, which combines a unified IR deployment framework with a hierarchical benchmark for edge workloads. The unified IR captures reusable computation, scheduling, and memory decisions and connects to each target backend through a compiler interface. The benchmark spans progressively larger program scopes, from individual operators to compact model graphs, across representative edge application domains. We instantiate the design with TVM as a reference compiler and validate it across heterogeneous edge backends. With a single generation attempt, the best model achieves 64.5% Correct@1 but only 25.5% Fast@1.0. This gap shows that functional correctness does not consistently translate into performance gains. We therefore introduce the Layered Edge Kernel Agent (LEKA), which organizes optimization through a layered IR. On the agent evaluation subset, LEKA achieves 43.5% Fast@1.0, compared with 13.0% for direct generation and 26.1% for Claude Code. LEKA and Claude Code use the same base model and optimization budget. Together, EdgeKernelBench and LEKA support unified evaluation and layered optimization of kernels generated by LLMs across edge backends.

1 INTRODUCTION

LLM kernel generation for edge devices must contend with fragmented backends and resource constraints, while existing benchmarks leave important edge workloads underrepresented. EdgeKernelBench provides a shared IR framework and hierarchical workload benchmark; LEKA’s results show the potential of layered optimization beyond one-shot generation.

1 INTRODUCTION

  • Edge kernel generation spans fragmented hardware and software ecosystems, requires joint performance and memory optimization under fixed resource budgets, and is not fully represented by benchmarks focused on server GPUs or single operators.The cited mobile benchmark tasks mainly assess individual operators and attribute variants, not cross-operator optimization or graph-level resource planning.Source: §1 · pp. 1–2
  • EdgeKernelBench organizes PyTorch reference tasks by compositional complexity and application domain to evaluate optimization across edge deployment settings.The complexity levels range from basic compute patterns through composed operator graphs to compact model graphs.Source: §1 · p. 2
  • Across 650 task instances, the strongest model reaches 64.5% Correct@1 but only 25.5% Fast@1.0; Level 3 Fast@1.0 scores across three backends range from 0.0% to 4.0%.The paper reports that one-shot generation struggles to coordinate increasingly coupled decisions as program scope grows.Source: §1 · p. 2
  • 43.5% Fast@1.0 for LEKA, versus 13.0% for one-shot generation and 26.1% for Claude Code, on a 69-instance agent-evaluation subset.LEKA and Claude Code use the same base model, task contract, and wall-clock budget.Source: §1 · p. 2
  • EdgeKernelBench combines a unified IR deployment framework with a hierarchical benchmark, and systematic experiments assess models and LEKA across representative edge backends.The framework places computation, scheduling, and memory choices in a shared optimization space and uses a deterministic backend adapter to lower programs.Source: §1 · pp. 2–3

2 RELATED WORK

Prior work addresses kernel generation, compiler portability, and iterative optimization, but the paper identifies gaps in edge-focused task coverage, shared representations, and decision-level feedback. EdgeKernelBench and LEKA are positioned to address these gaps with a common IR and level-specific agents.

Kernel generation benchmarks

  • EdgeKernelBench retains the KernelBench protocol while targeting heterogeneous edge backends and edge-application tasks, spanning single operators to compact model graphs through one shared IR.Prior benchmarks are described as largely server-oriented; the cited mobile benchmark focuses mainly on single operators and their attribute variants.Source: §2 · p. 3

Compilers and portable kernel representations

  • EdgeKernelBench places computation, scheduling, and memory decisions in a unified IR that a deterministic adapter lowers to each backend.The passage contrasts this with hand-written backend kernels and tensor-compiler schedule spaces, which cannot reach transformations outside their encodings [Jiang et al., 2020; Chen et al., 2018; Zheng et al., 2020].Source: §2 · p. 3

LLM-driven kernel optimization

  • LEKA assigns an agent to each IR level and routes measurements to the level owning the decision, rather than scoring or profiling only a whole native program.The passage contrasts this with training-time specialization, profiler-feedback agents, evolutionary search, repair loops, and generate-test-refine methods [Baronio et al., 2026; Wei et al., 2025; Chen et al., 2026; Saba et al., 2026].Source: §2 · p. 3

3 EDGEKERNELBENCH

EdgeKernelBench pairs a benchmark organized around edge-workload scope and domain with a shared IR and a unified generation-and-evaluation workflow. Its framework separates reusable optimization choices from backend-specific lowering.

3.1 BENCHMARK CONSTRUCTION AND ORGANIZATION

  • Figure 2 depicts benchmark construction and organization: edge workloads become PyTorch tasks, the LLM expresses graph, scheduling, and memory choices, and candidates are built, checked, and timed.The task organization uses compositional complexity and application domain; evaluation checks against a PyTorch reference and times against a paired baseline.Source: §3.1 · pp. 3–4, Figure 2 · p. 4
  • The 331 canonical tasks span three complexity levels and three edge application domains, with every domain represented at each complexity level.The levels comprise 128 Basic Compute Patterns, 162 Composed Operator Graphs, and 41 Compact Model Graphs; domains include Mobile and Personal Intelligence, Smart AIoT, and Embodied Robotics.Source: §3.1 · p. 4

3.2 UNIFIED GENERATION AND EVALUATION FRAMEWORK

  • The LLM expresses computation, scheduling, and memory choices in Unified Kernel IR, which backend adapters lower into target-specific executable programs.The shared task and evaluation contract remains separate from each backend’s deployment implementation.Source: §3.2 · p. 4
  • The evaluation pipeline constructs candidates, validates functionality, and measures performance to characterize compilation success, correctness, and execution performance.Figure 3 shows LEKA’s layer agents following progressive IR lowering, with narrowing scope and hardware feedback guiding search at the relevant level.Source: §3.2 · p. 4, Figure 3 · p. 5

4 LAYERED EDGE KERNEL AGENT (LEKA)

LEKA decomposes kernel optimization across IR levels, narrowing each agent’s scope while retaining links among candidate states. Device measurements guide subsequent search at the level associated with the reported bottleneck.

4 LAYERED EDGE KERNEL AGENT (LEKA)

  • LEKA gives each IR level an agent and links its focused search to other levels through a shared candidate tree and hardware feedback.For each executable candidate, the hardware-sensing agent identifies a level for further optimization and reports measurements to that level’s agent.Source: §4 · p. 5, §4.2 · p. 6

4.1 HIERARCHICAL IR OPTIMIZATION

  • The candidate tree records program states and transitions across within-level optimization and cross-level lowering, while level- and device-specific projections extract relevant history and measurements.The hardware-sensing agent profiles candidates and returns detected bottlenecks in a report for the current level.Source: §4.1 · p. 5
  • Compiler lowering preserves semantics and prior optimizations while performing legalization and interface conversion; performance choices remain with the layer agents.The compiler supplies the conversion between IR levels but does not search for candidates.Source: §4.1 · p. 5
  • Progressive lowering fixes upper-level decisions while exposing finer search spaces, aligning each optimization with the IR level that expresses it most directly.The decomposition reduces context required per call and makes execution feedback easier to attribute to the relevant level.Source: §4.1 · pp. 5–6
  • LEKA searches backend-shared IR layers before target-specific lowering exposes hardware-coupled choices, retaining high-level optimizations while adapting implementation details.The passage states this separation allows high-level optimizations to transfer to new backends.Source: §4.1 · p. 6

4.2 CANDIDATE SEARCH GUIDED BY EVIDENCE

  • Each optimization attempt adds a candidate while retaining earlier candidates and validation results; the tree preserves alternatives for comparison, rollback, or continued search.Hardware measurements identify the level needing further optimization, and the selected layer agent generates the next candidate.Source: §4.2 · p. 6

5 EXPERIMENTS

Experiments evaluate EdgeKernelBench and LEKA across heterogeneous edge devices, tasks, and backends. LEKA outperforms one-shot generation and Claude Code on the agent subset, while one-shot performance declines as task scope grows.

5.1 EXPERIMENTAL SETUP

  • The evaluation uses TVM 0.23.01 across CPU/LLVM, GPU/CUDA, and GPU/OpenCL devices, measuring compilation, correctness, and performance.Tasks are stratified across complexity levels for LEKA evaluation, and LEKA and Claude Code share the same task contracts, evaluator, and time budgets.Source: §5.1 · p. 6

5.2.1 COMPILER-BASELINE RESULTS

  • Across backends, the highest F@1 falls from 51.0% at Level 1 to 16.2% at Level 2 and 4.0% at Level 3.Longer dependency chains and coordination of computation and storage across larger program contexts accompany the decline.Source: §5.2.1 · p. 7
  • Among proprietary models, Gemini has the highest Level 3 FCR on all three backends, while DeepSeek reaches 75.3% CPU Level 2 FCR but only 3.2% F@1.GPT’s corresponding F@1 is 15.1%, illustrating that correctness and execution performance do not necessarily align.Source: §5.2.1 · p. 7

5.2.2 PRODUCTION-INFERENCE-ENGINE RESULTS

  • Against MNN, generated programs outperform the engine most often at Level 1, where F@1 peaks at 50.7% on CPU and 31.6% on GPU.The maxima at Levels 2 and 3 fall to 14.9% and 5.3%, respectively.Source: §5.2.2 · p. 8, Figure 4 · p. 8

5.2.3 ERROR ANALYSIS

  • Memory and syntax/API errors account for 34.0% and 25.4% of pooled failures, respectively; most memory errors concern allocation legality rather than insufficient capacity.Correctness and parallel errors contribute another 18.9% and 12.0%.Source: §5.2.3 · p. 8, Figure 5 · p. 8

5.3.1 RESULTS AGAINST ONE-SHOT GENERATION AND CLAUDE CODE

  • 43.5% F@1 over TVM is achieved by LEKA, versus 13.0% for one-shot generation; against Claude Code, LEKA reaches 43.5% versus 26.1%.The agent comparison reports an overall GM of 3.8× relative to Claude Code.Source: §5.3.1 · pp. 8–9, Figure 6 · p. 9

5.3.2 OPTIMIZATION CASE STUDY

  • On SqueezeNet1.1, LEKA’s successive Relax, TIR, and OpenCL optimizations raise speedup from 1.3× to 1.7× and then 3.5× over the initial implementation.The Relax agent adopts NHWC and rewrites 1 × 1 convolutions as matrix multiplications; later agents apply shared-memory tiling and register blocking.Source: §5.3.2 · p. 9

6 CONCLUSION

EdgeKernelBench and LEKA provide a framework for studying how model capability, program abstraction, and optimization freedom relate across heterogeneous edge backends. The conclusion also identifies shifting optimization toward lower-level representations and designing agent-friendly IR boundaries as open questions.

6 CONCLUSION

  • LEKA outperforms the compiler baseline on more evaluated tasks than direct generation or a budget-matched general coding agent across the evaluated backends.The evaluation also finds that optimization performance varies with program complexity and target hardware.Source: §6 · p. 9
  • EdgeKernelBench combines a unified IR deployment framework with a hierarchical benchmark, while LEKA organizes optimization search across IR levels and uses device feedback to refine candidates.Together, they provide an experimental foundation for studying the relationship between model capability, program abstraction, and optimization freedom.Source: §6 · p. 9
  • The conclusion leaves open how to define layer boundaries and design IRs that help agents reason about optimization choices and constraints.It also asks whether increasing model capability could shift the optimization boundary toward lower-level representations, which offer more hardware control but bring more complex decisions.Source: §6 · p. 9

APPENDIX OVERVIEW

  • The appendix covers benchmark specification, devices and evaluation protocols, baseline construction, and the unified IR and deployment framework.It also includes failure analysis and case studies, prompt templates, and limitations and future work.Source: APPENDIX OVERVIEW · p. 13

A.1 BENCHMARK TASK COVERAGE AND PROVENANCE

The benchmark organizes PyTorch reference tasks by compositional complexity and edge application domain, with task templates and static shape profiles supporting reproducible evaluation. The appendix also specifies the tested models, backend setups, correctness checks, and timing-stability procedure.

A.1 BENCHMARK TASK COVERAGE AND PROVENANCE

  • 331 canonical tasks span 128 Basic Compute Patterns, 162 Composed Operator Graphs, and 41 Compact Model Graphs, organized across three edge application domains.Evaluation sets are instantiated for CPU/LLVM, GPU/CUDA, and GPU/OpenCL, with task counts varying by level and backend.Source: §A.1 · p. 13

A.2 TASK TEMPLATES AND SHAPE PROFILES

  • Each task pairs a PyTorch semantic template defining the reference computation and initialization interface with a static shape profile tailored to its application domain.The benchmark combines authored and adapted open-source tasks, normalizing them to static, self-contained interfaces and checking importability, completeness, and shape materialization.Source: §A.1 · p. 13, §A.2 · p. 14

A.3 EVALUATED LLMS AND INFERENCE SETTINGS

  • One-shot experiments evaluate five LLMs, while LEKA and Pure Claude Code use GLM-5.3-Flash for iterative optimization because of inference-budget constraints.All models use provider-default inference settings.Source: §A.3 · p. 14

B.1 SYSTEMS AND COMPILER CONFIGURATION

  • The evaluation uses CPU/LLVM on a Snapdragon 8 Gen 3 phone, GPU/CUDA on a Jetson AGX Orin, and GPU/OpenCL on a Firefly RK3588S-PC.TVM 0.23.0 connects generated programs to compilation and deployment across these paths.Source: §B.1 · pp. 14–15

B.2 CORRECTNESS AND TIMING PROTOCOL

  • Generated programs must match their PyTorch references in output structure, shapes, and types, with numerical differences within task-specific tolerances, before performance testing.Paired timings use the same device session, inputs, and timing boundary for candidate and baseline; compilation and deployment preparation are excluded.Source: §B.2 · p. 15

B.3 TIMING STABILITY AND PAIRED MEASUREMENT

  • The protocol requires rCV no greater than 5% for candidate latency, baseline latency, and paired speedup, using scaled median absolute deviation relative to the sample median.Paired measurements balance execution order to reduce device-state-drift bias, and rCV checks the repeatability of latencies and relative performance.Source: §B.3 · p. 15

B.4 AGENT EVALUATION TASKS AND BUDGETS

The appendix specifies a stratified, budget-matched agent comparison, calibration-based TVM search budgets, and inference-engine baselines. It also details how the unified IR expresses optimization choices and connects generated programs to backend deployment.

B.4 AGENT EVALUATION TASKS AND BUDGETS

  • The agent comparison samples 69 task instances across three complexity levels, and LEKA and Pure Claude Code receive identical per-task budgets of up to 90 minutes for Level 1 and 155 minutes for Levels 2 and 3.F@1 is reported against TVM, while GM compares Pure Claude Code latency with LEKA latency, including tasks where LEKA is slower.Source: §B.4 · p. 15

C.1 TVM COMPILER BASELINE

  • TVM’s baseline uses the lowest-latency valid MetaSchedule schedule found for each workload and target within a calibrated search budget.Calibration groups extracted PrimFuncs by computation pattern and search complexity, then derives budgets from convergence behavior; Figure 7 shows representative calibration curves.Source: §C.1 · p. 16

C.2 MNN INFERENCE-ENGINE BASELINE; C.3 TENSORRT INFERENCE-ENGINE BASELINE

  • The MNN and TensorRT references use exported PyTorch graphs and their native inference-engine optimizations, with evaluation restricted to supported tasks that pass correctness checks.MNN is evaluated on CPU/LLVM and GPU/OpenCL; TensorRT provides the CUDA inference-engine comparison.Source: §C.2 · p. 17, §C.3 · p. 18

D.1 UNIFIED IR AND OPTIMIZATION SCOPE

  • The Unified Kernel IR expresses programs at graph-and-invocation and computation-and-scheduling levels, coordinating dependencies, fusion, loops, layouts, memory accesses, and parallel mappings.Fusion, decomposition, and local scheduling expose memory use alongside execution latency as optimization choices under device constraints.Source: §D.1 · p. 18

D.2 GENERATION CONTRACT AND PROGRAM CONSTRUCTION

  • The PyTorch reference, input/output specifications, and numerical tolerances define observable behavior, while the LLM may reorganize computations, layouts, fusion boundaries, and schedules within that contract.The framework checks generated program structure and backend compatibility, then validates outputs against the reference.Source: §D.2 · p. 18

D.3 COMPILER INTEGRATION AND BACKEND DEPLOYMENT

  • Progressive compiler lowering resolves storage and execution details under backend constraints, after which deployment loads the artifact and binds its entry point to the runtime.The TVM implementation supports program construction and device execution through layered IR, code generation, and runtime support.Source: §D.3 · pp. 18–19

D.4 COMPILER INSTANTIATION AND HARDWARE ADAPTATION

  • A new compiler integration implements the unified generation interface with its IR and adapters for lowering, code generation, and runtime paths.Adding hardware supported by that compiler requires extending target configuration and deployment adapters while retaining task definitions and evaluation criteria.Source: §D.4 · p. 19

E.1 BACKEND- AND MODEL-STRATIFIED FAILURE ANALYSIS

One-shot failures vary across models and backends: Syntax/API, memory, and parallel-execution errors dominate different settings, while numerical correctness also remains a challenge.

E.1 BACKEND- AND MODEL-STRATIFIED FAILURE ANALYSIS

  • Failure categories distinguish invalid syntax or interfaces, memory and parallel-legality violations, shape/type errors, output mismatches, and other exceptions.The figure's subcolumns show each category's share of unsuccessful one-shot outcomes and sum to 100.0% before rounding.Source: §E.1 · p. 19, Figure 8 · p. 19
  • 54.0% of GPT-5.6-sol CPU/LLVM failures are Syntax/API, while 56.2% of Gemini-3.7-Flash CPU/LLVM failures are Memory.Figure 8 reports failure composition, not failure rates.Source: §E.1 · p. 19, Figure 8 · p. 19
  • Memory accounts for more than half of Gemini-3.7-Flash and Claude Sonnet 5 failures on both GPU backends.The authors identify memory management as a persistent limitation for these models.Source: §E.1 · p. 19
  • 29.7% of GPT-5.6-sol CUDA failures are Parallel, while 30.9% of its OpenCL failures are Syntax/API.DeepSeek-V4-Pro OpenCL failures also include 32.5% numerical-correctness mismatches.Source: §E.1 · p. 19, Figure 8 · p. 19

E.2.1 ONE-SHOT GENERATION CASE STUDIES

Two one-shot implementations illustrate distinct optimization strategies: selective fusion on OpenCL and algebraic rearrangement on CPU each outperform their stated baselines.

E.2.1 ONE-SHOT GENERATION CASE STUDIES

  • 30.9×: Gemini-3.7-Flash reduces OpenCL ConvNeXt block latency from 18.0 ms to 0.6 ms over TVM using selective fusion and intermediate storage.Separate kernels handle depthwise convolution and channel transformations; the second fuses normalization, linear transformations, activation, and residual connection.Source: §E.2.1 · p. 20
  • The OpenCL comparison shows that extensive fusion does not necessarily improve performance; separating stages with different access patterns can reduce synchronization and improve data reuse.GPT-5.6-sol's fully fused implementation reaches only 0.9× its corresponding TVM baseline.Source: §E.2.1 · p. 20
  • 19.0×: Gemini-3.7-Flash reduces CPU dequantization-and-matrix-multiplication latency from 39.7 ms with MNN to 2.1 ms.Moving zero-point correction and scaling after reduction avoids a 44 MiB floating-point intermediate for the 2048 × 5632 weight matrix.Source: §E.2.1 · p. 20

E.2.2 LEKA OPTIMIZATION CASE STUDIES

The case studies trace LEKA's layered refinements and the appendix describes the prompts and roles used for direct generation and agent-based optimization. The examples show performance gains from layer-specific changes coordinated through evaluation feedback.

E.2.2 LEKA OPTIMIZATION CASE STUDIES

  • 1.2×: LEKA reaches 16.2 µs on the CUDA task, versus 19.8 µs for tuned TVM, through graph, scheduling, and device-level refinements.L0 fusion and parameter preparation, L1 matrix and reduction scheduling, and L2 memory and synchronization changes successively lower search-stage latency from about 104.0 µs to 20.0 µs and then about 16.2 µs.Source: §E.2.2 · p. 20
  • 1.6×: LEKA reaches 17.0 ms on the CPU task, versus 27.5 ms for TVM, with L1 scheduling refinements to the convolution bottleneck.Convolution accounts for approximately 85.0% of the profiled interval after the first effective scheduling pass; the implementation preserves the L0 graph and call interfaces.Source: §E.2.2 · p. 21

F.1 ONE-SHOT GENERATION PROMPT PROTOCOL

  • The generation template supplies the optimization objective, device profile, one demonstration, and a target task with a PyTorch reference and tensor specifications.One-shot generation returns one candidate without iterative evaluation feedback, and the task permits changes to computation organization, fusion, layouts, and equivalent algorithms while preserving semantics.Source: §F.1 · pp. 21–22
  • One-shot candidates must be complete and interface-compatible, preserve reference outputs within numerical tolerances, and are evaluated for performance only when correct.Generation and evaluation run in isolated environments.Source: §F.1 · p. 22

F.2.1 PURE CLAUDE CODE BASELINE PROMPT

  • Agent optimization uses the same public reference and tensor specifications as direct generation, with iterative evaluation within an assigned budget.The pure Claude Code baseline shares LEKA's base model, task specification, evaluator, and wall-clock budget, and uses execution evidence to guide changes.Source: §F.2 · p. 22, §F.2.1 · p. 22

F.2.2 LEKA HIERARCHICAL AGENT PROMPTS

  • LEKA assigns editable IR representations to resident agents, while a read-only hardware-sensing agent turns measurements into layer-specific optimization directions.The roles cover L0 graph and algorithm choices, L1 loops and scheduling, and L2 target-source refinements.Source: §F.2.2 · pp. 23–24
  • Each LEKA agent works within its representation's scope and requests cross-layer changes from the relevant agent, preserving semantics and compatibility.Candidate adoption requires verified correctness and improvement in the measured objective.Source: §F.2.2 · p. 23

G.1 LIMITATIONS

The evaluation excludes specialized accelerators and cooperative CPU–GPU execution; future work proposes expanding both the hardware scope and the optimization setting.

G.1 LIMITATIONS

  • The evaluation covers CPU/LLVM, GPU/CUDA, and GPU/OpenCL, but not specialized accelerators such as NPUs.Each task runs on a CPU or GPU separately rather than partitioning computation for cooperative execution.Source: §G.1 · p. 24

G.2 FUTURE WORK

  • Future work proposes NPU evaluation and joint CPU–GPU optimization of computation partitioning, data transfers, and synchronization.This would examine how heterogeneous execution benefits vary with workloads and resource budgets, including performance–memory trade-offs.Source: §G.2 · p. 25

Related papers

Loading 2610.24648v1…