PaperlayerView exact revision on arXiv

Source-linked AI summary

Metal-Sci: A Scientific Compute Benchmark for Evolutionary LLM Kernel Search on Apple Silicon

Víctor Gallego

arXiv:2605.09708v2cs.LGcs.AIcs.DC

TL;DR

LLM-driven kernel search on Apple Silicon Metal must span structurally distinct optimization regimes, where optimization patterns do not transfer reliably across tasks. Metal-Sci supplies a 10-task benchmark and evolutionary harness with a held-out gate, finding model-specific generalization failures beyond in-distribution speedups.

  • Problem

    LLM kernel search on Metal must handle structurally distinct optimization regimes because no single optimization template transfers reliably across tasks.

  • Method

    Metal-Sci combines a 10-task benchmark with a runtime-compiled evolutionary harness that feeds structured diagnostics to a frozen LLM and evaluates one unseen configuration at the end.

  • Results

    1.00× to 10.7× in-distribution self-speedups span matched sweeps, while Opus loses correctness, GPT loses performance, and Gemini remains robust at higher wall-clock cost.

  • Takeaways & Limitations

    A single held-out configuration acts as a cheap mechanical oversight check that catches confidently wrong outputs before deployment.

  • Takeaways & Limitations

    Static per-chip ceilings ignore SLC residency at small sizes, and the single-population (1+1) loop plateaus within 10–25 iterations on most tasks.

Abstract

from arXiv · show

We present Metal-Sci, a 10-task benchmark of scientific Apple Silicon Metal compute kernels spanning six optimization regimes (stencils, all-pairs in $n$-body problems, multi-field Boltzmann, neighbor-list molecular dynamics, multi-kernel PDE, FFT). Each task ships a CPU reference, a roofline-anchored fitness function, and a held-out generalization size. We pair the benchmark with a lightweight harness for automatic kernel search that runtime-compiles each candidate, scores it against the roofline across multiple sizes, and feeds structured compile and per-size correctness diagnostics back to a frozen LLM driving a $(1{+}1)$ evolutionary loop. We report matched single-model sweeps of Claude Opus 4.7, Gemini 3.1 Pro, and GPT 5.5 on M1 Pro: in-distribution self-speedups span $1.00\times$ to $10.7\times$. Beyond raw speedup, our central methodological claim is structural: the held-out gate scoring function $Φ_\mathcal{T}$ (evaluated once at end-of-run on a configuration the agent never sees during search) functions as a cheap mechanical oversight primitive on this automatic search loop, catching e.g. an Opus template <uint D> HMC win that returns wrong samples at unseen dimensions, and a GPT FFT3D best that wins in-distribution at $2.95\times$ speedup but collapses to $0.23\times$ on a $256^3$ held-out cube, a silent regression that the in-distribution score alone cannot see. Code at https://github.com/vicgalle/metal-sci-kernels

1. Introduction

Metal-Sci benchmarks scientific compute kernel search on Apple Silicon Metal across structurally distinct optimization regimes, combining roofline-based scoring, runtime compilation, structured diagnostics, and held-out generalization evaluation. Its matched model sweeps frame the held-out gate Φ_T as a mechanical oversight primitive for automatic evolutionary search.

  • Search harness: The harness runtime-compiles Metal candidates, scores in-distribution sizes against per-size roofline ceilings, and returns compile and correctness diagnostics to the LLM.A candidate becomes the incumbent only when its score strictly improves; the held-out evaluation Φ_T runs once at the end.
  • Motivation: Apple Silicon Metal is an out-of-distribution target for CUDA-centric training, while unified memory enables sub-second compile-run-verify cycles for evolutionary search.Frontier models reportedly mishandle Metal-specific syntax such as [[max_total_threads_per_threadgroup]], and Metal provides rich simdgroup intrinsics.
  • Benchmark: 10 tasks span six optimization regimes, each with a CPU reference, roofline-anchored fitness function, and multi-size generalization gate.The regimes stress structurally distinct GPU and memory bottlenecks whose canonical optimization recipes do not transfer between neighbors.
  • Evaluation: Three matched single-model sweeps on M1 Pro operationalize the held-out gate Φ_T as a cheap mechanical oversight primitive.The evaluated models are Claude Opus 4.7, Gemini 3.1 Pro, and GPT-5.5.
  • Positioning: The benchmark differs from prior kernel-generation evaluations by targeting scientific compute, per-chip roofline scoring, held-out sizes unseen during search, and Apple Silicon Metal.Prior benchmarks primarily target machine-learning operators on CUDA or other accelerator backends.

2. Benchmark tasks

Metal-Sci organizes 10 scientific compute tasks into six structurally distinct optimization regimes, each targeting a different GPU or memory-hierarchy bottleneck. Every task combines correctness checking, multi-size evaluation, and a roofline-based performance ceiling while testing a regime-specific optimization lever.

  • Benchmark design: 10 tasks span six regimes whose canonical optimization recipes do not transfer across structurally different GPU and memory-hierarchy bottlenecks.The benchmark contrasts halo blocking, register tiling, and atomic-contention strategies across regimes.
  • Benchmark design: Each task includes a Metal seed, CPU reference with task-specific tolerance, three in-distribution sizes, one held-out size, and a per-size roofline ceiling.Ceilings are expressed in GFLOPS for compute-bound tasks or GB/s for bandwidth-bound tasks.
  • R1 – Regular stencils: R1 tests halo handling and temporal blocking for bandwidth-bound 5-point heat and 7-point wave stencils, with wave3d exposing compounding NaN failures.The stencil traffic is 8 B/cell for heat and 12 B/cell for wave.
  • R2 – Compute-bound: R2 tests register tiling for O(N^2) n-body pair sums and O(d^2) HMC matvecs, while HMC statistically verifies sample mean and covariance.Both kernels run at approximately 20 FLOPs per memory transaction against a peak FP32 GFLOPS ceiling.
  • R3–R6: R3–R6 cover SoA and streaming choices in LBM and Ising, atomic load balancing in molecular dynamics, reduction composition in PDE iteration, and data movement in 3D FFTs.The FFT regime includes Stockham versus bit-reversal, mixed-radix butterflies, twiddle caching, and simdgroup shuffles.

3. Harness Design

The harness wraps a frozen LLM in a strict (1+1) evolutionary loop that runtime-compiles, evaluates, and promotes Metal kernels using roofline-normalized throughput and correctness diagnostics. It separates the agent’s in-distribution optimization target from a held-out end-of-run oversight signal on an unseen size.

  • Evolution loop: Each iteration runtime-compiles a Metal candidate, dispatches it across three in-distribution configurations, scores roofline-normalized throughput, and feeds compile, correctness, and throughput diagnostics back to the LLM.The feedback packet F_k summarizes the previous candidate, incumbent, and short per-iteration history.
  • Compile and dispatch: GPU timings use three warmups and ten timed dispatches, reporting the median, while all multi-size dispatches share one MTLCommandBuffer.Metal source is runtime-compiled through PyObjC’s MTLDevice.newLibraryWithSource rather than the offline xcrun metal toolchain.
  • Held-out oversight: Φ_T evaluates the incumbent only at the unseen held-out configuration σ⋆, providing an external oversight signal that the agent never sees during optimization.The held-out configuration satisfies σ⋆_T ∉ Σ_T.
  • Scoring: The per-size metric is f_T(κ, σ)=a_T(κ, σ)/c_T(σ), dividing achieved throughput by the task’s per-size roofline in GFLOPS or GB/s.Correctness is represented separately by χ_T(κ, σ), which records whether the CPU reference is matched within tolerance.
  • Scoring: The in-distribution score is the geometric mean of f_T across sizes and is hard-gated to zero when any size fails tolerance, discouraging overfitting and speed-for-correctness tradeoffs.The strict (1+1) rule promotes a candidate only when its score is strictly higher under S_T.

4. Experiments

Matched evolutionary kernel-search sweeps on Apple M1 Pro found substantial but uneven in-distribution gains across three models, while held-out evaluation exposed asymmetric generalization failures and silent regressions. The held-out gate therefore distinguishes robust speedups from configurations that win only on seen sizes.

  • Experimental setup: Three matched sweeps used Claude Opus 4.7, Gemini 3.1 Pro, and GPT-5.5 on Apple M1 Pro under identical per-task budgets and no human prompt intervention.The setup used μ=1+λ=1 across ten tasks, with exceptions for lbm and wave3d iteration budgets.
  • In-distribution results: 1.00× to 10.7× self-speedups spanned the ten tasks in-distribution.Opus led saxpy, nbody, lbm, and wave3d; Gemini led gradshaf and lj; GPT won fft3d at 2.95×.
  • Correctness and failures: 13 Opus correctness failures contrasted with zero Gemini failures across all candidates.Opus failures were concentrated in wave3d, where 10 of 15 candidates failed; compile failures also recurred around Metal grammar, reserved half, and unsupported C++ lambdas.
  • Held-out generalization: 2.05×, 2.91×, and 1.86× held-out gradshaf speedups showed clean extrapolation for Opus, Gemini, and GPT, respectively.The held-out results identify nbody, gradshaf, lj, and some fft3d configurations as generalizing populations, with gradshaf the standout across all three models.
  • Held-out generalization: 2.95× GPT fft3d in-distribution speedup collapsed to 0.23× of seed on the held-out 256^3 cube.The fixed-twiddle, fixed-geometry layout tuned for N≤128 exceeded the held-out configuration’s register-pressure and threadgroup-memory limits, triggering a slower fallback.

5. Discussion

METAL-SCI’s central contribution is a cheap held-out gate that mechanically exposes failures hidden by in-distribution self-scoring. Across matched model sweeps, systems achieve 1.00×–10.7× self-speedups but fail held-out evaluation in distinct ways, motivating stronger oversight and future search improvements.

  • Three-model results: 1.00×–10.7× in-distribution self-speedups were measured across matched (1+1) sweeps of three frontier models, with each failing the held-out gate differently.Opus loses correctness, while GPT-5.5 loses performance.
  • Held-out oversight: 10.6× on hmc passed all visible checks, yet Opus’s incumbent violated correctness at held-out d=24.The agent’s internal score could therefore endorse confidently wrong code.
  • Held-out oversight: 4× slower in deployment was revealed by the held-out gate despite the agent labeling its output a ∼3× improvement over the seed.Both correctness and performance failures arose from specialization to the visible workload.
  • Held-out oversight: One auxiliary problem instance per task, requiring one extra dispatch and seconds of GPU time, can surface both failure types before deployment.The held-out gate is evaluated at end-of-run and is excluded from feedback packets.
  • Three-model results: Opus loses correctness at hmc d=24, GPT loses performance at fft3d 256^3, and Gemini has zero correctness failures across the candidate budget.At matched iteration budgets, in-distribution scores are close, but held-out failure shapes differ.
  • Limitations and future work: The single-population (1+1) loop plateaus within 10–25 iterations on most tasks, motivating island-model or archive-based search with novelty signals.Future work also includes workload-aware rooflines that account for SLC residency at small sizes.

A. Task formulations · B. Evolution loop pseudocode

The appendix specifies ten scientific compute tasks across six optimization regimes, including their computational formulations, memory or compute ceilings, and verification criteria. It also formalizes an evolutionary harness that compiles, runs, scores, diagnoses, and separately withholds the held-out oversight score.

  • A. Task formulations: The benchmark groups tasks into regimes R1–R6, with training and held-out sizes matching Table 2.The appendix provides per-task equations, ceilings, and tolerances.
  • A.1. R1: Regular stencils: R1 contains heat2d and wave3d regular-stencil solvers, with heat2d bandwidth-bound at 8 B/cell and wave3d using 12 B/cell unique DRAM traffic.Wave3d uses CFL parameter α = 0.18 under the stated stability condition.
  • A.4. R4: Irregular memory and atomics: R4’s lj task uses three molecular-dynamics kernels and atomic_fetch_add to build cell occupancy for irregular neighbor-cell interactions.The kernels are clear_cells, build_cells, and step.
  • A.5. R5: Multi-kernel reductions: R5’s gradshaf task alternates an interior ψaxis max-reduction with a variable-coefficient nonlinear 5-point stencil.The outer Picard iteration therefore comprises two kernels per step.
  • A.6. R6: Data-shuffle / butterfly: R6’s fft3d performs sequential x, y, and z fp32 FFT kernels on power-of-two cubes, with ∼5N log2 N FLOPs per 1D FFT and 96 B/cell effective DRAM traffic.Verification compares against numpy.fft.fftn with tolerance 10^-3 + 10^-3∥Y∥∞.
  • B. Evolution loop pseudocode: The evolution loop’s EVALUATE routine compiles, runs, scores, and returns either structured compile or per-size correctness failures or a successful score with per-size fractions-of-ceiling.The held-out Φ_T(κ⋆) is never returned during search and serves as the oversight signal.

C. Code-level evidence for the in-distribution split

The section supports a mechanistic split between the models: Opus improves the same algorithm, whereas Gemini reaches for a different one. Code-level comparisons extend this evidence beyond hmc to lbm and fft3d.

  • Mechanistic split: Opus wins by tightening the same algorithm, while Gemini wins by reaching for a different one.The comparative claim is explicitly mechanistic rather than merely performance-based.
  • hmc: Figure 3 instantiates the mechanistic claim on hmc.
  • Additional tasks: The appendix extends the code-level comparison to lbm, favored by Opus, and fft3d, favored by Gemini.It reports the actual diff between each model’s incumbent best.

C.1. lbm: Opus tightens BGK with FMA folds and a pinned threadgroup

For lbm, Opus improves the shared pull-stream BGK kernel through local FMA algebraic folding and a pinned 32×2 threadgroup, achieving a slightly higher in-distribution gmean than Gemini. Its advantage is regime-dependent: Gemini leads at 128^2, while Opus leads at 256^2, where cache residency favors the optimized geometry and instruction sequence.

  • Method: Opus extracts A once per cell, factors equilibrium with two FMAs, folds relaxation into a third FMA, unrolls nine blocks, and pins a 32×2 threadgroup.Both models retain the seed’s pull-stream and BGK structure; Opus changes the collision step and kernel attribute locally.
  • Results: 0.576 vs 0.553: Opus achieves the higher in-distribution gmean than Gemini.Gemini retains the canonical BGK formula and default geometry, while Opus uses FMA folds, A-extraction, and a threadgroup pin.
  • Results: At 64^2, Opus scores 0.34 versus Gemini’s 0.32, but at 128^2 Gemini leads 0.51 to 0.47.The models are competitive at 64^2, and Gemini reverses the ordering at 128^2.
  • Results: At 256^2, Opus leads 1.22 to 1.03, where pinned 32×2 geometry and FMA-folded BGK exploit the cache-resident regime.This size has the largest absolute fraction-of-ceiling and therefore dominates the gmean.

C.2. fft3d: Gemini swaps the algorithm to simd_shuffle_xor

Gemini’s fft3d advantage reflects an algorithmic change rather than a tighter Stockham implementation. It replaces early-stage threadgroup-memory exchanges with simd_shuffle_xor when Cooley–Tukey butterfly partners remain within a 32-lane SIMD group.

  • The fft3d gmean gap is 0.282 versus 0.167, a 1.7× difference, indicating materially different performance.
  • Opus uses a textbook Stockham auto-sort radix-4 FFT that ping-pongs every stage through threadgroup memory with barriers between stages.
  • Gemini identifies first-five-stage butterfly partners as lane i ⊕ 2^s−1, with 2^s−1 < 32, enabling simd_shuffle_xor fetches.Here s ∈ {1, . . . , 5}.

C.3. GPT-5.5 fft3d: hand-coded fast paths for N≤128, O(N 2) direct DFT for everything else

GPT-5.5’s best fft3d kernel reaches a 2.95× in-distribution geometric-mean speedup through hand-coded fast paths for N=32, 64, and 128. For other sizes, dispatch falls back to a textbook direct O(N^2) DFT.

  • GPT-5.5 fft3d: 2.95× in-distribution gmean: GPT-5.5’s iter-10 fft3d best outperforms Opus at 1.03× and Gemini at 1.19×.The result uses hand-coded fft_line_32/64/128 routines.
  • GPT-5.5 fft3d: The N=32, 64, and 128 fast paths use simd_shuffle_xor for intra-simdgroup stages and a precomputed W128[64] twiddle table for multiplies.The passage identifies simd_shuffle_xor as the same simdgroup primitive used by Gemini.
  • GPT-5.5 fft3d: For N values outside {32, 64, 128}, dispatch falls into a textbook direct O(N^2) DFT.This fallback is the stated source of the held-out collapse.

C.4. GPT-5.5 hmc: defensive D enumeration covering d=24 · D. Related work

GPT-5.5 uses defensive HMC dimension handling by enumerating D∈{8, 16, 24, 32} with templated instances and a generic runtime-d fallback, unlike Opus’s incomplete dispatch and Gemini’s pure runtime-d approach.

  • C.4. GPT-5.5 hmc: defensive D enumeration covering d=24: GPT-5.5 explicitly enumerates D∈{8, 16, 24, 32} for HMC.The enumeration covers d=24 directly.
  • C.4. GPT-5.5 hmc: defensive D enumeration covering d=24: GPT-5.5 provides fully-templated HMC instances for each enumerated dimension.This design differs structurally from both Opus and Gemini.
  • C.4. GPT-5.5 hmc: defensive D enumeration covering d=24: GPT-5.5 adds a generic runtime-d fallback for dimensions outside the enumeration.The supplied passage ends mid-word after describing this fallback.
  • C.4. GPT-5.5 hmc: defensive D enumeration covering d=24: Opus enumerates only D∈{8, 16, 32} for HMC.Consequently, d=24 is dispatched to the D=32 branch.
  • C.4. GPT-5.5 hmc: defensive D enumeration covering d=24: Opus’s d=24 dispatch silently runs an unrolled matvec against the wrong size.The passage attributes this behavior to Opus’s D enumeration and dispatch strategy.
  • C.4. GPT-5.5 hmc: defensive D enumeration covering d=24: Gemini uses a pure runtime-d leapfrog with no template specialization.The passage states that Gemini pays for safety with in-distribution throughput.

D.1. LLM-driven kernel-generation benchmarks · D.2. LLM-driven evolutionary code search

D.1 situates Metal-Sci among benchmarks for LLM-generated high-performance kernels across GPU, DSL, NPU, and emerging accelerator settings. D.2 connects it to LLM-driven evolutionary code search, population-based optimization, and systems where the search process itself becomes the synthesis target.

  • D.1. LLM-driven kernel-generation benchmarks: KernelBench scores single-shot PyTorch ML-kernel generation on CUDA by speedup against the PyTorch eager baseline.TritonBench adds Triton DSL coverage, reference-code similarity, correctness, and performance.
  • D.1. LLM-driven kernel-generation benchmarks: BackendBench evaluates correctness alone, while NPUEval broadens coverage to CUDA, Ascend C, and Pallas and adds multi-turn tool-use feedback.NPUEval targets the AMD AIE NPU in C++.
  • D.1. LLM-driven kernel-generation benchmarks: KernelCraft benchmarks bare-metal assembly generation across PLENA, AMD NPU, Coral NPU, and Sonic BOOM using native tool interfaces and a three-tier ML…The supplied passage truncates the final tier description.
  • D.2. LLM-driven evolutionary code search: FunSearch and AlphaEvolve established and scaled LLM optimization within evolutionary code-search loops, while AI CUDA Engineer and EvoEngineer target CUDA-kernel optimization.These approaches use populations or archives driven by fitness signals rather than single-shot or agentic regeneration.
  • D.2. LLM-driven evolutionary code search: Eureka synthesizes RL reward functions through GPU-parallel rollouts, while ReEvo evolves combinatorial-optimization heuristics with explicit short- and long-term reflect…The supplied passage truncates the final description of ReEvo.
  • D.2. LLM-driven evolutionary code search: Autoresearch edits a nanoGPT training script under a held-out validation-loss budget, whereas Gallego rewrites the synthesis pipeline of an inner-loop policy synthesizer.Metal-Sci is positioned at the inner-loop level of this hierarchy, with its harness, scoring rule, and prompt scaffold described in the supplied passage.
Loading 2605.09708v2…