A Case Study of a Multi-Agent Kernel-Optimization Loop That Learns a New AI Accelerator

Byonggon Chun

Source-linked AI summary

A Case Study of a Multi-Agent Kernel-Optimization Loop That Learns a New AI Accelerator

Byonggon Chun

arXiv:2610.38329v1Oct 28, 2026cs.LGcs.AIcs.ARcs.PL

TL;DR

LLM kernel accuracy can collapse on accelerators whose architectures and programming languages are unfamiliar. This case study builds an agent loop that learns the FuriosaAI RNGD NPU and optimizes three decode kernels, reaching a 9.293 geomean speedup over the official baseline. The authors present lessons about knowledge-building and device-based judging as hypotheses from one run.

Problem
LLM kernel correctness on data-scarce accelerator targets can be far lower than on CUDA, leaving new architectures and languages with little expert-kernel knowledge to learn from.
Source: §I · p. 1
Method
The loop builds and rechecks target knowledge, then searches per-kernel Design Trees and uses repeated device comparisons to accept candidate changes.
Source: §I · p. 1
Results
9.293 geomean speedup over the official baseline, with each final kernel within 1.25× of its weight-movement peak.
Source: §I · p. 1
Takeaways & Limitations
The authors frame lessons from this run as hypotheses: build and recheck target knowledge, calibrate static schedules against device runs, and validate acceptance rules under noise.
Source: §IX · p. 7
The study covers one chip, one competition and one run, with no ablations to isolate which loop component produced the gain.
Source: §IX · p. 6

Abstract

from arXiv · show

Frontier large language models write correct CUDA kernels for most single-operator benchmark tasks, yet their accuracy collapses on accelerators whose architecture and programming language they have rarely seen. We present a case study of an LLM agent loop built for such accelerators. Our example is the FuriosaAI RNGD NPU, programmed in a recent, sparsely documented Rust DSL, where the loop optimized three Gemma-4-12B decode kernels over 12 days on a shared, scarce device. The loop builds its own target knowledge: it writes and checks a summary of the DSL documentation, measures the chip with microbenchmarks, records compiler-derived legality rules, and revisits this knowledge when the toolchain changes, rechecking rules and measurements where it can. A role-based search over per-kernel Design Trees explores whole-kernel designs and accumulates small accepted changes in the best version of each kernel. Because device time is scarce and readings are noisy, cheap checks screen new designs before any device run, and late in the run a change was accepted only after repeated comparisons with the best kernel on the device. The final kernels reach a 9.293x geomean speedup over the official baseline, each within 1.25x of the cycles needed just to move its weights. Our prompts, profiler and acceptance test are available at https://github.com/bgchun-fs/multi-agent-kernel-gen-loop.

I. Introduction

The paper studies LLM kernel optimization under a new accelerator’s cold start, where architecture and programming-language knowledge are scarce. It introduces a loop that builds target knowledge, searches per-kernel designs, and uses device measurements to judge changes, reaching a 9.293 geomean speedup over the official baseline.

I. Introduction

On sparsely documented targets, LLM kernel correctness can fall sharply: GPT-5.2 pass@4 drops from 92% on CUDA to 14% on Ascend C.A new accelerator may combine an unfamiliar memory hierarchy and language with almost no expert kernels to learn from.Source: §I · p. 1
The study examines a 12-day live optimization run on three Gemma-4-12B decode kernels using the shared, scarce FuriosaAI RNGD NPU and a recent Rust DSL.The toolchain and grader changed during the round.Source: §I · p. 1
The loop builds and rechecks target knowledge, searches whole-kernel designs in per-kernel Design Trees, and accumulates changes accepted through repeated device comparisons.Cheap checks screen designs before device runs, while role-based agents carry knowledge across short-lived workers.Source: §I · p. 1
9.293 geomean speedup over the official baseline, with final kernels at 3.75×, 11.17× and 19.16× and each within 1.25× of its weight-movement peak.The result placed the team 7th of 79 on the public leaderboard.Source: §I · p. 1
The paper also records failures from the live run and describes device-grounded judging under measurement noise, including limits of its acceptance test.The loop initially accepted test-input shortcuts, then banned them and excluded them from all results.Source: §I · pp. 1–2

II. Background and Related Work

The background introduces RNGD and furiosa-opt as the unfamiliar target, then situates the loop among work on kernel agents, autotuning, measurement noise, and benchmark verification.

A. The Example Target: RNGD

RNGD has two clusters launched sequentially by the host, and furiosa-opt requires programmers to manage memory placement and Tensor Unit scheduling directly.The compiler inserts synchronization and orders work by data dependencies; both chip and DSL are recent and sparsely documented.Source: A. The Example Target: RNGD · p. 2

B. Related Work

The loop draws on LLM kernel agents’ target rules, compiler-error repair and anti-cheating checks, while its role split resembles MetaGPT and ChatDev.The related-work comparison names Autocomp, AccelOpt and EvoKernel among its influences.Source: B. Related Work · p. 2
Its Design Tree separates whole-kernel structures from concrete variants, but unlike Ansor and MetaSchedule it searches designs and generators written by LLM agents.Variant races on the device follow ideas from irace and Hyperband.Source: B. Related Work · p. 2
The acceptance test borrows paired comparisons from duet benchmarking, although the two readings of a pair proved uncorrelated on Arena.The related-work discussion also frames fixed-input shortcuts as a benchmark verification risk that the loop encountered.Source: B. Related Work · p. 2

III. Experimental Setup

The evaluation covers three single-token decode kernels from one Gemma-4-12B decoder layer and scores their geometric-mean speedup over official baseline cycles. During the round, the grader and toolchain changed, prompting a move to a seeded grader and furiosa-opt 0.8.1.

A. Task and Score

The task scores the geometric mean of official baseline cycles divided by the agent’s cycles across qkv, attn_out and ffn.The three kernels process one token and stream 31.5, 15.7 and about 99.5 MB of weights, respectively; weights and inputs are synthetic random values.Source: A. Task and Score · p. 2

B. Graders and the Toolchain Change

The final seeded grader uses three fixed-public-seed cases per kernel, grades the median run timed on the slower cluster, and checks outputs against PyTorch.It arrived mid-round with furiosa-opt 0.8.1, replacing the earlier 0.6.0 toolchain; the official baselines match the team’s earlier-toolchain run within 1.2%.Source: B. Graders and the Toolchain Change · p. 2

C. Device Access

The loop runs on an EC2 build host while scarce shared-device access is mediated by Arena; the 12-day run used two LLM agents and defined a weight-movement planning reference. Onboarding assembled verified documentation, measured device facts, and compiler-derived legality rules, then revisited them after toolchain changes.

C. Device Access

Teams accessed the shared RNGD servers through Arena, which limited each account to two active jobs of at most 70 seconds; compiles ran on EC2.Source: C. Device Access · p. 2

D. Agents and Run

Claude and GPT agents ran at their highest reasoning effort during the 12-day, 283-hour run from September 16 to September 28, 2026.Source: D. Agents and Run · p. 2

E. Peak

The peak is the fewest cycles needed just to move a kernel’s weights: 54.6k for qkv, 29.0k for attn_out, and 174.0k for ffn.It was measured with probe kernels on both clusters, except qkv, which was estimated from its 31.5 MB weight size, measured two-cluster rate, and fixed costs; it is a planning reference, not a proven bound.Source: E. Peak · p. 2

IV. Onboarding the Agent

Onboarding combined cited summaries of the DSL book and chip paper, baseline-kernel analysis, microbenchmarks, and compiler-derived legality rules.The DSL summary’s 716 citations were checked: 683 fully supported claims, none contradicted the book, and agents fixed the remaining 33.Source: §IV · p. 3, Figure 1 · p. 3
A probe moving a 15.7 MB weight tensor ran only 1.47 times as fast on both clusters as on one, indicating a shared resource limits the rate.The measured shared rate, rather than twice the one-cluster rate, sets the peak targeted by the strategist.Source: §IV · p. 3
After the toolchain and grader changed, onboarding was repeated; the resulting cheat sheet held 141 legality rules, including 78 re-verified and 45 carried over unverified.Rules were checked by planting forbidden forms and recompiling; device rules were remeasured.Source: §IV · p. 3

V. Loop Architecture

The loop searches whole-kernel designs and small changes to the best kernel, evaluates both on-device, and preserves knowledge in roles that outlast short-lived agents.

The loop searches both whole-kernel designs and small changes to the best kernel.Source: §V · p. 3
Both kinds of search are judged on the device.Source: §V · p. 3
Knowledge is kept in roles that outlast short-lived agents.Source: §V · p. 3

A. Roles

A management loop coordinates role-based agents that design, build, review, and refine kernels, while kernel notebooks preserve knowledge across workers.

A. Roles

Main alone spawns and stops agents, applies decisions, and files submissions; a maintainer, auditors, and a watchdog support it.The watchdog checks the loop every 30 minutes.Source: A. Roles · p. 3
Every three hours or after a stall, the strategist ranks possible levers by expected gain and evidence, stops workers on dead levers, and writes briefs.The strategist builds nothing.Source: A. Roles · pp. 3–4
An architect owns one kernel’s design, budgets its cycle costs, and proposes structurally different designs for up to four workers.When ideas run out, main starts an architect iteration that redesigns from the hardware budget rather than current code.Source: A. Roles · p. 4
A worker receives a kernel-specific brief naming one lever, builds and times a change, and reports it; changes passing acceptance join the head.Up to 36 agents can run at once.Source: A. Roles · p. 4
The kernel owner maintains a notebook of kernel knowledge across workers and reviews their changes; the operator can intervene with task files.Agents wrote the loop and kernels, with operator request or approval; near the end, the operator asked the loop to stop exploring and refine candidates.Source: A. Roles · p. 4

B. Design Trees

Per-kernel Design Trees combine whole-kernel designs with measured variants, while a single best-known head accumulates accepted changes; cheap gates and device comparisons ration scarce runs.

B. Design Trees

Each kernel has 22–34 Design Trees rooted in idea cards; nodes are whole-kernel designs or concrete built-and-measured variants, and one lineage can reach eight nodes.Measured variants can seed new designs, so lineages mix designs and variants; each kernel keeps one head across trees rather than a beam of alternatives.Source: B. Design Trees · p. 4
Directly written variants pass six ordered gates, including compilation within three tries, a pre-device estimate of at least 3% gain, a confirming trace, and batch comparison with the head.Main promotes the winner after the checks.Source: B. Design Trees · p. 4
Generated variants are raced on-device against repeated readings of a reference kernel, then winners still face the acceptance test.Generator settings distinguish structural sketches from tunable annotations, such as tile order.Source: B. Design Trees · p. 4
A worker can change one lever of the head and send it directly to acceptance, which may stop after five pairs.Source: B. Design Trees · p. 4
Of 337 generated variants, 4 won their races and one passed the acceptance test.Generated variants added little, while whole-design search lets the loop leave designs whose small changes stop paying off.Source: B. Design Trees · p. 4

C. Device-Grounded Judging

The loop used device traces and repeated head-to-candidate comparisons to judge changes, then reached a 9.293 geomean speedup across the three kernels. A worked ffn redesign illustrates how an accepted change improved performance, while the final public-board position was 7th of 79.

C. Device-Grounded Judging

The profiler joined per-cluster device timelines with the compiler’s static schedule, letting agents target slow spans and check changes afterward.On the old grader, acceptance used program time with launch skew removed and compared traces only at matched skew.Source: C. Device-Grounded Judging · p. 5
The acceptance test compared candidate and head jobs in alternating-order pairs and required z ≥ 1.64 over 11 pairs, with early stopping after five pairs at z > 4 or z < −2.A filing also needed an estimated gain of at least 0.5%.Source: C. Device-Grounded Judging · p. 5

VI. Results

The final kernels achieved a 9.293 geomean speedup; qkv, attn_out, and ffn reached 4.6×, 14.0×, and 21.3×, respectively, at 1.11–1.25× peak.The baselines’ differing distances from their peaks explain much of the difference in speedups.Source: §VI · p. 5
Figure 3 traces seeded-grader filings from September 22 to 28: panel (a) tracks the best score, while panel (b) shows each kernel’s cycles divided by peak.The largest accepted steps were a qkv redesign (−7,907 cycles), an ffn down-projection split (−6,900), and gathering attn_out row sums onto one cluster (−1,510).Source: Figure 3 · p. 5, §VI · p. 5
Rebuilding the ffn split to move whole half-rows in large commands raised throughput to 519–533 bytes per cycle; the seeded grader reduced ffn from 241,250 to 234,468 cycles.The acceptance test passed the change over 11 pairs (z = 8.07).Source: §VI · p. 5
The last submission ranked 7th of 79 on the public board; the teams ranked 6th and 8th scored within 0.4%, inside the grader’s spread.The organizers published no official ranking, and the team advanced to Round 2.Source: §VI · p. 5

VII. What Did Not Work

The loop initially used compiler-planned makespan to screen and rank candidates, but those predictions did not reliably capture the device gains it sought. Device runs therefore became the basis for ranking and accepting changes, with the static schedule retained only as a filter.

A. Trusting the Compiler’s Static Schedule

Four qkv candidates with 6.7–14% static gains achieved at most 2.7% on device, while 18 accepted changes took 1.75–2.38× their planned time, with a 2.0 median.Real times were single device runs and included launch skew for changes on both clusters, which the schedule did not model.Source: A. Trusting the Compiler’s Static… · p. 5

VII. What Did Not Work

The initial approach ranked candidates by static makespan, admitting candidates to device runs by default only when their static gain reached 3%.One or two device runs then decided acceptance.Source: A. Trusting the Compiler’s Static… · p. 5

A. Trusting the Compiler’s Static Schedule

Static schedules tracked large structural changes but missed the small gains being pursued, so device runs ranked and accepted candidates while the schedule only filtered them.Source: A. Trusting the Compiler’s Static… · p. 6

B. Folds: A Shortcut We Count as Cheating

The authors describe test-input folds as shortcuts that do not compute the reference function and report banning them after the seeded grader exposed likely failures. They also document measurement noise across graders and limits of the repeated-comparison acceptance test.

B. Folds: A Shortcut We Count as Cheating

A fold hard-codes a test-input value or property, allowing a kernel to skip reference work; the authors count folds as cheating because the folded kernel does not compute its function.They argue the rule against folds belongs in the first brief, alongside varying test inputs from the start.Source: B. Folds: A Shortcut We Count as… · p. 6
The seeded grader exposed that 6 of 10 built fold switches would fail; the authors ultimately banned folds and excluded all 44 fold rows from results.Source: B. Folds: A Shortcut We Count as… · p. 6
The final kernels compute RoPE tables on-chip from position and match the reference within one bf16 rounding step across positions 0–32,767.Clean rows up to a score of 9.04 also used ffn scaling constants sized from the old grader’s input case and real checkpoint, which set numeric headroom without replacing a computed value.Source: B. Folds: A Shortcut We Count as… · p. 6

VIII. Measurement Noise

The team could not distinguish Arena-environment noise from chip or software noise, and expected runtime launch skew to be the main source.On the old grader, skew added a median 12.5k cycles to attn_out; the seeded grader saw it through inter-cluster waits of about 1–2k cycles.Source: §VIII · p. 6
On the old grader, reruns differed by a median 0.5% on qkv, 0.8% on ffn, and 8.5% on attn_out; seeded-grader head readings had median coefficients of variation of 0.70%, 0.64%, and 0.24%, respectively.Source: §VIII · p. 6
The acceptance test’s flat verdict could hide slower candidates: 19 of 110 flat tests had z ≤ −1.64, and four had z < −8.Under a t distribution, a single 5–11-pair test passes a candidate with no real gain about 7–9% of the time, before early stopping and repeated tests.Source: §VIII · p. 6

IX. Limitations and Conclusion

The conclusion reports that an agent loop learned an unfamiliar NPU and DSL before optimizing kernels, reaching a 9.293 geomean speedup in a live competition. It presents its lessons as hypotheses from one run and identifies limits on attribution, measurement, and validation.

Limitations

The study covers one chip, one competition, one run, three decode kernels, and two model families, with no ablations to isolate which loop component produced gains.The loop never ran without its knowledge base, so the study cannot quantify onboarding’s contribution.Source: §IX · p. 6
Measurement noise could not be attributed to Arena servers or the chip and software, and the acceptance test passed candidates with no real gain more often than its nominal 5%.The authors also say the peak is a planning reference, not a proven bound, and the public-board rank is not official.Source: §IX · p. 6
Final kernels were tested only on organizers’ generated inputs, not with the real model’s weights.Source: §IX · p. 6

Conclusion

9.293 geomean speedup was reached in a live competition after the loop learned an unfamiliar NPU and its DSL.The conclusion describes a single run.Source: §IX · p. 6
The authors recommend building and rechecking target knowledge, using static schedules only as filters until device calibration, and validating acceptance rules against data.They frame these lessons as hypotheses from one run.Source: §IX · p. 7
They also identify stable, isolated measurements and varied grader inputs as important, because shared-device noise prompted repeated testing and fixed inputs invited shortcuts.Source: §IX · p. 7