MetaKernelBench: Measuring GPU Kernel Knowledge Transfer Beyond Code

1 INTRODUCTION

• Kernel benchmarks score each implementation for correctness and speed, leaving the reuse value of retained experience unmeasured.The paper asks whether experience distilled from an attempt in one DSL can improve a fresh attempt at the same problem in another.Source: §1 · p. 1
• MetaKernelBench pairs 74 fused-subgraph problems across CuTe DSL and TIRx under fixed semantics, workloads, hardware, and verification rules.Each solo attempt distills findings into a natural-language skill, which is transferred even when the source attempt fails verification.Source: §1 · pp. 1–2
• The skill is the only extra input to a fresh same-model attempt in the other DSL, compared with a solo attempt under matched per-attempt budgets.The source implementation and trajectory are withheld; comparisons measure correctness and end-to-end runtime.Source: §1 · p. 2
• Paired lift ranges from −19% to +29% across six models and both directions; four models gain in both directions, but every model and direction has regressions on 16% to 45% of problems.Comparisons improve in 71% of cases when the source stands above the target’s solo attempt, and regress in 54% when it stands below.Source: §1 · p. 2
• The paper frames same-problem cross-DSL transfer as an evaluation dimension complementary to the correctness and performance of an individual implementation.Its analyses relate outcomes to source standing and to skill content, cases, and failures.Source: §1 · p. 2

2.1 TASK FORMULATION

• Each problem has a DSL-neutral executable specification and fixed workload; its CuTe DSL or TIRx variant adds the DSL and software environment, with end-to-end speed measured against eager PyTorch.Valid implementations may combine eager PyTorch, ordinary Python, and DSL kernels, but must execute a DSL kernel that contributes to a returned output.Source: §2.1 · p. 3

2.2 TRANSFER PROTOCOL

• The shared directive asks agents to maintain evidence-backed natural-language skills covering optimization findings, work partitioning, failure reasons, applicability conditions, and open questions.The directive forbids code, commands, and API instructions; conclusions should not depend on source-DSL syntax or APIs.Source: §2.2 · p. 3
• The target’s solo and skill-conditioned attempts share the task, verifier, feedback, and implementation rules; only the source skill is added, and paired lift does not isolate the contributions of different guidance types.Skills are transferred whether or not the source implementation passes verification, while the source implementation and trajectory are withheld.Source: §2.2 · p. 3

2.3 PROBLEM CONSTRUCTION

• The 74-problem catalog screens fused subgraphs from four upstream sources and assigns retained problems to six families based on their main implementation constraint.The catalog combines 42 problems from SOL-ExecBench, 20 from FlashInfer sources, 8 cache-maintenance problems, and 4 linear-attention problems.Source: Figure 2 · p. 4, §2.3 · p. 4

3.1 MODELS AND AGENT SCAFFOLD

• Six models use the terminal-only Terminus-2 scaffold through Harbor, with no overrides to sampling temperature or reasoning effort.The panel includes three Flash-tier models and three larger models.Source: §3.1 · p. 4

3.2 EXECUTION ENVIRONMENT

• Agents work in isolated CPU-only sandboxes for up to 40 minutes and 10 feedback submissions, with collected implementations evaluated on an NVIDIA B200.Evaluation uses fresh input seeds; the sandbox includes offline documentation and fixed-version DSL dependencies.Source: §3.2 · pp. 4–5

3.3 EVALUATION METRICS

• A scored pass requires float64-anchored correctness and end-to-end timing; solo metrics report pass rate, passing-only geometric mean speedup, and the 2×-or-better pass rate.Failures remain in both rate denominators, and speedup is capped at 100×.Source: §3.3 · p. 5
• Equation 2 scores pass rescues as 4, pass breaks as 1/4, and joint failures or skipped treatments as 1; Equation 3 aggregates equal-weight paired lift, help, and harm.These measures are reported by model and direction, within each family and over all 74 problems.Source: §3.3 · p. 5

4.1 OVERALL TRANSFER OUTCOMES

• Paired lift ranges from −19.1% to +29.3%; four models gain in both directions, while every model and direction shows both help and harm.Help rates range from 21.6% to 40.5%, and harm rates from 16.2% to 44.6%.Source: §4.1 · pp. 5–6, Table 1 · p. 6
• Across all 74 problems, pooling families yields opposite lift signs in 11 of 12 model-direction cells; among 277 helped comparisons, the reverse direction is harmed 41.5% and helped 23.1%.Source: §4.1 · p. 6

4.2 IMPLEMENTATION QUALITY AND TRANSFER OUTCOMES

• Solo pass rates and passing-only speed rank models differently; TIRx has higher geometric mean speedup than CuTe DSL for every model, while pass rates favor neither DSL.DeepSeek-V4-Pro passes every variant but has the lowest geometric means; Qwen3.8-Flash has the lowest pass rates but the highest TIRx geometric mean. Six models are too few to conclude solo pass rate predicts paired lift.Source: §4.2 · pp. 6–7, Table 2 · p. 6
• When the source stands above the target’s solo tier, 71.2% of comparisons are helped and 8.1% harmed; when it stands below, 10.6% are helped and 53.8% harmed.Help rates follow this ordering in every model, so a passing source alone does not guarantee beneficial reuse.Source: §4.2 · p. 7, Figure 4 · p. 7

4.3 SKILL CONTENT ACROSS TRANSFER OUTCOMES

• Unsupported closure is 7 to 17 points more prevalent in harmed comparisons for five models, while verified reformulation and ambition-capping patterns vary across models.Verified reformulation is 7 to 9 points more prevalent in helped comparisons for three models but 12 points more prevalent in harmed comparisons for GPT-5.6 Sol; ambition capping differs between GLM-5.3-Flash and DeepSeek-V4-Pro.Source: §4.3 · pp. 7–8, Figure 5 · p. 7
• Two GLM-5.3 traces show contrasting uptake: one transfer reuses a thread mapping and diagnostic control, while another retains eager execution and misses the native recurrence.The helped attempt fixes a suspected indexing fault and fuses the recurrence; the runtime comparisons do not isolate uptake’s causal contribution.Source: Figure 6 · p. 8, §4.3 · p. 8

4.4 FAILURE CAUSES AND SKILL UPTAKE

• Skill-side truncation and unverified adoption dominate assigned harm causes for the three larger models, while execution and related failures dominate for two Flash-tier models.Skills are followed fully or partially in over 80% of harmed comparisons, and unverified claims reappear in attempts’ own skills more often in harmed than helped comparisons for every model.Source: §4.4 · pp. 8–9, Figure 7 · p. 8

• Kernel-generation work includes model adaptation, execution-guided search, reinforcement learning, finetuning, and evolutionary search; benchmarks assess correctness, speed, verification, hardware, languages, or generalization.The cited suites include KernelBench and AgentKernelArena.Source: §5 · p. 9
• Prior reuse approaches retain experience in knowledge bases, reasoning graphs, or skills, while code-level work translates kernels, adapts CUDA, or extracts verified skills for reuse.One cited approach admits self-distilled skills only after they meet an execution-based speedup threshold on the original task.Source: §5 · p. 9
• MetaKernelBench transfers unscreened self-distilled skills from passing or failed attempts to fresh attempts on the same problem in another DSL, measuring whether they help or harm.Source: §5 · p. 9

6 CONCLUSION

• The conclusion positions MetaKernelBench as a complementary measure of reuse value: source implementation quality alone does not determine whether a skill helps.The source’s standing against the target’s solo attempt matters, and both should be measured.Source: §6 · p. 9

A.1 TASK INSTRUCTION

• Every attempt receives the same instruction structure, and paired prompts differ only in the named DSL, with corresponding differences in sandbox documentation and dependencies.Source: §A.1 · p. 13

A.2 WORKLOADS AND RUNTIME CONFIGURATION

• Workloads use fixed configurations with fresh tensor values, and some generators also vary index structures or sequence lengths; evaluations run in single-use B200 containers with specified software and resource limits.The agent budget excludes environment setup and final evaluation.Source: §A.2 · p. 13

A.3 CORRECTNESS AND TIMING

• Correctness checks compare outputs with a float64-anchored reference, while timing uses synchronized CUDA events and excludes input generation and correctness checks.Reference-generation and eager-timing precision settings are not imposed on submissions, but paired lift compares two implementations under the same evaluation setup.Source: §A.3 · p. 13

A.4 IMPLEMENTATION RULE CHECK

• Without the static rule check, paired-lift signs remain unchanged in every model and direction, despite excluding 42 verifier-passing implementations.The largest lift change is Gemini 3.7 Flash’s CuTe DSL to TIRx result, from 2.3% to 0.6%.Source: §A.4 · p. 13

B.1 OUTCOME TIERS

• At sources 2× or better, the three larger models’ paired lift is 30.0% to 38.8%, versus −1.7% to +18.8% for the Flash tier.Across solo outcome tiers, including tiers where solo passes, these sources yield the highest paired lift; pooled lift rises with source tier in every model.Source: §B.1 · p. 15, Figure 9 · p. 15
• Even when solo already reaches 2× speedup, the highest source tier has negative paired lift.The tier comparison uses speedup relative to the eager reference for passing attempts, and counts skipped treatments as neutral.Source: §B.1 · p. 15, Figure 9 · p. 15

B.2 DIRECTION ASYMMETRY AND SOURCE TRACKING

• Across 444 model–problem pairs, the two directions’ log paired ratios correlate at −0.26, reaching −0.34 among 283 pairs whose four attempts all pass.The correlation is negative within every model and family; none of 20,000 reference draws is as negative as the observed correlation.Source: §B.2 · p. 15
• Skill-conditioned attempts correlate with source speedup at 0.51, versus 0.35 for solo attempts, across 841 comparisons.None of 2,000 reference draws produces a correlation gap at least as large as observed.Source: §B.2 · p. 15
• With a nonpassing source, skill-conditioned attempts pass in 62.9% of comparisons, versus 75.3% for solo attempts on the same variants.The comparisons cover 97 cases with a nonpassing source.Source: §B.2 · p. 15

C.1 CONTENT CATEGORIES AND UPTAKE RATINGS

• Source skills receive binary labels across 11 content categories, while attempts are rated for full, partial, absent, contradicted, or inapplicable uptake.The labels characterize claims and stated evidence without verifying them; propagation tracks false or unverified source claims repeated in an attempt’s own skill.Source: §C.1 · p. 15, Table 3 · p. 16
• Skills are followed fully or partially in 82.9% to 100% of harmed comparisons and 97.1% to 100% of helped comparisons.Figure 10 reports uptake and propagation shares by transfer outcome and model.Source: §C.1 · p. 16, Figure 10 · p. 16

C.2 FAILURE CLASSES

• Of 223 harmed comparisons, 211 target-side failures receive a primary failure class; the other 12 are excluded from failure-cause counts.When multiple classes apply, precedence runs from rule violation through faulty execution, budget and artifact mismanagement, toolchain friction, unverified adoption, tolerance-edge loss, and search truncation.Source: §C.2 · pp. 16–17, Table 4 · p. 16
• Gemini 3.7 Flash’s 15 rule and precision failures comprise 12 inline CUDA exclusions, two other rule violations, and one tolerance-edge loss.Source: §C.2 · p. 17

D ADDITIONAL CASE STUDIES

• On p74, DeepSeek-V4-Pro’s TIRx source stood above its target’s solo attempt and yielded a faster CuTe DSL transfer, while the lower-standing CuTe DSL source yielded a slower TIRx transfer.The helpful attempt ports a blueprint into a native recurrence; the harmful attempt accepts unverified framing and never tries a native recurrence.Source: D ADDITIONAL CASE STUDIES · p. 17
• On p65, a CuTe DSL transfer helped after CPU-tested batched-matmul guidance was GPU-validated; on p09, TIRx was harmed after adopting unvalidated SDPA rules without trying variable-length flash attention.Neither source in these cases stood above its target’s solo attempt.Source: D ADDITIONAL CASE STUDIES · pp. 17–18

E.1 LIMITATIONS

• The evaluation covers six models transferring between CuTe DSL and TIRx on a B200 under one scaffold, skill directive, and per-attempt budget.Source and skill-conditioned attempts use the same model, so the study does not separate skill distillation from skill use.Source: §E.1 · p. 18
• The same-problem design does not establish generalization to new problems or workloads, or the accumulation of skills over rounds.The content analysis relates outcomes to content categories, not individual statements or their wording.Source: §E.1 · p. 18

E.2 FUTURE WORK

• Comparing transferred materials, matched-content skill formats, cross-model transfers, and individual guidance edits could isolate factors affecting reuse and target-attempt outcomes.The proposed comparisons include source implementations, trajectories, free-form versus structured skills, model pairs, and statement-level edits.Source: §E.2 · p. 18
• Transfers to related problems and chained rounds could test whether findings remain relevant and whether unverified claims are refined, corrected, or propagated.The proposed work would track how retained knowledge changes and affects later attempts.Source: §E.2 · p. 18