FastKernels: Benchmarking GPU Kernel Generation in Production

1 Introduction

• Existing isolated GPU-kernel benchmarks reward synthetic, single-GPU optimizations that often fail in production through interface mismatches, compilation conflicts, or correctness degradation.These failures emerge in real serving interfaces, compilation stacks, workloads, and model-scale execution.Source: §1 · p. 1

Our approach

• FastKernels is a self-contained inference framework whose production-matching task interfaces let optimized kernels run in real pipelines and transfer into systems such as vLLM and SGLang.It supports continuous batching, chunked prefills, multimodal inputs, and an OpenAI-compatible serving API, while matching vLLM and SGLang on mainstream serving.Source: Our approach · p. 2, Contributions · p. 2
• Its compositional hierarchy connects primitives, fused operators, layers, and full models, allowing agents to reuse lower-level optimizations instead of solving independent task levels.Production-model tensors are captured and replayed; a MoE routing study shows synthetic inputs change load skew and hot-expert identity.Source: Our approach · p. 2, Contributions · p. 2

Contributions

• FastKernels evaluates generated kernels against production baselines using captured tensors, compilation-stack effects, multi-GPU communication, and tensor and expert parallelism.This production-faithful evaluation targets deployment conditions rather than hardware-theoretical bounds.Source: Contributions · pp. 2–3
• It injects generated kernels into full model executions, checks downstream quality, and reports MACROEVAL across correctness, coverage, and end-to-end throughput–latency speedup.MACROEVAL combines calibrated correctness, coverage, and end-to-end throughput–latency speedup across model families.Source: Contributions · p. 2
• The benchmark covers 46 architectures across 8 categories, including dense and MoE LLMs, linear attention and SSMs, and multimodal and specialized models.The listed categories also include vision, audio, video, robotics, 3D graphics, recommendation, and world models.Source: Contributions · pp. 2–3

GPU kernel benchmarks

• Existing benchmarks mainly assess isolated operators on single GPUs, with limited robustness, backend, or inference-engine coverage.KERNELBENCH, ROBUST-KBENCH [8], SOL-EXECBENCH [13], FLASHINFER-BENCH, CUDABENCH [26], and TRITONBENCH exemplify these settings.Source: §2 · p. 3
• FastKernels adds multi-GPU communication, a primitive-to-model compositional hierarchy, production-kernel speedup measurement, and production-matching module interfaces.Its interfaces support copy–paste deployment into frameworks such as vLLM and SGLang.Source: §2 · p. 3

LLM-based kernel agents

• Kernel-agent research combines LLM reasoning with profiling or evolutionary search, or trains specialized models, but typically relies on operator-level benchmarks.FastKernels provides a production-aligned evaluation surface complementing these approaches.Source: §2 · p. 3

3.1 Top-Down, Model-Driven Construction

• Tasks are recursively derived from real model forward passes, audited for semantic fidelity, actual execution behavior, configurations, and production-library interface compatibility.The construction pipeline supports adding HuggingFace architectures with minimal effort, and every kernel corresponds to a real inference operation rather than a synthetic task.Source: §3.1 · p. 3

3.2 Task Hierarchy

• The benchmark organizes optimization from primitive operators through fused operators and full layers to integrated end-to-end model architectures.Levels 1–3 enable isolated optimization, while Level 4 tests whether kernels compose correctly in full model pipelines.Source: §3.2 · pp. 3–4

3.3 Architecture Coverage

• 96.2% (409/425) of HuggingFace Transformers architectures are covered without a new compute primitive by 46 representative architectures spanning 8 categories.Only 5 architectures require a genuinely new kernel and 2 require an external library; near-identical operators are consolidated to maximize coverage with few kernels.Source: §3.3 · p. 4

3.4 Interface-Compatible Design

• Module-level interfaces mirror corresponding production-library modules, enabling optimized kernels to transfer into vLLM, SGLang, or other systems with essentially copy-paste deployment.The framework supports both direct deployment as an inference engine and transfer deployment into an existing production framework.Source: §3.4 · pp. 4–5

3.5 Multi-GPU Communication Kernels

• FastKernels includes multi-GPU collectives, synchronization-sensitive overlap kernels, and communication tasks for tensor- and expert-parallel inference as first-class benchmark targets.These tasks capture communication–computation interactions that single-GPU benchmarks cannot evaluate and that can determine end-to-end throughput.Source: §3.5 · p. 5

• The three tiers increase in scope from comparing individual kernel outputs and runtimes, to full-model throughput and latency, to standardized evaluation sweeps.Tier 1 uses a registry derived from Tier 3 workloads, covering multiple models, batch regimes, tensor-parallel degrees, shapes, dtypes, and initialization arguments; data-dependent operators also replay captured real-model inputs.Source: §4 · p. 5
• Profiling is integrated through Nsight Compute for kernel analysis and Nsight Systems for end-to-end traces, with bottleneck extraction and MLflow-based lineage and history tracking.MLflow logs performance, correctness, and profiling data for comparing agentic runs and inspecting generated kernels.Source: §4 · p. 5

• Candidate kernels are evaluated by substituting them into end-to-end model executions and comparing each run with a production-baseline reference.This design measures kernel effects within complete model executions rather than in isolation.Source: §5 · p. 5
• MACROEVAL handles family-specific outputs—including tokens, embeddings, labels, rankings, trajectories, audio, and video—by calibrating correctness to a common [0, 1] scale.It uses an output-type-specific discrepancy before mapping results to calibrated correctness.Source: §5 · pp. 5–6
• It measures end-to-end speedup and macro-averages results across architecture families so no single architecture dominates the leaderboard.Families include LLMs, diffusion, vision, speech and audio, retrieval and recommendation, and robotics or world models.Source: §5 · p. 6
• Correctness thresholds are fixed before evaluation: gi is calibrated from reference-vs-reference nondeterminism, while fi is set from quality cliffs produced by deliberately wrong baselines.The released tolerances include FP32 (10−5, 10−3), FP16/BF16 (10−2, 10−2), FP8 (0.125, 0.125), and FP8-E5M2 (0.25, 0.25).Source: §5 · p. 6

• Macro-averaged correctness gives each architecture family equal influence, preventing overrepresented families such as LLMs from dominating the correctness score.Correctness is averaged over requests and items within each family before macro-family aggregation.Source: Appendix C · p. 6
• Invalid items receive no speedup credit, and macro-coverage captures whether agents produce valid kernels across architecture families.Invalidity includes crashes, hangs, shape or type errors, illegal memory access, NaNs, or correctness below a pre-specified threshold; macro-coverage is preferred when architecture balance matters.Source: Appendix C · pp. 6–7
• The default speed metric blends throughput and latency with λ = 0.5 and aggregates valid-item speedups using geometric means.Throughput and latency tradeoffs receive no universal credit when one improves while the other equivalently degrades; users may choose another pre-specified λ.Source: Appendix C · p. 7
• The leaderboard reports macro-geometric speedup, macro-calibrated correctness, and macro-coverage, alongside an equally weighted product as the default ranking.Families without valid items are excluded from speedup aggregation but remain penalized through macro-coverage, and bootstrap confidence intervals are reported where sample sizes permit.Source: Appendix C · p. 7
• Throughput-only and latency-only rankings expose deployment-specific tradeoffs, while all speedups are measured empirically against production references without hardware-limit estimates.Users can rank agents by individual metrics or task-specific combinations emphasizing performance, accuracy, coverage, or mixtures of these objectives.Source: Appendix C · p. 7

6.1 Reference Performance and Correctness

• 1.24× average throughput and 1.04× median throughput are achieved across 46 representative benchmarks against strongest production or upstream references.Performance is bimodal: mainstream LLM serving is near parity, while under-served architectures show larger gains; correctness remains exact or near-exact across most modalities.Source: §6 · p. 7, §6.1 · p. 8, Figure 3 · p. 8

6.2 Benchmarking Existing Kernel Agents

• 0.943×, 0.777×, and 0.527× aggregate speedups are achieved by Codex, KernelAgent, and Dr. Kernel, respectively, on FASTKERNELS.These results use production-shaped references and captured inputs across 88 L1/L2 target families.Source: §6.2 · p. 8, Table 1 · p. 9
• 1.16× versus torch-eager-wrapped families falls to 0.93× against vendor- or hand-written kernels, showing that production references eliminate many apparent gains.Wins cluster on operators lacking specialized production kernels, while production hot paths such as moe_align, linear, flashinfer, and allreduce fall below 1×.Source: §6.2 · pp. 8–9
• KernelAgent’s L2 correctness drops from 26/40 to 2/36, while Codex’s speedup falls from 1.03× to 0.84× and Dr. Kernel achieves 0/3 correct.L2 failures primarily violate production module contracts involving signatures, parameter layouts, prepared weights, or KV-cache behavior.Source: §6.2 · p. 9

Conclusion

• Even the strongest current agent falls below parity with production kernels, with gains concentrated on operators lacking specialized production implementations.This exposes the cost of benchmark–production misalignment and motivates agents whose benchmark gains translate into production throughput.Source: §7 · p. 9
• FASTKERNELS grounds evaluation in production inference through top-down tasks, production-matched interfaces, captured tensors, and deployed-baseline references.It is designed as a benchmark-as-framework for translating benchmark gains into production throughput.Source: §7 · p. 9

Limitations

• The headline agent study exercises L1+L2 only; coverage is an indicative upper bound, performance is measured on locked-clock H100 SXM5, and MACROEVAL choices encode value judgments.The L3/L4 harness is shipped but unused for headline numbers; rankings may differ on B200, MI300X, or consumer GPUs, and component metrics are reported for re-aggregation.Source: §7 · p. 9

Architecture composition

• Table 2 defines each architecture’s composition by reference checkpoint, default inference dtype, and counts of L1–L4 benchmark tasks.The levels are primitive, fused/composite, layer/block, and end-to-end tasks.Source: Table 2 · p. 12, A Benchmark Composition · p. 12

Task levels

• L4 contains exactly one end-to-end task for every architecture by construction.L1–L4 respectively represent primitive, fused/composite, layer/block-level, and end-to-end tasks.Source: Table 2 · p. 12

Implementation notes

• A zero task count indicates that the current implementation exposes no benchmark task at that level.Quantized configurations retain the underlying architecture’s high-level task graph while using distinct quantized kernels and execution paths.Source: A Benchmark Composition · p. 12

Interface compatibility

• Matching FastKernels and vLLM constructor signatures and forward arguments enable direct module substitution, illustrated with attention modules.The example pairs FastKernels Attention with vLLM LlamaAttention using config, cache_config, quant_config, positions, hidden_states, kv_cache, and attn_metadata.Source: B Interface Compatibility Examples · p. 13
• The same interface-compatible design extends beyond attention to MLP blocks, normalization layers, and MoE routing modules.The MoE example matches config, hidden_states, and router_logits arguments between corresponding modules.Source: B Interface Compatibility Examples · p. 13

Threshold calibration procedure

• MACROEVAL sets per-item correctness bands from reference self-nondeterminism and unacceptable-quality thresholds at family-specific quality cliffs, with values frozen across submissions.The default validity threshold is τ_i = 1.0, and the released manifest records per-dtype tolerances and family-level thresholds for task-typed outputs.Source: C MACROEVAL Calibration and Sensitivity · p. 13, Table 3 · p. 14

Sensitivity analysis

• A 0.25× tolerance scale lowers Codex’s combined geomean from 0.943× to 0.909×, whereas loosening above 1× leaves the headline unchanged.The tighter threshold removes 9 L1 and 5 L2 target families that passed the default checker; KernelAgent’s 0.777× and Dr. Kernel’s 0.527× are upper bounds under stricter thresholds.Source: C MACROEVAL Calibration and Sensitivity · p. 14, Table 4 · p. 14

Harness-gap accounting

• Codex remains strongest under both permissive and punitive harness-gap accounting, with KernelAgent intermediate and Dr. Kernel weakest.Blocked targets reduce coverage but are excluded from the default speedup geomean; alternative policies bracket the contribution of harness coverage versus agent quality.Source: C MACROEVAL Calibration and Sensitivity · p. 14, Table 5 · p. 15

Bootstrap confidence intervals

• 95% confidence intervals confirm Codex’s combined-level ordering over KernelAgent, but Dr. Kernel cannot be statistically separated from KernelAgent because only eight correct kernels support its wide interval.Codex’s interval is [0.868, 1.031]× versus KernelAgent’s [0.633, 0.963]×; Dr. Kernel’s interval is [0.171, 1.049]×.Source: C MACROEVAL Calibration and Sensitivity · p. 15, Table 6 · p. 15

Input-dependent routing

• Only 4 of 16 hot experts overlap between random token IDs and real WildChat requests, while random tensors produce near-uniform routing.The Qwen3-VL-30B-A3B-Instruct-FP8 first MoE layer uses 128 experts with top-8 routing.Source: Figure 4 · p. 15, D Case Study: Captured Inputs Matter… · p. 15
• Random tensors yield Gini 0.232, whereas random token IDs and real requests yield Gini 0.480 and 0.339, respectively.Random token IDs over-concentrate load and select different hot experts from real WildChat requests.Source: D Case Study: Captured Inputs Matter… · p. 15

Capture-and-replay

• Capture-and-replay rewards data-dependent kernels on tensors produced by production requests rather than on synthetic control flow.This path targets the actual execution patterns induced by production inputs.Source: D Case Study: Captured Inputs Matter… · p. 16

Hardware

• All experiments use NVIDIA H100 SXM5 80GB GPUs, with 8-GPU NVLink nodes for multi-GPU tests and locked 1,593 MHz clocks.Single-GPU experiments use one GPU; multi-GPU experiments use eight H100 GPUs.Source: E Experimental Details · p. 16

Serving benchmark

• End-to-end serving throughput uses ShareGPT prompts with batch size 32, 512-token inputs, 128-token outputs, three runs, and 10 warmups.Source: E Experimental Details · p. 16

Perplexity evaluation

• Perplexity is evaluated on a held-out 10K-token WikiText-103 subset using sliding windows with stride equal to each model’s maximum context length.Source: E Experimental Details · p. 16

Agent configuration

• Dr. Kernel, KernelAgent, and Codex use default published configurations, while the internal Codex agent uses profiled multi-turn optimization for 10 iterations.The internal agent uses NCU profiling feedback, temperature sampling, and the highest available OpenAI Codex CLI reasoning tier.Source: E Experimental Details · p. 16

Aggregation conventions

• Per-row speedups are arithmetic means across public scenarios, or a single ratio when only one scenario is reported.A dash indicates that the current README does not publish a numeric throughput table for that architecture.Source: F Reference Performance and… · pp. 16, 18

Benchmark coverage

• Tables 7 and 8 cover representative FASTKERNELS architectures using public benchmark checkpoints or workloads, comparison references, throughput ratios, and alignment values.The tables identify the production or upstream library used as each comparison target and report a scalar alignment value when publicly available.Source: F Reference Performance and… · p. 16, Table 7 · p. 17, Table 8 · p. 18

Interpretation

• Values below 1 indicate slower execution than the reference, while “n/a” indicates that no single public scalar alignment summary is reported.The cross-architecture summary in Section 6.1 follows the same arithmetic-mean convention, whereas MACROEVAL uses geometric means.Source: F Reference Performance and… · pp. 16, 18

Coverage audit

• 96.2% (409/425) of HuggingFace Transformers modeling entries are covered without a new compute primitive, though this headline is an indicative upper bound because covered verdicts were sampled rather than exhaustively re-audited.Five architectures—mra, reformer, rwkv-v4, xlstm, and yoso—require genuinely new kernels; two require external libraries, and nine use PyTorch for one operation.Source: G HuggingFace Transformers Coverage · p. 18

Mapping methodology

• Table 9 provides the per-class kernel mapping, with wiring-only and duplicate-signature classes collapsed unless they introduce otherwise uncovered kernels.The mapping was produced by 16 LLM agents reading HuggingFace source and kb-nano files, with reproducibility scripts in the audit branch.Source: G HuggingFace Transformers Coverage · p. 18, Table 9 · p. 19

Uncovered primitives

• Autoformer is a partial match because AutoformerAttention lacks a kb-nano kernel for FFT-based autocorrelation and top-k delay aggregation.Its missing primitive uses rfft/irfft and replaces canonical attention.Source: G HuggingFace Transformers Coverage · p. 20

Representative mappings

• The mapping covers diverse architectures through composable FastKernels primitives, including Bamba’s attention and Mamba mixer, BitNet’s specialized kernels, and multimodal models such as BLIP and CLAP.Representative entries map model components to reusable L1 and L2 kernels rather than requiring architecture-specific implementations.Source: G HuggingFace Transformers Coverage · pp. 20, 21, 23

Language and multimodal architectures

• CLIP, CLIPSeg, CLVP, CodeGen, Cohere, and ConvBert components are mapped to attention, MLP, embedding, normalization, convolution, and decoder kernels.The mappings include CLIP and CLIPSeg task heads, CLVP decoder components, CodeGen and Cohere causal-language models, and ConvBert task heads.Source: G HuggingFace Transformers Coverage · p. 23

Additional model families

• ConvNext, ConvNextV2, CTRL, CVT, CWM, and CPM-Ant are represented with convolutional, attention, recurrent-position, normalization, and model-wiring kernels.Several entries also expose per-task heads, including image classification for ConvNext, ConvNextV2, and CVT, and sequence classification for CTRL.Source: G HuggingFace Transformers Coverage · p. 24

Detection, audio, and Data2Vec architectures

• Conditional DETR, D-FINE, DAB-DETR, and Data2Vec variants include detection, segmentation, classification, speech, and language task heads built from shared kernels.The listed implementations cover deformable attention, convolutional encoders, vision backbones, audio modules, text heads, and multiple task-specific wiring configurations.Source: G HuggingFace Transformers Coverage · pp. 24–25

Mixture-of-experts and transformer architectures

• DBRX, DeBERTa, Decision Transformer, and DeepSeek V2–V4 entries cover mixture-of-experts, grouped GEMM, MLA attention, rotary embeddings, and model-level wiring.DeepSeek implementations include V2, V3, and V4 model compositions with expert routing and task-head configurations where specified.Source: G HuggingFace Transformers Coverage · pp. 25–26

Multimodal and specialized models

• Multimodal and specialized coverage spans DeepSeek-VL, EdgeTAM image and video models, EoMT variants, EfficientLoFTR, EfficientNet, and Encodec through mapped convolutional, attention, embedding, and recurrent kernels.EdgeTAM video coverage includes memory fusion, mask downsampling, perceiver resampling, prompt encoding, and mask decoding components.Source: G HuggingFace Transformers Coverage · pp. 26, 28, 29

Vision and detection

• Coverage includes vision and detection models such as DeiT, DINOv2, DINOv3, DETR, Deformable DETR, Deformable DETR variants, and Depth Anything with mapped model components and task heads.The listed implementations reuse primitives including convolutions, attention, MLPs, normalization, activation, pooling, and task-specific linear heads.Source: G HuggingFace Transformers Coverage · pp. 26–27

Additional architectures

• Additional architectures include Depth Pro, DPT, EfficientNet, encoder-decoder models, DPR, and universal-segmentation models, with their listed backbones, heads, and task wiring implemented.The catalog covers depth estimation, semantic segmentation, image classification, span prediction, and universal segmentation task interfaces.Source: G HuggingFace Transformers Coverage · pp. 27–29

Language models

• The cited coverage includes Dia, DiffLlama, DistilBERT, Doge, and Dots1, with documented attention, normalization, MLP, embedding, routing, and MoE implementations across these models.DiffLlama and Doge list task-head wiring for sequence, question-answering, token, and sequence-classification tasks, respectively. Ernie4_5 MoE lists grouped GEMM, Mixtral-style routing, and shared-expert MoE primitives.Source: G HuggingFace Transformers Coverage · pp. 27–29

Coverage limitation

• Emu3 is marked incomplete because Emu3Attention is a missing primitive, despite listed implementations for its RMSNorm, MLP, VQ-VAE, rotary embedding, and related components.The missing attention is specified as L2/attention.py with Llama-style GQA, RoPE, and KV cache behavior.Source: G HuggingFace Transformers Coverage · p. 29

Architecture coverage

• FastKernels implements representative components and task heads for architectures including ESM, EuroBERT, Exaone, Falcon, Flaubert, Flava, FocalNet, Funnel, Gemma, and Gemma3.The listed implementations span attention, MLP, normalization, embeddings, vision modules, mixture-of-experts layers, and per-task model wiring.Source: G HuggingFace Transformers Coverage · pp. 30–32
• FastKernels covers multimodal and specialized architectures through Exaone4_5 vision components, Florence2 vision backbones, Gemma3 multimodal projection, and a Gemma3n audio Conformer attention component.Source: G HuggingFace Transformers Coverage · pp. 30, 32

Coverage limitations

• FastSpeech2Conformer remains incomplete because FastSpeech2ConformerDurationPredictor is missing, requiring a direct linear regression path and inference-time clamp(round(exp() - 1)).The missing predictor wires FastSpeech2ConformerPredictorLayer but lacks the corresponding primitive implementation.Source: G HuggingFace Transformers Coverage · p. 31
• FNet remains incomplete because FNetBasicFourierTransform has no kb-nano kernel, leaving torch.fft.fftn or TPU DFT matmul unsupported.Other FNet embeddings, transformer layers, prediction heads, pooling, and task-head wiring are listed as implemented.Source: G HuggingFace Transformers Coverage · p. 32

Gemma and GLM coverage

• Gemma4 entries cover audio, vision, text, multimodal, assistant, and expert-routing components through shared attention, convolution, embedding, normalization, and MoE kernels.The listed implementations include Gemma4AudioAttention, Gemma4VisionAttention, Gemma4TextAttention, Gemma4TextExperts, Gemma4MultimodalEmbedder, and Gemma4TextModel wiring.Source: G HuggingFace Transformers Coverage · p. 33
• GLM-family coverage spans text, MoE, multimodal, image, OCR, and speech models, with implementations for attention, rotary embeddings, MLPs, routing, vision modules, and encoder wiring.The inventory includes GlmForCausalLM and task heads, Glm4Moe models, Glm4v and Glm4vMoe vision components, GlmImage modules, GlmOcrModel, and GlmAsrEncoder.Source: G HuggingFace Transformers Coverage · pp. 33–34

Other vision coverage

• Additional vision coverage includes GLPN, Got-OCR2, and Grounding DINO components such as patch embeddings, efficient attention, deformable attention, feature fusion, depth-estimation heads, and frozen normalization.These entries map the architectures to convolution, linear, normalization, activation, interpolation, and deformable-attention kernels.Source: G HuggingFace Transformers Coverage · pp. 34–36

GPT coverage

• GPT-family entries implement GPT-2, GPT-BigCode, GPT-Neo, GPT-NeoX, GPT-NeoX-Japanese, GPT-OSS, and GPT-J modules, including attention, KV caching, MLPs, embeddings, model wiring, and task heads.The inventory explicitly lists task heads for sequence classification, token classification, and question answering where supported.Source: G HuggingFace Transformers Coverage · p. 35

Granite coverage

• Granite coverage includes standard, speech, MoE, hybrid-MoE, and shared-MoE variants, with kernels for attention, conformer operations, routing, expert computation, Mamba mixing, normalization, and model assembly.GraniteSpeechCTCEncoder and multiple GraniteMoe model entries combine these components into end-to-end architecture wiring.Source: G HuggingFace Transformers Coverage · pp. 35–36

Vision and multimodal architectures

• Vision and multimodal coverage spans GroupViT, Hiera, Idefics variants, InstructBLIP, InternVL, Janus, and Kosmos, using convolution, embedding, attention, MLP, normalization, and resampling kernels.Janus additionally includes VQ-VAE components, while Idefics and InstructBLIP include separate vision and language or Q-Former modules.Source: G HuggingFace Transformers Coverage · pp. 36–38

Language and mixture-of-experts models

• Coverage includes language-model families such as Helium, Hunyuan dense/MoE, HY-V3, Jamba, and JetMoE, spanning attention, normalization, rotary embeddings, MLPs, routing, and expert kernels.Several entries also provide complete model wiring and sequence-classification or causal-language-model task heads.Source: G HuggingFace Transformers Coverage · pp. 36–38

Specialized and document architectures

• The inventory also covers specialized models such as HGNetV2, IJepa, ImageGPT, Informer, IBERT, LayoutLM variants, and GroundingDINO, with task-specific heads for image, text, document, detection, and forecasting workloads.These entries reuse common kernels while adding architecture-specific components such as deformable attention, patch embeddings, probabilistic sparse attention, and layout-aware transformer blocks.Source: G HuggingFace Transformers Coverage · pp. 36–39

Speech architectures

• Speech and sequence-processing architectures include HuBERT, Kyutai Speech-to-Text, and LASR, covering convolutional front ends, attention, rotary embeddings, feed-forward layers, normalization, and task heads.HuBERT and LASR entries explicitly include CTC or classification-oriented model wiring and heads.Source: G HuggingFace Transformers Coverage · pp. 37, 39

Multimodal and decoder architectures

• The catalog covers LayoutLMv3, LED, LeViT, LFM2, LightGlue, LILT, and LLaMA-family components, including attention, embeddings, normalization, MLPs, multimodal modules, and task heads.LFM2 and LFM2-MoE include short convolutions, attention, grouped experts, routing, and shared-expert modules; LLaMA and LLaMA 4 include causal-language-model, MoE, and vision components.Source: G HuggingFace Transformers Coverage · pp. 39–40

Multimodal and encoder architectures

• Llava, Llava-Next, Llava-Next-Video, Llava-OneVision, Longformer, LongT5, Luke, and LXMERT entries cover multimodal projectors, sparse or local attention, encoder-decoder stacks, entity embeddings, and multiple task heads.The listed task-head counts include four for Longformer, eight for Luke, and two for LXMERT.Source: G HuggingFace Transformers Coverage · pp. 40–41

Sequence, vision, and segmentation architectures

• The catalog includes M2M-100, Mamba, Mamba2, Marian, MarkupLM, MaskFormer, and MaskFormer-Swin implementations spanning sequence modeling, convolutions, encoder-decoder attention, detection, segmentation, and vision backbones.MaskFormer includes instance-segmentation wiring and components such as convolution, group normalization, interpolation, and mask attention; Mamba and Mamba2 include causal convolutions and mixer modules.Source: G HuggingFace Transformers Coverage · p. 41

Additional modality and task coverage

• Additional entries cover MBART, Megatron-BERT, MGP-STR, Mimi, MiniCPM-V4.6, and MiniMax, combining text, vision, audio, pooling, routing, mixture-of-experts, and task-specific model wiring.Megatron-BERT lists six task heads, while MGP-STR lists scene-text recognition and MiniMax lists three classification, token-classification, or question-answering heads.Source: G HuggingFace Transformers Coverage · pp. 41–42

Mistral-family architectures

• The portion concludes with Mistral, Mistral3, and Mixtral support, including attention, rotary embeddings, normalization, multimodal projection, top-k routing, and fused or expert mixture modules.Mistral and Mixtral each list three task heads for sequence classification, token classification, and question answering.Source: G HuggingFace Transformers Coverage · p. 42

Vision coverage

• Vision coverage includes MobileNetV1, MobileNetV2, MobileViT, MobileViTV2, and OmDet-Turbo components, with classification, segmentation, and deformable-attention support.The listed implementations use convolution, normalization, activation, pooling, transformer, and deformable-attention kernels.Source: G HuggingFace Transformers Coverage · pp. 43, 46

Language-model coverage

• FastKernels covers numerous language-model families, including ModernBERT, MPNet, MPT, MT5, NomicBERT, Olmo variants, Nemotron variants, and Nystromformer.These implementations combine reusable attention, embedding, normalization, MLP, routing, and task-head kernels, with several models supporting multiple downstream tasks.Source: G HuggingFace Transformers Coverage · pp. 43–45

Speech and generative-model coverage

• Speech, multimodal, and generative-audio coverage includes Moonshine, Moshi, MusicGen, MusicGen Melody, and MVP models with streaming, attention, convolution, and decoder components.Moonshine additionally lists streaming encoder and decoder components, while MusicGen and MVP expose decoder and task-head implementations.Source: G HuggingFace Transformers Coverage · pp. 43–44

Coverage limitation

• MRA is unsupported because MraSelfAttention requires a custom mra2_attention CUDA kernel with no kb-nano equivalent.Other MRA modules and four task heads are listed, but the missing primitive prevents complete coverage.Source: G HuggingFace Transformers Coverage · p. 44

MoE and hybrid coverage

• Mixture-of-experts and hybrid architectures are represented by Nemotron-H, NLLB-MoE, Olmoe, and Olmo-Hybrid, including grouped GEMM, routing, recurrent, and gated-delta components.The implementations cover both sparse expert computation and hybrid attention or state-space-style mechanisms.Source: G HuggingFace Transformers Coverage · pp. 45–46

Vision and segmentation

• Coverage includes vision and segmentation architectures such as OneFormer, Ovis2, OWLv2, OwlViT, and Perceiver, with deformable attention, convolutional, embedding, normalization, and task-head components.OneFormer includes pixel-decoder and transformer components; Perceiver exposes eight task heads spanning classification, optical flow, and multimodal autoencoding.Source: G HuggingFace Transformers Coverage · pp. 46, 48

Language models

• Language-model coverage includes OpenAI GPT, OPT, Persimmon, Phi, and Phi-3, using attention, MLP, positional-encoding, KV-cache, normalization, embedding, and task-head kernels.The listed models support sequence- and token-level tasks, while OpenAIPrivacyFilter adds mixture-of-experts, top-k softmax, RMS normalization, and rotary-embedding components.Source: G HuggingFace Transformers Coverage · pp. 46, 48

Multimodal and audio/video

• Multimodal coverage spans PaddleOCR, PaliGemma, Phi-4 multimodal, and audio/video encoders, combining vision embeddings, SigLIP attention and MLPs, convolutions, rotary embeddings, normalization, and contrastive heads.Phi-4 multimodal specifically includes separate vision, audio, and feature-embedding components, including depthwise-separable convolutions and audio subsampling modules.Source: G HuggingFace Transformers Coverage · pp. 47–48

Speech, time series, and encoder-decoder models

• The catalog also covers speech, time-series, and encoder-decoder models including Parakeet, PatchTSMixer, PatchTST, Pegasus, Pegasus-X, and Perceiver, with convolutional, attention, normalization, positional, and task-head implementations.PatchTSMixer and PatchTST expose classification and regression-related heads, while Parakeet provides convolutional subsampling and encoder attention components.Source: G HuggingFace Transformers Coverage · pp. 47–48

Transformer and multimodal coverage

• Coverage includes Phimoe, PI0, Pix2Struct, Pixio, Pixtral, PLBart, PoolFormer, and Pop2Piano components with architecture-specific attention, embedding, MLP, normalization, and task-head wiring.Examples include Phimoe mixture-of-experts routing, Pixtral vision attention, PLBart sequence classification, and PoolFormer image classification.Source: G HuggingFace Transformers Coverage · p. 49

Formula, vision, and OCR coverage

• PP-FormulaNet, PPLCNet, PPLCNetV3, and PPOCRV5 variants extend coverage to formula recognition, image classification, object detection, and text recognition.These models use reusable convolution, attention, pooling, activation, normalization, and task-head kernels across multiple OCR and vision architectures.Source: G HuggingFace Transformers Coverage · pp. 50–51

Depth, language, and vision coverage

• Additional coverage includes PromptDepthAnything depth estimation, ProphetNet sequence modeling, and PVT vision components built from convolution, attention, embedding, dropout, and normalization kernels.PromptDepthAnything includes depth-estimation heads and reassembly layers, while ProphetNet includes n-gram self-attention and decoder wiring.Source: G HuggingFace Transformers Coverage · p. 51

Text architectures

• Qwen2 and Qwen3 text families cover attention, MLP, normalization, rotary embeddings, model wiring, and sequence, token, and question-answering heads.The catalog includes Qwen2, Qwen2-MoE, Qwen3, Qwen3-MoE, and Qwen3-Next implementations.Source: G HuggingFace Transformers Coverage · pp. 51–53

Audio and omni-modal architectures

• Multimodal audio and omni-modal entries cover audio attention and encoders alongside speech, vision, diffusion-transformer, convolutional, and codec components.The listed Qwen2.5-Omni, Qwen2-Audio, and Qwen3-Omni-MoE modules use reusable linear, attention, convolution, normalization, embedding, and routing kernels.Source: G HuggingFace Transformers Coverage · pp. 51–53

Vision-language architectures

• Qwen2.5, Qwen2, and Qwen3 vision-language architectures include visual patch embeddings, rotary embeddings, attention, MLPs, multimodal projection, and text-model components.Coverage spans Qwen2.5-VL, Qwen2-VL, Qwen3-VL, and their vision-language model wiring.Source: G HuggingFace Transformers Coverage · pp. 51–53

Mixture-of-experts architectures

• Mixture-of-experts coverage includes expert grouped GEMMs, top-k routers, sparse MoE blocks, decoder layers, and multimodal model wiring.These components appear across Qwen2-MoE, Qwen3-MoE, Qwen3.5-MoE, Qwen3-Omni-MoE, and Qwen3-VL-MoE families.Source: G HuggingFace Transformers Coverage · pp. 52–54

Specialized architectures

• The catalog also records emerging and specialized families, including Qwen3.5, Qwen3.5-MoE, Qwen3-Next, Qwen3-VL-MoE, and RecurrentGemma components.Listed implementations include gated delta networks, gated RMS normalization, vision modules, recurrent normalization, and rotary embeddings.Source: G HuggingFace Transformers Coverage · pp. 52–54

Unsupported architectures

• Reformer is unsupported because LSHSelfAttention is missing as a bespoke LSH-attention primitive.The remaining Reformer components are mapped to available tensor, embedding, dropout, linear, normalization, activation, and attention kernels.Source: G HuggingFace Transformers Coverage · p. 54

Covered architectures

• The covered architectures include RegNet, RemBERT, ResNet, RoBERTa and RoFormer, RT-DETR variants, SAM families, and SAM3 tracker components built from reusable convolution, attention, normalization, embedding, and task-head modules.The listed implementations span image classification, language understanding, object detection, mask generation, video processing, and related backbone/model paths.Source: G HuggingFace Transformers Coverage · pp. 54–55

Unsupported architectures

• RWKV is unsupported because RwkvLinearAttention depends on a bespoke wkv_cuda linear-attention scan not directly compatible with the available RWKV-7 recurrence.Other listed RWKV modules use linear, squared-ReLU, sigmoid, normalization, and embedding kernels.Source: G HuggingFace Transformers Coverage · p. 55

Limitations

• Exact coverage remains incomplete for SeamlessM4T and SeamlessM4T-v2 because their conformer self-attention requires manual softmax-matmul attention with optional positional handling.The corresponding missing primitives are reported for both model variants.Source: G HuggingFace Transformers Coverage · p. 57

Model coverage

• FastKernels covers diverse language and vision architectures, including Seed-OSS, SegFormer, SegGPT, SigLIP, SigLIP2, SLANet, SLANeXt, SmolLM3, SmolVLM, Solar, Splinter, SqueezeBERT, StableLM, and StarCoder2.Implementations combine reusable attention, normalization, convolution, embedding, MLP, rotary, mixture-of-experts, and task-head components.Source: G HuggingFace Transformers Coverage · pp. 58–59

Task coverage

• FastKernels provides explicit task wiring for several architectures, including three Seed-OSS and SmolLM3 tasks, two SEW and SEW-D tasks, and four SqueezeBERT tasks.The listed tasks include classification, token classification, question answering, multiple choice, and CTC heads.Source: G HuggingFace Transformers Coverage · pp. 58–59

Limitations

• Additional unsupported attention primitives affect SEW-D and SpeechT5, requiring disentangled relative-position attention or BART-style manual bmm-softmax-bmm attention.SEW-D also requires XSoftmax, while SpeechT5 has only a closest match through the Whisper attention implementation.Source: G HuggingFace Transformers Coverage · pp. 58–59

Speech coverage

• Speech coverage includes Speech2Text, SEW, SEW-D, SpeechT5, and SeamlessM4T variants, with encoder, decoder, convolutional, attention, vocoder, and task-head components represented.The listed implementations reuse primitives such as conv1d, linear layers, Whisper-style attention, normalization, activations, embeddings, and HiFi-GAN modules.Source: G HuggingFace Transformers Coverage · pp. 57–59

• Implemented Swin, Swin2SR, SwinV2, SuperPoint, SwiftFormer, TableTransformer, TextNet, Timesformer, and UperNet architectures with convolutional, attention, normalization, pooling, and task-head components.Swin and SwinV2 each include three task heads, while SuperPoint, SwiftFormer, TableTransformer, TextNet, Timesformer, and UperNet expose their listed architecture-specific heads.Source: G HuggingFace Transformers Coverage · pp. 60–61
• Implemented T5, T5Gemma, T5Gemma2, SwitchTransformers, UMT5, UDOP, TAPAS, Starcoder2, and VaultGemma using shared attention, MLP, normalization, embedding, and linear-kernel components.Task-head coverage ranges from zero for SwitchTransformers, UDOP, and VaultGemma to three for T5 and UMT5; Starcoder2 includes sequence- and token-classification heads.Source: G HuggingFace Transformers Coverage · pp. 60–62
• Implemented time-series and audio-generation models including TimeSeriesTransformer, TimesFM, TimesFM2.5, TrOCR, TVP, and UnivNet with their listed model components and wiring.TimeSeriesTransformer, TimesFM, TimesFM2.5, TrOCR, TVP, and UnivNet have zero task heads in the supplied inventory, while TrOCR and TVP include attention and embedding components.Source: G HuggingFace Transformers Coverage · pp. 61–62
• The inventory marks timm_backbone and timm_wrapper as unsupported, with no task heads or implemented model components listed for either.Both entries are explicitly marked with a cross and task heads (0).Source: G HuggingFace Transformers Coverage · p. 62
• Implemented UniSpeech and UniSpeech-SAT with convolutional feature projections, attention, feed-forward, normalization, quantization, and multiple speech-task heads.UniSpeech lists three task heads, whereas UniSpeech-SAT lists four, including CTC, sequence classification, audio-frame classification, and X-vector tasks.Source: G HuggingFace Transformers Coverage · p. 62

Specialized multimodal architectures

• Specialized coverage includes VibeVoice acoustic-tokenizer and ASR feed-forward, causal-convolution, and encoder-stem components, alongside VideoLLaMA3 vision operators and a VideoLLaVA multimodal projector.Source: G HuggingFace Transformers Coverage · p. 63

Video and multimodal architectures

• Video architectures include VideoMAE, VideoMT, ViLT, Vivit, VJEPA2, and related multimodal models with implemented embeddings, attention, MLP, projection, convolution, normalization, and model-wiring components.VideoMAE, VideoMT, ViLT, and Vivit include task heads for video classification, segmentation, retrieval, question answering, and token classification; VJEPA2 includes predictor and attentive-pooler components.Source: G HuggingFace Transformers Coverage · pp. 63–64

Task-head coverage

• Across the listed models, task-head implementations are represented as model wiring plus per-task linear heads for image, video, audio, visual-language, and multimodal prediction tasks.Examples include ViLT’s four heads, VideoMAE’s video-classification head, Vivit’s video-classification head, and the four-head configurations of Wav2Vec2-family and WavLM models.Source: G HuggingFace Transformers Coverage · pp. 63–65

Vision architectures

• Vision coverage includes ViT, ViT-MAE, ViT-MSN, ViTDet, ViTPose, VisualBERT, and related models using convolutional patch embeddings, encoder attention, feed-forward blocks, normalization, and task-head wiring.ViT and ViT-MSN provide image-classification heads, while VisualBERT provides multiple-choice, question-answering, and visual-reasoning heads.Source: G HuggingFace Transformers Coverage · pp. 63–64

Speech and audio architectures

• Speech and audio coverage includes VITS, Voxtral, Voxtral Realtime, Wav2Vec2, Wav2Vec2-BERT, Wav2Vec2-Conformer, and WavLM components built from convolutions, attention, projections, embeddings, normalization, and feed-forward layers.The Wav2Vec2, Wav2Vec2-BERT, Wav2Vec2-Conformer, and WavLM entries each expose four task heads for CTC, sequence classification, audio-frame classification, and X-vector prediction.Source: G HuggingFace Transformers Coverage · pp. 64–65

Implemented architectures

• FastKernels includes implementations for XLM, XLM-RoBERTa, XLM-RoBERTa-XL, XMod, YOLOS, Youtu, Zamba, Zamba2, and ZoeDepth components and task heads.The inventory covers attention, embeddings, normalization, MLP, convolutional, recurrent, and task-specific modules across these families.Source: G HuggingFace Transformers Coverage · pp. 66–67

Task coverage

• XLM and XLM-RoBERTa support includes multiple task heads for sequence classification, question answering, token classification, and multiple choice.XLM lists five task heads, while XLM-RoBERTa and XLM-RoBERTa-XL list four each.Source: G HuggingFace Transformers Coverage · p. 66

Unsupported architectures

• XLNet is marked missing because its relative two-stream attention requires content, position, and segment scoring, rel_shift, and two query streams.Other XLNet components and five task heads are listed, but the missing XLNetRelativeAttention primitive limits the architecture.Source: G HuggingFace Transformers Coverage · p. 66
• xLSTM is marked missing because its xLSTMLayer requires gated projections, soft_cap, xLSTMBackend, multi-head layer normalization, and gated output processing.Supporting xLSTM primitives such as RMSNorm, LSTM backend, feed-forward, embedding, and sigmoid modules are listed separately.Source: G HuggingFace Transformers Coverage · p. 66
• YOSO is marked missing because sparse attention depends on a custom CUDA LSH-cumulation kernel, despite surrounding model and task-head components being listed.The missing YosoSelfAttention primitive includes q/k/v linear projections, YosoCumulation or YosoLSHCumulation, and optional Conv2d.Source: G HuggingFace Transformers Coverage · pp. 66–67