PaperlayerView exact revision on arXiv

Source-linked AI summary

LLM4LLM: Bridging Kernel Benchmarks and Real Deployment via Closed-Loop Agentic Optimization

Hui Zeng, Pengfei Yang, Yanxin Chen, Fusong Ju, Xinran Wei

arXiv:2608.21836v1cs.AI

TL;DR

Standalone kernel benchmarks provide only a proxy for deployment behavior because integrated inference can change performance, safety, and phase behavior. LLM4LLM profiles a target workload, constructs phase-aware tasks, searches with episodic experience, and validates patches in-model. Across evaluated workloads, it improves end-to-end latency on both A100 and H100 GPUs, with supporting gains on KernelBench Level 2.

  • Problem

    The benchmark-to-deployment gap is the mismatch between isolated qualification and deployment-time acceptance for kernels used in language-model inference.

  • Method

    LLM4LLM profiles a target inference workload, extracts phase-aware tasks, performs experience-guided episodic search, and accepts patches through model-integrated validation.

  • Results

    LLM4LLM improves end-to-end latency across evaluated language-model workloads, achieving 3.91×/6.98× geometric-mean speedups on A100/H100 and 2.745×/2.628× GeoMean kernel speedups on KernelBench Level 2.

  • Takeaways & Limitations

    Deployment-aware profiling, phase-aware constraints, and target-runtime validation align generated-kernel search with the execution context determining inference performance.

  • Takeaways & Limitations

    LLM4LLM currently targets single-GPU inference and requires renewed profiling and acceptance checks when prompts, decode lengths, batching behavior, or model configurations differ substantially.

Abstract

from arXiv · show

Large language models have become increasingly capable agents for low-level code and kernel optimization, but isolated kernel benchmarks provide only a proxy for the deployment behavior that matters in language-model inference. We identify a benchmark-to-deployment gap: candidate kernels that appear correct and fast in standalone harnesses can exhibit different performance, safety, or phase behavior after integration into a real inference workload. We introduce LLM4LLM, a deployment-aware closed-loop optimization framework that starts from a target inference script, extracts phase-aware optimization tasks, searches with an experience-guided episodic agent, and accepts patches through in-model validation. Across ten language-model inference workloads on A100 and H100 GPUs, LLM4LLM improves end-to-end latency for every evaluated model, achieving 3.91$\times$/6.98$\times$ geometric-mean speedups on A100/H100; as supporting kernel-level evidence, it also attains up to 2.745$\times$ GeoMean speedup on KernelBench Level 2.

1 Introduction

LLM-based kernel optimization faces a benchmark-to-deployment gap because standalone measurements can diverge from behavior in real inference workloads. LLM4LLM closes this loop with phase-aware search and in-model validation, producing end-to-end and kernel-level gains.

  • Standalone kernel latency is only an intermediate signal because deployment also depends on phase semantics, cache state, dispatch overhead, memory residency, and end-to-end latency.
  • Prefill and decode differ in shapes, cache states, memory-access patterns, and latency sensitivity, so optimization transfer between phases can be uneven.
  • LLM4LLM profiles a user-provided inference script, extracts deployable phase-aware tasks, searches through experience-guided episodic optimization, and accepts patches after in-model validation.
  • The benchmark-to-deployment gap is characterized as a taxonomy of cases where isolated qualification diverges from deployment-time acceptance.
  • 3.91×/6.98× geometric-mean speedups on A100/H100 are reported for end-to-end inference, alongside 2.745×/2.628× GeoMean kernel speedups on KernelBench Level 2.

2 The Benchmark-to-Deployment Gap

The benchmark-to-deployment gap arises when isolated correctness and speed do not predict behavior after integration into a target model. It appears through context-dependent performance, runtime-safety failures, and phase-specific specialization, motivating closed-loop evaluation around the real workload.

  • An optimized convolution candidate can be faster in a standalone harness yet make the integrated model slower end to end.
  • Warm-cache execution inside the model can change candidate rankings because preceding layers supply activations with different cache state and producer-consumer locality.
  • A latent out-of-bounds access may pass isolated correctness under a fresh allocator state but trigger a deployment-time CUDA error in the full model.
  • The resulting evaluation concerns are performance transfer, runtime validity, and phase-correct behavior, distinguishing isolated qualification from deployment-time acceptance.
  • Evaluation proceeds from standalone qualification and search-time validation to deployment-time acceptance and final end-to-end measurement under the full workload.
  • A kernel specialized for prefill attention may be inapplicable to decode attention, preventing kernel-level gains from transferring across generation phases.

3 Method

LLM4LLM optimizes kernels in the context of real inference deployments by extracting phase-aware tasks, searching with episodic experience, and validating patches inside the target model.

  • Phase-aware task extraction: The framework profiles a target inference script and selects deployable module instances whose replacement can affect end-to-end latency.Deployable modules have stable call boundaries, reproducible tensors, and fallback implementations.
  • Phase-aware task extraction: Phase-aware tasks serialize representative inputs, references, shapes, hardware scope, and phase tags, separating prefill and decode when their execution semantics diverge.A dispatch template later reassembles phase-specialized candidates into one patched module.
  • Experience-guided search: Candidate kernels are generated, compiled, checked for correctness, measured for latency, and diagnosed through repeated generate-verify-decide episodes.Each episode records code, compiler diagnostics, numerical errors, runtime failures, and latency measurements.
  • Experience-guided search: At episode boundaries, useful validation evidence is distilled into compact experience while transient repair history is archived before search restarts from the task specification.The retained record includes constraints, failure signatures, phase predicates, and the best observed performance regime.
  • Deployment-time acceptance and patching: Deployment-time acceptance inserts candidates into the target model and evaluates in-context correctness, runtime compatibility, shape coverage, phase compatibility, and latency.Accepted candidates become patches, rejected candidates feed subsequent search, and a minimum deployment improvement δ is required for acceptance.

4 Evaluation

LLM4LLM is evaluated on language-model workloads and KernelBench using deployment-aware latency measurements and scope-matched baselines. It improves end-to-end performance across model families while ablations show benefits from sampling, restart, and deployment-aware validation.

  • Experimental Setup: Experiments span transformer, Mamba-family, and RecurrentGemma workloads on A100 and H100 GPUs, measuring end-to-end latency and speedup over eager execution.Candidates are validated after insertion into the target model and measured through the inference script.
  • End-to-End Results: End-to-end gains occur across all evaluated language-model families after patch insertion, reflecting dispatch overhead, shape guards, cache state, and surrounding model code.The evaluation therefore measures realized deployment effects rather than standalone kernel timing alone.
  • End-to-End Results: Profiling selects different dominant targets by family: attention regions for transformers, state-space mixers for Mamba, and the RGLRU core for RecurrentGemma.Accepted patches also depend on model structure and hardware execution context.
  • Strong Deployment Baselines: LLM4LLM reaches competitive latency in several attention regimes, while specialized attention implementations retain advantages for some workload shapes.The comparison separates model-integrated correctness from whether attention is the best search target.
  • Strong Deployment Baselines: Generated mixer replacements can absorb surrounding reshapes, projections, and elementwise updates, enabling improvements over mamba_ssm and causal-conv1d in several regimes.This evidence supports profiling-driven targeting across attention, mixers, and recurrent kernels.
  • Ablation Study: Iter-10 achieves a 100% pass rate for all three models, but its GeoMean and Fast2 metrics trail stronger sampling runs in several settings.The search can spend many turns repairing one trajectory after the first viable implementation.
  • Ablation Study: With restart, the 15-trial setting gives the best GeoMean against eager execution for all three models: 2.153, 2.546, and 2.745.Restart improves over a larger optimization budget alone, and increasing the budget from 10 to 15 trials consistently improves GeoMean and Fast2.

5 Related Work

Prior work spans compiler and scheduling systems, LLM-based code optimization, and language-model serving infrastructure. LLM4LLM differs by accepting generated kernels through the patched model under a deployment workload.

  • Compiler and Scheduling Systems: Compiler and scheduling systems generate tensor programs and optimize schedules across operator and graph levels.Examples include Halide, Tensor Comprehensions, TensorIR, TVM, AutoTVM, Ansor, Triton, and OpenTuner.
  • LLM4LLM: LLM4LLM accepts generated kernels through the patched model under a deployment workload, connecting kernel generation to model-level execution.This deployment-facing acceptance distinguishes its framing from the related systems described here.
  • LLM-Based Code Optimization: LLM-based systems support code synthesis, repair, feedback-driven improvement, and repository-level editing.The cited foundations include Codex, CodeT5, InCoder, Code Llama, and search or self-refinement systems.
  • Language-Model Serving: Language-model serving systems optimize memory, batching, scheduling, and execution phases for generation workloads.Examples include ORCA, vLLM, Sarathi, SGLang, FlexGen, DeepSpeed Inference, and DeepSpeed-FastGen.

6 Conclusion

LLM4LLM reframes LLM-based kernel optimization as a deployment-aware closed-loop problem. Its evaluations connect real-model profiling and phase-aware search with end-to-end gains across language-model families and GPU platforms.

  • Conclusion: LLM4LLM connects real-model profiling, phase-aware task construction, experience-guided episodic search, and deployment-time acceptance.The target workload participates in candidate generation and acceptance.
  • Conclusion: Across diverse language-model families and two GPU platforms, the loop converts generated candidates into end-to-end inference gains.It also clarifies the relationship between isolated benchmark performance and deployment behavior.

Limitations

LLM4LLM currently targets single-GPU inference with a representative script and specializes patches to observed shape and phase regimes. Broader serving settings require additional acceptance criteria and renewed profiling.

  • Scope: The method targets single-GPU inference and assumes access to a representative inference script.Tensor or pipeline parallelism, continuous batching, and multi-tenant serving introduce communication, scheduling, and batch-interference criteria.
  • Workload Dependence: Substantially different prompts, decode lengths, batching behavior, or model configurations require re-profiling and renewed acceptance checks.The method specializes patches to observed shape and phase regimes.
  • Practical Constraints: Optimization cost is most attractive when a resulting patch is reused across many inference calls or related model instances.Candidate quality also depends on the underlying model's coding ability, search budget, and relevant implementation patterns.

A Appendix

The appendix expands the main paper with analyses of restart, search diversity, API compatibility, and model-backbone behavior.

  • The appendix analyzes mechanisms summarized in the main paper using saved run artifacts and the same correctness-before-timing acceptance principle.It covers restart as search-state control, search diversity versus linear repair, API compatibility, and model-backbone behavior.

A.1 Search-State Control by Restart

Restart prevents prolonged local repair from constraining subsequent search, enabling the agent to reconsider optimization scope and discover broader fusion opportunities.

  • 1.42 ms was the best restart-enabled latency, versus 8.01 ms without restart for the Matmul_BatchNorm_BiasAdd_Divide_Swish case.Both variants used GPT-5.4, Triton fp32, an A100, and a 15-candidate budget.
  • Restart preserves compact correctness and implementation constraints while discarding a long repair history tied to one partially fused layout.This allows the next episode to reselect the optimization scope without losing durable feedback.
  • Without restart, repeated output-mismatch feedback led the search back to correct epilogue-only implementations around 8 ms.The no-restart trace repeated the same mismatch signature for iterations 2–4 before returning to the earlier implementation family.
  • With restart, full-fusion candidates appeared at iterations 2 and 3, reducing the operation scope from epilogue fusion to GEMM-level fusion.The full-fusion candidates measured 1.45 ms and 1.42 ms.

A.2 Sampling and Iteration Are Complementary

Sampling and iterative repair provide complementary search behaviors: sampling explores alternative optimization scopes, while iteration improves nearby candidates and repairs failures.

  • For Gemm_Add_ReLU, sampling found full GEMM-plus-epilogue fusion, whereas iteration retained PyTorch GEMM and fused only the bias-ReLU epilogue.The contrast demonstrates that search procedures can select different optimization scopes for the same graph.
  • For Conv2D_ReLU_BiasAdd, iterative repair followed an output mismatch and produced a cleaner NCHW-specialized epilogue while preserving the sampled module boundary.The iterative run required one repair turn before retaining that structure.
  • LLM4LLM combines sampling, feedback, and restart because sampling exposes alternative decompositions while linear repair improves nearby candidates.Restart combines fresh scope selection with compact constraints learned from earlier episodes.

A.3 API Compatibility and User-Defined Fixes

The appendix separates version-dependent kernel-DSL compatibility from performance design and shows that deployment kernels must respect model- and phase-specific contracts.

  • An unavailable Triton math entry point caused the iterative Matmul_Divide_GELU candidate to fail before repair.The failing candidate called tl.math.tanh in its GELU approximation.
  • After API repair, iteration produced a correct epilogue-only kernel, while an independent sample reached fused Linear + divide + GELU.The two candidates differed at the kernel boundary and optimization scope.
  • The API-correction layer supports backend-specific fixes for Triton, TileLang, and CUDA extension patterns, which users can extend for local versions.These fixes are kept separate from fusion scope, tiling, masking, accumulator precision, cache updates, and launch structure.
  • The three model backbones differ in search behavior, so identical correctness, timing, and deployment-acceptance rules accommodate conservative, guarded, and aggressive candidates.GPT-5.4 can become conservative after partial fusion, Claude emphasizes guarded fallbacks, and GLM-5 produces more aggressive candidates.
  • A.5 Deployment Kernel Families: Accepted deployment kernels are phase specialized: attention uses distinct prefill and decode contracts, while Mamba and recurrent blocks target different state and memory behaviors.The extracted-task speedups in Table A.4 are measured before full-model aggregation.
Loading 2608.21836v1…