PaperlayerView exact revision on arXiv

Source-linked AI summary

KREX: Concurrent Kernel Benchmarking on Shared GPUs via Region-Granular Exclusivity

Tianyu Feng, Haoxuan Yu, Tianyuan Wu, Lingyun Yang, Daocheng Ying, Yuxiao Wang, Ruibo Fan, Yinghao Yu, Guodong Yang, Liping Zhang, Wei Wang

arXiv:2609.30057v1cs.DCcs.OScs.PF

TL;DR

Coarse GPU reservations leave capacity idle, while unprotected sharing can distort kernel timings and candidate rankings. KREX applies exclusivity to marked timing regions and allows concurrency elsewhere, delivering higher benchmarking throughput with low p95 timing inflation.

  • Problem

    Session- and command-level GPU reservations leave capacity idle, while sharing can compromise measurement fidelity and misdirect kernel-agent searches.

    Source: §1 · p. 2

  • Method

    KREX enforces GPU and CPU exclusivity within marked timing regions and uses persistent GPU contexts to support concurrent work outside them.

    Source: §1 · p. 2

  • Results

    KREX delivers up to 3.4× the benchmarking throughput of command-granular exclusivity, with p95 timing inflation of 0.30%, 1.58%, and 3.90% for kernels longer than 10, 1, and 0.1 ms, respectively.

    Source: §1 · p. 2

  • Takeaways & Limitations

    KREX avoids trading GPU utilization for measurement fidelity by running only critical regions alone and preserving relative performance signals that guide agent search.

    Source: §9 · p. 12

    KREX relies on cooperative candidates and author-marked boundaries, does not validate region correctness, and supports timing fidelity empirically without a theoretical worst-case error bound.

    Source: §7 · p. 11

Abstract

from arXiv · show

LLM agents automate GPU kernel optimization by repeatedly composing candidates and measuring their duration on real GPUs. Existing systems preserve measurement fidelity by reserving a GPU for an entire agent session or benchmarking command. However, this results in poor utilization because only a small fraction of command execution requires exclusive GPU access. Sharing GPUs could recover this idle capacity, but introduces contention that compromises measurement fidelity and misdirects the agent's search. We present KREX, a runtime for concurrent kernel agent benchmarking with region-granular exclusivity. KREX lets agents mark critical regions involving timing-sensitive operations within a benchmarking command. The runtime then enforces exclusivity within marked regions and allows concurrent execution outside them, achieving high throughput while preserving measurement fidelity. To enforce in-region exclusivity, KREX blocks new competing GPU submissions and drains outstanding work before freezing sibling processes and isolating CPU cores, protecting both GPU execution and the host threads that drive measurements. To maximize off-region concurrency, KREX reuses GPU contexts in persistent context processes to avoid repeated, node-wide serialized context creation. We evaluate KREX on NVIDIA and AMD GPUs. Compared with command-granular exclusivity baselines, KREX delivers up to $3.4\times$ the benchmarking throughput with a negligible p95 timing inflation of $0.30\%$, $1.58\%$, and $3.90\%$ for kernels longer than 10 ms, 1 ms, and 0.1 ms, respectively.

1 Introduction

Kernel-agent benchmarking needs faithful timing, but reserving GPUs for whole sessions or commands leaves much capacity idle. KREX confines exclusivity to marked timing regions and shares the GPU outside them.

1 Introduction

  • Kernel agents repeatedly measure candidates to guide optimization, making benchmarking a large and growing GPU workload.The production environment receives over 300,000 kernel benchmarking jobs daily across six GPU models from three vendors.Source: §1 · p. 1
  • Queued benchmarking requests demand 4.8× peak online GPU capacity, yet allocated GPUs remain idle much of the time under coarse reservations.One profiled agent session used the GPU for only 3.4% of its wall-clock time.Source: §1 · p. 1
  • Only timing loops require exclusive GPU access; E3 (KREX) reserves the device for those regions while allowing sharing elsewhere.In one eight-agent run, the device was held for 82% of a nine-second window but executed timed regions for only 7%.Source: §1 · p. 2, Figure 1 · p. 2
  • KREX drains in-flight GPU work, freezes competing host processes, and runs the measuring command on reserved CPU cores inside critical regions.Outside regions, persistent context processes avoid repeated node-wide serialized context creation.Source: §1 · p. 2
  • KREX achieves up to 3.4× the benchmarking throughput of command-granular exclusivity, with p95 timing inflation of 0.30%, 1.58%, and 3.90% for kernels longer than 10, 1, and 0.1 ms, respectively.Its candidate-ranking experiment kept ranking-flip rates within three percentage points of an uncontended repeat.Source: §1 · p. 2

2 Background and Motivation

Benchmarking commands spend only a small fraction of their time in the timing phase, while coarse reservations waste capacity and unprotected sharing distorts measurements. Region-granular exclusivity addresses this mismatch, but requires explicit boundaries, fidelity protections, off-region concurrency, and support for heterogeneous GPUs.

2.1 Kernel Agent and Benchmarking

  • The timed loop takes 8.5% of a representative command—0.22 s of 2.6 s—while startup and compilation take 81% and correctness checks take 10%.Only the timing phase directly determines the kernel-duration metric being optimized.Source: §2.1 · p. 3

2.2 Why Current Approaches Fail

  • Session- and command-granular reservations hold GPUs beyond the timing phase, while per-command context creation serializes admission across GPUs.Context creation acquires a node-wide driver lock and costs hundreds of milliseconds; creating 64 CUDA contexts took more than 30 seconds.Source: §2.2 · p. 3
  • Unprotected sharing inflates reported kernel durations to 1.5×–3.2× those measured alone, and contention can reverse candidate rankings.In one trajectory, sharing reversed six of ten pairwise rankings, whereas a second uncontended run reproduced the original ordering.Source: §2.2 · p. 3, Figure 2 · p. 3

2.3 Region-Granular Exclusivity

  • Region-granular exclusivity protects the marked timing span while leaving the rest of each command available for concurrent execution.This targets the phase sensitive to contention without reserving the whole command.Source: §2.3 · pp. 3–4
  • KREX’s requirements are complete boundaries without kernel changes, faithful timings and rankings, concurrent off-region work without serialized admission, and NVIDIA/AMD support.Enforcement mechanisms may differ across the two hardware vendors.Source: §2.3 · p. 4

3 System Overview

KREX accepts commands with explicitly marked critical regions and lets commands share a GPU whenever none is executing such a region. A dispatcher assigns persistent GPU contexts, while the runtime coordinates GPU and CPU protection at region entry and exit.

3 System Overview

  • Users mark only the code requiring exclusivity; KREX protects GPU execution and CPU cores in regions and permits concurrent execution otherwise.Regions can be marked by the agent or by the author of a fixed benchmarking scaffold.Source: §3 · p. 4
  • A dispatcher pairs each client command with a pooled context process, which persists across commands so its GPU context can be reused.The client forwards its GPU calls to the assigned context process.Source: §3 · p. 4
  • At region entry, KREX blocks new submissions, drains outstanding GPU work, then freezes competing process trees; on exit, it thaws tenants and reopens submissions.Measuring threads are pinned to reserved cores within per-GPU CPU slices.Source: §3 · p. 4
  • The design combines a one-line marking API, GPU and CPU protection, off-region concurrency with persistent contexts, and vendor-specific implementations for NVIDIA and AMD.Persistent contexts pay creation costs at startup or recovery rather than on every command.Source: §3 · p. 5

4 Design Mechanisms

KREX uses explicitly marked critical regions to isolate GPU and CPU activity during timing, while persistent contexts enable sharing outside those regions.

4.1 Region Marking

  • Region boundaries must reflect the intended measurement, so KREX requires authors to mark them rather than infer them automatically.Agents mark regions when they write the benchmark; scaffold authors mark them when evaluation scripts are fixed, and agents can revise incorrect boundaries.Source: §4.1 · p. 5

4.2 In-Region Exclusivity

  • KREX blocks new GPU submissions, drains outstanding work, and freezes sibling processes before measurement, while reserving and pinning CPU cores for submitting threads.Siblings are thawed before submissions reopen at region exit.Source: §4.2 · pp. 5–6

4.3 Off-Region Concurrency

  • Persistent context processes reuse GPU contexts across candidate commands, avoiding repeated context creation that is serialized by a node-wide driver lock.Each candidate runs in its own client process, paired with a context process that serves candidates sequentially.Source: §4.3 · p. 7
  • KREX forwards GPU calls through shared-memory RPCs, using asynchronous forwarding for launches and request–reply communication for timing events and synchronization.Launch forwarding can overlap GPU execution; spin–park waiting avoids wake-up costs for entries that arrive during the spin interval.Source: §4.3 · p. 7
  • After each candidate, KREX releases its allocated resources, checks context health, and replaces contexts that fail the check.A background check also applies recovery when clients terminate unexpectedly or exceed their execution limit.Source: §4.3 · pp. 7–8

5 Implementation

KREX implements GPU-call interception and dispatch for NVIDIA and AMD, with platform-specific options for enforcing the region protocol.

5 Implementation

  • KREX supports NVIDIA and AMD with about 20,000 lines of C++ for GPU-call interception and dispatch, plus 4,800 lines of Python for services outside the GPU-call path.Another 41,000 lines of C++ CUDA and NVML forwarding stubs are machine-generated from annotated headers.Source: §5 · p. 8
  • Both GPU implementations follow the same region protocol, but NVIDIA gates calls in the context process while AMD can use a pooled path or a ROCm-loaded tools library.The AMD library patches the runtime dispatch table to intercept queue kernel launches.Source: §5 · p. 8
  • On AMD, the in-flight counter tracks device completion, and tenants enter freezer cgroups only after initialization because earlier freezing caused failures on the evaluated system.Source: §5 · p. 8

6 Evaluation

The evaluation replays real kernel-agent workloads across NVIDIA H20 and AMD MI308X GPUs, comparing KREX with command-granular, serial, and unprotected configurations. It measures throughput, timing and ranking fidelity, configuration trade-offs, and GPU-call-forwarding effects.

6.1 Experimental Setup

  • The testbed comprises two 8-GPU nodes on each of two platforms, and the replay covers more than 20,000 commands from three kernel benchmarks and real agent runs.The configurations include native, KREX-serial, full KREX, and unprotected sharing; default command concurrency is 16 on H20 and 4 on MI308X.Source: §6.1 · p. 8

6.2 Workload Characterization

  • Command durations have medians of 10.2 s on NVIDIA and 17.7 s on AMD; on H20, the median critical-region fraction is 12%, and 73% of commands spend under 25% of execution in regions.Median region fractions are 4.4% for KernelBench, 18.2% for FlashInfer-Trace, and 22.7% for Atrex-Bench.Source: Figure 6 · p. 8, §6.2 · p. 8

6.3 Throughput

  • KREX reaches 54.8 cmds/min on H20, 3.4× native’s 16.0 cmds/min, and 50.4 cmds/min on MI308X, 2.6× native’s 19.4 cmds/min.Ungated reaches 74.4 cmds/min on MI308X, but its measurements are substantially less faithful.Source: §6.3 · pp. 8–9, Figure 7 · p. 9

6.4 Measurement Fidelity

  • KREX’s p95 timing inflation is 2.5% on H20 and 2.4% on MI308X, versus 154% and 150% for ungated sharing.For kernels below 0.1 ms, native repeats already show 12.1% inflation on H20 and 7.6% on MI308X, so distortion at that scale is not attributable solely to sharing.Source: Figure 8 · p. 9, §6.4 · pp. 9–10
  • KREX’s candidate-pair ordering-flip rate stays within three percentage points of a native repeat in every duration-gap group, and median trajectory Kendall’s τb is 0.857 versus 0.820 for the repeat.Ungated’s median Kendall’s τb is 0.432.Source: Figure 10 · p. 10, §6.4 · p. 10

6.5 Ablation and Sensitivity Study

  • At concurrency 16 with MPS on, H20 timing inflation is 2.45% for KREX, 4.63% for GPU-only protection, and 23.87% for ungated; on MI308X at concurrency 8, the values are 7.6%, 8.6%, and 495.3%.On H20, MPS raises KREX throughput from 44.2 to 54.8 cmds/min while p95 inflation rises from 2.23% to 2.45%; the selected concurrency is 16 on H20 and 4 on MI308X.Source: Figure 11 · p. 10, §6.5 · pp. 10–11

6.6 Impact of GPU-Call Forwarding

  • Under single-shot timing, GPU-call forwarding shifts median duration by 62 μs for GPU-only protection and 28 μs for KREX; CPU protection reduces KREX’s p95 shift from 840 to 296 μs.Under median-of-ten timing, CPU protection reduces the p95 shift from 264 to 238 μs.Source: Figure 12 · p. 11, §6.6 · p. 11

7 Discussion and Limitations

KREX’s guarantees depend on cooperative candidates and correctly marked regions, while some device-wide operations require suspending pooled work. Its scheduling and recovery policies leave fairness and dispatcher-failure handling as constraints.

7 Discussion and Limitations

  • KREX does not validate author-marked region boundaries, and timing fidelity has empirical support but no theoretical worst-case error bound.Clock settings require deployment-level management; commands needing device-wide control receive a whole-device lease that suspends pooled work.Source: §7 · p. 11
  • Slot-based admission offers no region-access fairness guarantee, so commands monopolizing critical regions can cause starvation.Individual candidate failures are recoverable, but dispatcher failure interrupts service and requires restarting the pool and resubmitting in-flight evaluations.Source: §7 · p. 11
  • Extensions beyond kernel-agent benchmarking require explicit measurement boundaries and surrounding work that tolerates sharing; multi-GPU candidates and multi-node pools remain planned.KREX could support other GPU-based self-improvement workflows that select generated implementations using performance measurements.Source: §7 · p. 11

8 Related Work

Related work covers GPU sharing and remoting, kernel-generation approaches, and benchmarking methods. KREX focuses on concurrent benchmarking that preserves measurement fidelity through exclusion in marked regions.

8 Related Work

  • GPU-sharing work spans time-sharing, spatial sharing, cluster scheduling, and API remoting, while kernel agents and autotuners choose candidates for evaluation.KREX addresses concurrent execution of their benchmarking rather than proposing another search strategy.Source: §8 · p. 11
  • KernelBench, competition platforms, agent benchmarks, and industrial systems use exclusive or isolated evaluation, while FastKernels evaluates generated kernels under production serving conditions.FastKernels focuses on serving-interface compatibility, unlike KREX’s focus on measurement fidelity under GPU sharing.Source: §8 · p. 11
  • Timing routines, environmental-bias studies, layout randomization, and statistical benchmarking address measurement practice; KREX complements them by excluding competing work in marked regions.Source: §8 · p. 11

9 Conclusion

KREX uses region-granular exclusivity to run only critical regions alone while allowing concurrency elsewhere. It reports higher throughput than command-granular exclusivity with negligible timing inflation.

9 Conclusion

  • KREX delivers up to 3.4× the throughput of command-granular exclusivity at negligible timing inflation.The system drains competing GPU work and freezes sibling host processes inside regions, while overlapping commands and reusing persistent GPU contexts outside them.Source: §9 · p. 12
  • Only critical regions need to run alone for kernel-agent benchmarking to avoid trading GPU utilization for measurement fidelity.Source: §9 · p. 12
  • KREX preserves relative performance signals that guide agents’ search for better-performing implementations.Source: §9 · p. 12
Loading 2609.30057v1…