PaperlayerView exact revision on arXiv

Source-linked AI summary

AKTS: Sub-Microsecond Kernel Policy Switching for Language-Model Agents

Mohammadali Khodabandehlou, Mahdi Alizadeh

arXiv:2609.12276v2cs.OS

TL;DR

GPU-backed LLM servers need to balance burst-time serving with background batch work, but scheduler policies must switch safely and quickly. AKTS verifies a policy library at load time and lets an agent select a policy with an integer index. In a vLLM workload, it retains 97% of a throughput policy’s batch work while matching a latency policy’s burst response.

  • Problem

    GPU-backed LLM serving shares CPUs between interactive serving and background work, while a fixed scheduler policy loses one objective as workloads shift.

    Source: §1 · p. 1

  • Method

    AKTS pre-verifies a library of kernel policies and reduces the agent’s runtime action to selecting one with an integer index.

    Source: §1 · p. 2

  • Results

    AKTS matches the latency policy’s burst response within one standard deviation while retaining 97% of the throughput policy’s batch work.

    Source: §4.4 · p. 4

  • Takeaways & Limitations

    The closed action space means agent output cannot cause unverified code to execute in the kernel.

    Source: §2 · p. 2

    The design rests on three undocumented assumptions, and all measurements use one host.

    Source: §3 · p. 3

Abstract

from arXiv · show

GPU-backed LLM servers multiplex interactive requests with background batch work on the same CPUs; a fixed kernel policy serves one objective and loses the other, so agentic OS control needs to switch scheduler behavior as the workload changes. The hard part is applying the switch safely and fast enough for the kernel: scheduler events occur every 1-10 $\mu$s, and any code running there must satisfy the eBPF verifier. Scalar knobs are fast but limited, while generating eBPF policy code puts compilation, verification and possible rejection on the runtime path. We present AKTS, which verifies a policy library once, at load time, and reduces the agent's runtime action to writing an integer index into an in-kernel array of preverified policies, resolved by a tail call. Because the agent emits an index rather than code, verifier failure is not a runtime outcome. On Linux 6.14, AKTS applies a policy switch in 920 ns (p50); makes an invalid index inert across 60,217 invocations on an attached scheduler; and switches policies in a vLLM workload to capture 97% of a throughput policy's batch work while matching a latency policy's burst response.

1 Introduction

AKTS addresses the difficulty of switching scheduler behavior safely and quickly as GPU-backed LLM workloads shift between serving bursts and background work. It replaces limited scalar tuning and slow code synthesis with a preverified policy library selected by an integer, and reports workload results combining burst response with batch work.

1 Introduction

  • A scheduler for GPU-backed LLM serving must balance burst-time low time-to-first-token against background progress in quiet periods.The same CPUs handle request intake, tokenization, prefill orchestration, and background jobs.Source: §1 · p. 1
  • Scalar tuning makes decisions with a single write to an exposed kernel knob, but its limitation is expressiveness rather than timing.LumOS tunes Completely Fair Scheduler hyperparameters with an LLM agent.Source: §1 · p. 1
  • Code synthesis recovers full policy expressiveness, but compilation and verification take seconds to a minute per change and generated code can be rejected [15].SchedCP [15] generates sched_ext policy code; learned schedulers adapt policies without synthesizing mechanisms [10] [11].Source: §1 · pp. 1–2
  • AKTS compiles and verifies eBPF policies once at load time, then has the agent write an integer that selects a policy through a kernel tail call.Figure 1 depicts the preverified policies, integer selector, and inert empty slot for an invalid index.Source: §1 · p. 2, Figure 1 · p. 2
  • Because the agent selects only among already-verified programs, an unsafe hallucinated decision cannot introduce unverified kernel code; program arrays and tail calls are applied as a constrained sched_ext action space.The paper contrasts this with approaches that validate or roll back proposed actions.Source: §1 · p. 2
  • AKTS switching in a vLLM workload captures 97% of a throughput policy’s batch work while matching a latency policy’s burst response.The paper also reports actuation at parity with scalar tuning and an invalid decision proven inert on an attached scheduler.Source: §1 · pp. 1–2

2 Design

AKTS separates slow agent decisions from kernel-time execution: the agent periodically writes a policy index, and a dispatcher tail-calls into the selected preverified program. The closed action space prevents unverified code from executing, though a valid wrong index can still select the wrong policy.

2 Design

  • A dispatcher reads the selected index and tail-calls into a preverified policy from a program array; an out-of-range or empty slot runs the default action.Tail calls transfer control without growing the stack and use one of 33 permitted calls per event.Source: §2 · p. 2
  • The reasoning plane reads telemetry about every 100 ms and writes the index, while kernel dispatch continues between writes without a model call or userspace round trip.The agent’s cadence is separated from execution on each scheduling event.Source: §2 · p. 2
  • The closed action space prevents agent output from causing unverified code to execute, but an incorrect valid index can choose the wrong policy.An in-kernel guard that could reset the slot without waiting for the agent is designed but unimplemented.Source: §2 · pp. 2–3

3 Feasibility on stock Linux

Feasibility tests on stock Linux 6.14 show that tail-call dispatch can verify, populate a program array, and execute through an attached sched_ext scheduler. The tests rely on undocumented assumptions and measurements from one host.

3 Feasibility on stock Linux

  • The design depends on three undocumented sched_ext and tail-call assumptions, and all measurements use one host running Linux 6.14.0-27.The host had an NVIDIA A100-SXM4-40GB, a 30-vCPU AMD EPYC 7J13, and 216 GB RAM; Gates 1 and 2 were load-time checks without an attached scheduler.Source: §3 · p. 3
  • All three Gate 1 programs loaded: the sched_ext program with a tail call, an XDP control, and a struct_ops control without the tail call.The JITed size grew from 76 B to 176 B with the tail call.Source: §3 · p. 3
  • Gate 2 insertion into a program array succeeded; a second, never-registered map avoided libbpf’s refusal to load an unreferenced struct_ops program.Entries must agree on prog_type and attach_btf_id.Source: §3 · p. 3
  • 64,160 of 64,160 invocations transferred control to the target policy on an attached sched_ext scheduler, with zero fall-throughs.The test instrumented both the target policy and the path after the tail call because a failed call raises no error.Source: §3 · p. 3

4 Evaluation

The evaluation measures policy-actuation cost, decision validity and safety, then tests policy switching against fixed policies under CPU contention. AKTS matches latency-policy burst response while retaining most throughput-policy batch work.

4 Evaluation

  • Attached schedulers managed only processes explicitly placed in SCHED_EXT, leaving unrelated system activity outside the scheduling experiment.Source: §4 · p. 3

4.1 Actuation latency

  • AKTS matches scalar tuning on actuation while switching whole policies, whereas code synthesis puts compilation and verification on the runtime path.The evaluation isolates the cost of applying a change after a decision has been made.Source: §4.1 · p. 3

4.2 Agent decision cost and decision validity

  • Median decision latency is 13.5 ms, but 16 of 48 decisions (33%) were invalid and stronger models gave constant, chance-level answers across telemetry regimes.The constant-answer result held for Qwen2.5 at 0.5B, 1.5B and 3B; constraining decoding to {0, 1} changed nothing.Source: §4.2 · p. 4

4.3 Safety under faulty decisions

  • Across 60,217 invocations of a never-populated slot, the dispatcher fell through to default CPU selection; workers completed and the scheduler detached cleanly.An arbitrary integer either names a policy verified at load time or names nothing, which reverts to default scheduling rather than undefined behavior.Source: §4.3 · p. 4

4.4 Does switching beat any fixed policy?

  • The serving test used Qwen2.5-0.5B under vLLM on four cores with eight CPU-bound antagonists and steady–burst–steady request phases.Contention raised steady-state TTFT p99 from 27.0 ms to 308.0 ms; TTFT was used because GPU decode would dominate end-to-end latency.Source: §4.4 · p. 4
  • AKTS matches the latency policy’s burst response within one standard deviation while retaining 97% of the throughput policy’s batch work.The latency policy retains 91% of batch throughput; the throughput policy has 59% worse burst TTFT p50 and serves 35% fewer requests.Source: §4.4 · p. 4

5 Conclusion and future work

AKTS verifies a policy library once and exposes an integer selector for sub-microsecond policy switching. The evaluation uses oracle-driven switching, an upper bound on an online detector; an agent-driven arm must show a language model beats a bandit baseline.

5 Conclusion and future work

  • AKTS verifies a policy library once and exposes an integer selector, making policy switching a sub-microsecond kernel operation.Source: §5 · p. 4
  • Oracle-driven switching is an upper bound on any online detector.Source: §5 · p. 4
  • An agent-driven arm must show that a language model beats a bandit baseline.Source: §5 · p. 4
Loading 2609.12276v2…