PaperlayerView exact revision on arXiv

Source-linked AI summary

Adaptive KV Retention for LLM Agents at Human-Approval Timescales

Minseo Choi, Ananya Joshi

arXiv:2608.30830v1cs.OS

TL;DR

Human-approval pauses can last minutes to hours, creating a retention problem in which preserving suspended KV consumes active-serving capacity while eviction increases resume cost. The paper develops a tiered controller using GPU opportunity cost and calibration waits to choose among retention and recomputation options without per-request wait prediction. On human-scale workloads, it improves active-request goodput over vLLM, MORI, and Continuum.

  • Problem

    Human approvals can suspend agentic requests for minutes to hours, beyond the seconds-scale pauses primarily studied by prior agent-serving systems, making KV retention a capacity–resume tradeoff.

  • Method

    The controller uses GPU opportunity cost to select HBM retention, host retention, or recomputation, with host behavior chosen between indefinite retention and load-indexed expiration from calibration waits.

  • Results

    23–51% higher active-request goodput than vLLM baselines, 22–29% than MORI, and 41–52% than Continuum is reported under host-memory pressure.

  • Takeaways & Limitations

    Platform-calibrated, load-aware retention navigates the residency–resume tradeoff and outperforms the evaluated agent-serving baselines in active-request goodput.

  • Takeaways & Limitations

    The evaluation uses synthetic wait durations and LLM-simulated approval outcomes, requires online load estimation and periodic recalibration, and lacks production evidence for crossing λcrit.

Abstract

from arXiv · show

Unlike the seconds-scale tool-call pauses targeted by prior agent-serving systems, agentic LLM requests can be suspended for minutes or hours while waiting for human approval. We study how suspension and resumption affect GPU serving performance and develop a retention policy that balances active-serving capacity against future recomputation under uncertain approval waits. The central tension is severe because retaining suspended KV preserves fast resume but can consume enough GPU capacity to reduce active-serving goodput by 41%, while evicting it avoids that residency cost at the expense of nearly $10\times$ higher resume latency when the request returns. We develop a tiered retention controller around GPU opportunity cost, which expresses the serving capacity consumed by preserving or reconstructing a suspended request's KV state in a common GPU-time cost. Within host memory, the controller selects between indefinite retention and load-indexed expiration using calibration wait samples, without requiring per-request wait prediction. On human-scale approval workloads, our controller improves active-request goodput by 23-51% over the vLLM baselines, 22-29% over MORI, and 41-52% over Continuum.

1 Introduction

Human approvals suspend agentic requests for minutes to hours, creating a retention tradeoff unlike seconds-scale tool-call pauses. The paper uses GPU opportunity cost to select retention behavior under heterogeneous waits and reports substantial goodput gains.

  • Motivation: Minutes-to-hours human approvals create a longer suspension regime than the seconds-scale pauses studied by MORI and Continuum.Existing systems mainly target tool-call pauses, while human approval can last minutes to hours.
  • Motivation: Retaining suspended KV preserves attention state but consumes serving capacity, while host offload or discard shifts cost to transfer or recomputation.GPU opportunity cost expresses the capacity forgone to preserve or restore suspended state.
  • Motivation: At a 30-minute reference wait, resume latency becomes secondary while GPU opportunity cost continues accumulating during suspension.The 1-second resume stall is 0.06% of the reference wait.
  • Motivation: Heterogeneous approval waits make a single host-retention timer unsuitable, whereas age can separate contexts likely to resume soon from those likely to remain suspended.Under memoryless waits, elapsed time is uninformative and expiration adds churn.
  • Approach: The controller selects indefinite host retention or load-indexed expiration from calibration waits before the held-out serving run, without per-request wait prediction.Its contribution includes choosing among HBM, host DRAM, and recomputation using GPU opportunity cost.

2 System-level characterization of human approval suspension

The study isolates how suspended KV residency affects active serving and resume latency under human-approval suspension. It finds a sharp capacity–resume tradeoff, motivating controlled residency measurements under saturation.

  • Measurement setup: The retention decision depends on reusable KV state left at the approval gate and active-serving capacity lost by keeping it resident.These two quantities determine the system-level retention tradeoff.
  • Workload: 93–96% of approval gates are approved, making the private suspended suffix potentially valuable to retain.The median suspended context contains 8.4K tokens, including approximately 3.0K private suffix tokens after prefix reuse.
  • Measurement setup: Under memory pressure, vLLM residency falls from 100% at N = 25 to 10–14% for N ≥50, so stock behavior cannot isolate residency cost.The sweep occupies 31–89% of the KV pool and uses background load that engages eviction.
  • Measurement setup: A victim-exclusion experiment with 48 concurrent active clients makes suspended-KV capacity appear directly as lost goodput.The operating point is chosen near saturation, where withheld KV competes with active requests.
  • Results: 41% lower goodput at N = 72 under enforced residency quantifies the active-serving cost of retaining suspended KV.The experiment compares enforced residency against unmodified vLLM behavior.
  • Results: Eviction raises resume TTFT from 66–75 ms to 687–705 ms, approximately 10×, shifting the cost from serving capacity to resumption.Resident contexts resume faster, while evicted contexts require substantially more time to first token.

3 Method

The method prices HBM retention, host-DRAM retention, and recomputation in GPU-seconds, then uses break-even rules and load-dependent congestion to select retention. It offloads suspended KV at suspension and chooses host retention or expiration through calibration.

  • Retention cost model: GPU opportunity cost puts HBM holding, host restoration, and recomputation on one serving-capacity scale.HBM, host DRAM, and discard represent the three retention states.
  • Host-memory congestion: Host DRAM receives a load-dependent holding price α2(λ) after capacity becomes scarce, linking congestion to the expiration threshold t2(λ).The threshold is derived from host capacity and recomputation cost.
  • Retention cost model: The cost model calibrates three platform-specific coefficients: α1, β2, and β3.Other holding costs are either zero in the uncongested regime or derived from load.
  • Host-memory congestion: Little’s law maps host capacity and reference residence time to the saturation threshold λcrit = C2/Wref.Above λcrit, excess arrivals motivate a mean-field congestion price.
  • Retention policy: The controller moves KV to host DRAM at suspension because the HBM–DRAM break-even t1 is approximately 1 second.It then decides how long to retain the host copy.
  • Retention policy: Calibration selects between indefinite host retention and load-indexed expiration, freezing the lower-cost branch before held-out serving.Runtime evaluation of t2(λ) requires only the offered load λ.

4 Evaluation

The evaluation measures live serving at twice the host-capacity threshold and compares the controller with vLLM, MORI, and Continuum. The controller selectively retains suspended KV and improves goodput, while the preferred host policy depends on the wait distribution.

  • Evaluation setup: At 2λcrit, the evaluation compares the controller with unmodified vLLM, host-offloaded vLLM, MORI, and Continuum.Live measurements use Llama-3.1-70B on 4×H100 NVL GPUs with a 375–380 GB host-memory budget.
  • End-to-end serving performance: 23–51% higher active-request goodput than vLLM and 41–52% higher goodput than Continuum result from selectively retaining suspended KV.The controller retains 38.1% and 17.6% of suspended KV on the lognormal and mixture workloads, respectively.
  • End-to-end serving performance: 29% higher goodput than MORI occurs on the mixture workload despite MORI retaining more suspended KV, 21.9% versus 17.6%.The controller achieves 1.07 req/s versus MORI’s 0.83 req/s.
  • Evaluation caveats: MORI is evaluated through a policy port on vLLM, while the Continuum comparison uses a public fork that omits its pause-duration estimator.These implementation differences qualify the baseline comparisons.
  • Controller selection: The controller’s host-policy choice depends on short-wait prevalence: expiration crosses over host retention at P(W ≤t∗) = 0.39.Expiration hurts when few requests resume before t∗ and helps as short waits become more common.

5 Discussion

Prior agent-serving systems mainly target seconds-scale tool-call pauses, whereas this work addresses minute-to-hour human-approval waits. It frames KV retention as a capacity-allocation problem and reports gains over existing baselines, while identifying important deployment limitations.

  • Prior systems such as MORI and Continuum mainly study seconds-scale tool-call pauses, while this work targets minute-to-hour approval waits.
  • Production deployment would require online load estimation and periodic recalibration as approval-wait distributions change.
  • The experiments use LLM-simulated approval outcomes, synthetic waits, a single model family, and non-native comparisons for MORI and Continuum.
  • The results establish a measurement-backed retention policy rather than a complete production controller.

A Residency-enforcement instrument

The residency-enforcement instrument varies suspended-context occupancy and measures its effect near serving saturation. Victim exclusion isolates the cost of retaining KV, while unmodified vLLM exposes pressure-induced eviction behavior.

  • Table 1 measures realized HBM residency for suspended Llama-3.1-70B contexts under 3 req/s background load.
  • 31–89% of the KV pool is occupied across N = 25 to 72 suspended contexts, spanning the usable capacity range.
  • 10–14% residency under pressure for unmodified vLLM contrasts with 93–100% under victim exclusion.
  • Victim exclusion removes selected suspended blocks from the eviction queue, keeping them unavailable to active requests so their capacity cost appears in scheduling.
  • 48 concurrent active clients is fixed because goodput peaks at C = 48, while tail latency continues increasing at C = 64.

B Price-vector calibration

The controller calibrates GPU opportunity-cost prices for HBM retention, host resumption, and recomputation using platform measurements. These measurements account for memory occupancy, transfer overhead, and suffix re-prefill cost.

  • α1 charges HBM holding cost from one suspended suffix’s fraction of the measured TP4 KV pool across four GPU ranks.The calibrated expression is α1 = 4 × 3,000 / 680,768 = 0.0176 GPU-s/s.
  • 0.023–0.030 GPU-s/s is the empirical closed-loop slope, bracketing the accounting price from above.
  • 0.0480 s device-to-host and 0.0478 s host-to-device median transfer times are measured for the 2.75 GB full-context payload.
  • β2 = 0.02 GPU-s conservatively represents GPU-side scheduling and synchronization overhead rather than charging DMA transfer wall time one-for-one.
  • Suffix re-prefill has 0.479 s median resume TTFT, while a cold full-context run takes 1.366 s.

C Cross-platform ablation

Cross-platform replay recalibrates the controller separately on H100 NVL, A100 SXM, and L40S hardware. Although calibrated constants and thresholds shift substantially, the preferred policy ordering remains stable across wait families.

  • The cross-platform study measures host bandwidth, suffix recomputation cost, and KV-pool capacity before replaying each platform’s calibrated controller.
  • Table 3 reports calibration and replay at 2λcrit using platform-specific price vectors and break-even times.
  • The controller holds C2 = 425 fixed across platforms to isolate GPU platform effects.
  • α1 varies by nearly 7× across platforms, while t∗ ranges from 43.5 s to 112 s.
  • Host-retain remains preferred for lognormal and exponential waits, whereas cpu_ttl(λ) remains preferred for the mixture.
  • The main text defines λcrit = C2/Wref with Wref = 1,800 s, scaling all wait families to the same mean.

E Extended related work

Prior retention policies primarily target seconds-scale tool-call pauses or rely on request-level signals, while this work addresses human-scale waits with system-load-based retention.

  • Prior systems: MORI’s relative-idleness signal becomes less discriminative as waits grow from tool-call scale to half-hour scale.Across 2,161 approval-gated trajectories, the p5–p95 spread contracts from 0.359 to 0.018.
  • Prior systems: Continuum indexes retention using a predicted pause-duration bound, but its published system primarily targets seconds-scale tool calls.The evaluation uses the released implementation with its shipped 2 s pin TTL.
  • Retention signals: Prior agent-serving policies primarily use request-level signals, whereas this work indexes retention on system load.The comparison covers MORI, Continuum, and other systems summarized in Table 4.
  • Motivation: At longer waits, continuous GPU retention becomes increasingly expensive, motivating host-tier release decisions under load.The paper distinguishes preserving, swapping, and recomputing intercepted KV while pricing retention against shared accelerator and host-memory capacity.

F.3 Load-sweep replay

The controller is calibrated once per wait family and evaluated across a broad load range rather than tuned separately for each load.

  • Load-sweep replay: The controller matches the best fixed policy throughout the lognormal and exponential workloads.The comparison spans 12 conditions from 0.5–3×λcrit at C2 = 425.
  • Load-sweep replay: Within 0.2% of the best fixed policy at 2λcrit, the controller remains competitive on the mixture workload.The reported gain is relative to the best fixed policy for each condition.
  • Load-sweep replay: 3.6% better than the best fixed policy at 3λcrit, the controller benefits from its calibration-selected branch without per-load TTL tuning.The branch is selected once per wait family and then fixed across all loads.

G Compute resources

The experiments use a single four-GPU H100 NVL node with substantial host-memory capacity, while replay and plotting are CPU-only.

  • Hardware: 4× NVIDIA H100 NVL GPUs with 94 GB HBM each, 128 vCPUs, and 1.5 TiB host DRAM support the live experiments.The available shared-memory budget limits the host tier to 375–380 GB.
  • Hardware: 375–380 GB is the available shared-memory budget for the host tier.The host-capacity limit is stated for the live experimental setup.
  • Compute resources: Replay, trace generation, and plotting are CPU-only and complete within minutes.This separates the broader replay workload from the live GPU-serving measurements.
  • Compute resources: 250–300 GPU-hours covered the full project, including exploratory and invalidated runs not reported in the paper.The figure includes compute beyond the experiments presented in the paper.
Loading 2608.30830v1…