Continuum: Efficient and Robust Multi-Turn LLM Agent Scheduling with KV Cache Time-to-Live

1 INTRODUCTION

• End-of-turn eviction discards completed requests’ KV cache when new requests are waiting, a policy that can degrade tool-interleaved agent workloads.Tool calls can take less than or equal to two seconds, and the returned tool output is appended before inference resumes.Source: §1 · pp. 1–2
• After eviction between tool calls, an agent may incur prefix recomputation or CPU reload and wait again for GPU memory, adding queue delay across turns.Tool-call durations vary, so retention must balance keeping cache available against avoiding indefinite memory occupancy.Source: §1 · p. 2
• Continnum sets a KV-cache TTL by weighing prefill/reload cost and queue-delay reduction against GPU-memory occupancy, then evicts the cache when the TTL expires.If a tool call returns within the TTL window, the next request can resume immediately; expiration limits memory pressure when tool-call predictions are inaccurate.Source: §1 · p. 2
• The vLLM implementation uses a modular tool-call handler to identify calls, predict their durations, and choose KV-cache pin time based on throughput and request ordering.The design makes minimal changes to the inference engine’s scheduling logic.Source: §1 · p. 2
• Up to 8.18x improvements in both latency and throughput over previous methods are reported for emulated and real cases; across three hardware/model setups, delay falls 1.12x–3.66x and throughput rises 1.10x–3.22x.The real SWE-agent workload shows delay reductions of up to 8.18x.Source: §1 · p. 2

2.1 ReAct Paradigm for Agents

• ReAct agents alternate LLM reasoning with external tool actions, and long-horizon workloads repeat this loop across many turns.The text points to benchmarks covering tool-agent-user interaction, multi-turn tool use, and multi-turn decision-making.Source: §2.1 · p. 3

2.2 Limitations of Existing Methods

• Static-workflow schedulers such as Teola, Alto, and Parrot assume predefined or deterministic DAGs, unlike ReAct agents whose dependencies evolve at runtime.This limits their ability to optimize for varied agents in practice.Source: §2.2 · p. 3
• Existing serving work often lacks tool-call awareness: it does not account for variable tool durations and their effects on KV-cache management in dynamic multi-turn workflows.The text distinguishes approaches targeting individual LLM requests from those addressing agent workflows.Source: §2.2 · p. 3
• Prior retention approaches either overlook accumulating per-turn queue delays or use fixed preservation that cannot adapt to actual tool-call durations.Pie offers programmable handlers but leaves developers to design scheduling and provides no method for adapting to dynamic tool-call latencies or multi-turn dependencies.Source: §2.2 · p. 3

3.1 Agentic Traces

• Later turns have shorter expected finish times, making program-level FCFS or turn progress a possible approximation to shortest-remaining-time scheduling.Maintaining that ordering is non-trivial when tool calls are involved.Source: §3.1 · p. 4
• The slowest 10% of fetch_url calls account for 52.5% of total delay, while the slowest 10% of cd calls account for 94.1%.These examples illustrate the long-tailed execution times of tool functions.Source: Figure 5 · p. 4

3.2 Challenges for Agentic Workloads

• End-of-turn eviction during a tool call forces prefill or KV reload when the next inference step begins.The added work can delay execution and reduce system throughput.Source: §3.2 · p. 4
• After eviction, returning requests wait behind ongoing GPU work for memory, adding queueing delay across turns and disrupting program continuity.The queueing gap remains even when the KV cache is stored in CPU DRAM.Source: §3.2 · p. 4
• TTL must balance memory use against prefill and queueing costs: long pin times block requests, while short ones can evict the cache before a tool call completes.Either setting can leave the system with avoidable cost.Source: Figure 6 · p. 5
• InferCept’s preserve policy considers reload cost but not re-entry queueing, so eviction bubbles persist under CPU offloading.Figure 4 profiles total waiting-queue bubble time per program and reports persistent bubbles despite InferCept’s KV retention.Source: §3.2 · pp. 4–5, Figure 4 · p. 4
• Highly variable tool durations make static pinning risky: long calls can waste GPU memory or cause potential deadlocks.A long-pinned cache can occupy GPU memory until the next request arrives.Source: §3.2 · p. 5

4 CONTINNUM SCHEDULING ALGORITHM

• The algorithm targets cache reuse after tool calls while bounding how long KV cache remains in GPU memory, so long-running or failed calls do not retain it indefinitely.The scheduling problem is to retain cache for likely near-term reuse while reducing prefill/loading overhead and preserving continuity under variable tool-call latencies.Source: §4 · p. 5

4.1 Utility Model

• The TTL must balance early eviction, which can trigger recomputation or waiting, against excessive pinning that occupies GPU memory and can reduce throughput.The model estimates pinning cost from memory usage, active-request memory footprint, and TTL; cache-miss cost uses prefill or reload time.Source: §4 · p. 5, §4.1 · p. 6
• Continnum’s utility model weighs the cost of occupying GPU memory against avoided prefill/reload and per-turn queueing costs when a cache is reused.The queueing term can justify retention even when CPU offloading makes reloading fast; its size depends on workload memoryfulness.Source: §4.1 · p. 6
• The queueing benefit is scaled by workload memoryfulness: progress can predict less remaining work, while anti-memoryful workloads may favor frequent switching.The model defines memoryfulness using the correlation between requests already served and requests remaining; fully memoryful workloads have η = 1, while memoryless workloads have η = 0.Source: §4.1 · p. 6

4.2 Setting the TTL Value

• Continnum selects the TTL that maximizes expected net benefit, estimating tool-call completion probability from the empirical CDF of historical durations.It searches recorded durations, including τ = 0, as candidate TTLs; for sparse per-tool history it falls back to global records, then to a fixed TTL derived under an exponential unit-mean duration and fully memoryful workload assumption.Source: §4.2 · p. 7

4.3 Scheduling Priority

• The TTL-aware scheduler prioritizes preempted requests, then requests still pinned within their TTL, and finally program-level arrival order within each category.This ordering elevates pinned requests while retaining program-level FCFS ordering.Source: §4.3 · p. 7

5 CONTINNUM SYSTEM DESIGN

• A tool-call handler records tool-call timing, and the scheduler uses its TTL hint to pin a request’s KV cache for possible reuse by the next step.The scheduler unpins cache when the TTL expires or the program terminates.Source: §5 · p. 7, §5.2 · p. 8

5.1 Tool Call Handler

• The handler parses tool calls from completed LLM responses and estimates tool duration from the interval until the next request in the same program arrives.The parser extracts the tool name from standardized function-call messages; SWE-Bench bash calls use the first word in the extracted command as the tool name.Source: §5.1 · pp. 7–8

5.2 Efficient Pin with TTL in Scheduler

• The scheduler pins cache without freeing its KV blocks and tracks expiration; expired pins remain protected when a follow-up request is already waiting.When a program’s last step finishes, remaining pins for that program are proactively released.Source: §5.2 · p. 8
• If pinned requests consume all GPU memory and prevent scheduling, Continnum unpins victims in latest-program-arrival order until a request can run.The victim is removed from its queue, its KV cache is freed, and it is re-queued as needed.Source: §5.2 · p. 8

5.3 Implementation

• The implementation adds three handler functions to vLLM for recording tool-call start and finish times and setting a request’s TTL.The system also profiles CPU–GPU bandwidth and prefill time by context length for online cost estimation.Source: §5.2 · p. 8, §5.3 · p. 8

6.1 Setup

• Continnum is evaluated with Llama-3.1-8B, Llama-3.1-70B, and Gemma-3-12B on A100-SXM, H100, and B200 GPUs, using SWE-Bench, BFCL V4 (Web Search), and OpenHands workloads.Baselines include vanilla vLLM, CPU DRAM offloading, Autellix, and InferCept.Source: §6.1 · pp. 8–9

6.2 End-to-End Experiments

• Up to 2x lower average response time than vanilla vLLM is reported for Llama-3.1-8B, with improvements across model sizes and hardware configurations.The experiments also report significant average-response-time and throughput improvements on BFCL and SWE-Bench.Source: Figure 8 · p. 9, §6.2 · p. 9, Figure 10 · p. 10
• Continnum improves average delay on OpenHands and P90 and P95 latency on SWE-Bench, and outperforms CPU-offloading baselines.The reported P90/P95 result uses Llama-8B on one B200 with 200GB CPU offloading per GPU; OpenHands uses Llama-8B on one H100.Source: Figure 9 · p. 9, Figure 11 · p. 10, §6.2 · p. 10
• On real SWE-agent tasks, Continnum achieves lower average delay at equal pass rates and has higher pass rates than baselines.The 500-task SWE-Bench-Verified experiment runs on Company A’s internal H100 testbed; baseline runs exceeding the 15-minute container limit are treated as failures.Source: Figure 12 · p. 10, §6.2 · p. 10

6.3 Sensitivity Analysis

• Continnum’s improvements remain stable across tested batch sizes and chunk sizes, while baseline delay worsens as simulated turn counts increase.With more turns, Continnum maintains stable, low-latency performance; it also improves average delay with SSD offloading across tested disk sizes.Source: Figure 13 · p. 11, §6.3 · p. 11, Figure 14 · p. 11

6.4 Ablation Studies and Microbenchmarking

• The ablation reports gradual performance improvements from program-level FCFS and static TTL, while scheduling overhead remains in the single-digit milliseconds.A microbenchmark also reports higher single-node rollout throughput than ThunderAgent for OpenHands with GLM-4.5-fp8 on an 8xH100 node.Source: Figure 16 · p. 11, §6.4 · p. 12, Table 4 · p. 12

• Continnum extends LLM inference resource-management work to long-horizon, multi-turn agent workloads.Prior approaches include kernel optimizations, continuous batching, chunked prefill, CPU/DRAM and disk offloading, and distributed-inference techniques.Source: §7 · p. 12
• Unlike traditional TTL uses, Continnum applies TTL to GPU KV-cache retention based on predicted tool-call durations, scheduling-side delay, and workload patterns.The passage characterizes this as the first such use of TTL for LLM KV-cache regulation.Source: §7 · p. 12
• Continnum is optimized for ReAct-style tool-interleaving; speculative branches, asynchronous multi-agent coordination, and context folding remain future-work extensions.These non-linear workflows are described as mostly experimental and not yet tested in real production workloads.Source: §7 · p. 12

• Agentic workloads pose scheduling challenges through frequent tool calls, variable inter-step delays, and the need to preserve multi-turn continuity.Source: §8 · p. 12
• Continnum balances KV-cache reuse benefits against GPU-memory blocking costs using TTL-based pinning and program-level FCFS.The system is described as reducing unnecessary prefills, mitigating per-turn queueing delays, and adapting to unpredictable tool-call latencies.Source: §8 · p. 12
• Continnum reports consistent end-to-end job-completion improvements across model sizes, hardware configurations, and real-world agent workloads.Source: §8 · p. 12