PaperlayerView exact revision on arXiv

Source-linked AI summary

ThunderAgent: A Simple, Fast and Program-Aware Agentic Inference System

Hao Kang, Ziyang Li, Weili Xu, Xinyu Yang, Yinfang Chen, Junxiong Wang, Beidi Chen, Tushar Krishna, Chenfeng Xu, Simran Arora

arXiv:2602.13692v3cs.OScs.MA

TL;DR

Existing agentic inference systems separately manage model requests and tool execution, lacking an end-to-end workflow view for stateful resources. ThunderAgent models workflows as programs and uses program-aware scheduling and tool management; across diverse workloads, it reports 1.48–3.58× serving and 1.79–3.92× RL-rollout throughput improvements.

  • Problem

    Existing systems schedule model and tool requests independently without end-to-end workflow knowledge, limiting coordinated management of KV caches and tool environments.

  • Method

    ThunderAgent abstracts each workflow as a persistent program and uses program-aware scheduling plus lifecycle-aware tool-resource management.

  • Results

    1.48–3.58× serving and 1.79–3.92× RL-rollout throughput improvements were achieved across diverse agentic workloads.

  • Takeaways & Limitations

    ThunderAgent provides a program-level inference layer that dynamically schedules workflows across GPU nodes, reduces cache thrashing and memory imbalance, and prevents tool-resource leakage.

Abstract

from arXiv · show

Large language models(LLMs) are now used to power complex multi-turn agentic workflows. Existing systems run agentic inference by loosely assembling isolated components: an LLM inference engine (e.g., vLLM) and a tool orchestrator (e.g., Kubernetes). Although agentic workflows involve multiple LLM and tool requests, these systems schedule and allocate resources separately on a per-request basis, without end-to-end knowledge of the workflow. This leads to sub-optimal management of KV cache and tool execution environments. To address the challenges, we propose ThunderAgent, a fast, simple, and program-aware agentic inference system. We first abstract agentic workflows as LLM Programs, enabling a unified view of heterogeneous resources, including KV caches, system states, and external tool assets such as disk memory and network ports. Built upon this abstraction, ThunderAgent introduces a program-aware scheduler and a tool resource manager designed to maximize KV cache hit rates, mitigate memory imbalances, and enable asynchronous environment preparation. Evaluations across coding, routing, and scientific discovery agents demonstrate that ThunderAgent achieves 1.5-3.6x throughput improvements in serving, 1.8-3.9x in RL rollout, and up to 4.2x disk memory savings compared to state-of-the-art inference systems. To facilitate reproducibility and support future development, we open-source the system implementations of the whole ThunderAgent at: https://github.com/Agentic-Kinetics/ThunderAgent.

1 Introduction

Agentic workloads interleave reasoning and tool calls, but request-level systems manage components separately, causing inefficient cache, memory, and tool-resource management. ThunderAgent addresses this with program-aware abstraction, scheduling, and resource management, improving serving and rollout throughput.

  • Motivation: Agentic workflows combine long reasoning with external tool calls, and throughput degrades as concurrent requests increase.Rollout accounts for over 70% of RL wall-clock time.
  • Challenges: Request-level systems separately schedule inference and tools without end-to-end workflow knowledge, causing KV-cache thrashing, memory imbalance, and tool-lifecycle problems.These systems cannot coordinate GPU memory with program-level resource scheduling.
  • Challenges: KV-cache eviction during tool execution forces interaction-history re-prefill, increasing average end-to-end latency by up to 7.14×.The resulting recomputation decreases throughput.
  • ThunderAgent: ThunderAgent represents each workflow as a persistent agentic program that exposes execution state, tokens, and tool resources to the runtime.The abstraction decouples scheduling from execution backends.
  • ThunderAgent: Its program-aware scheduler pauses tool-acting workflows under memory pressure and migrates programs across GPU nodes to reduce eviction and imbalance.A global program-aware waiting queue enables migration across data-parallel nodes.
  • ThunderAgent: ThunderAgent overlaps tool-environment initialization with reasoning and garbage-collects resources after program termination.The manager tracks dependencies and reclaims Docker sandboxes and network ports.
  • Results: 1.48–3.58× serving and 1.79–3.92× RL-rollout throughput improvements were achieved across diverse agentic workloads.The system also scales near-linearly to 64 H100 GPUs and remains effective with KV-cache offloading.

2 Background

Agentic workflows maintain persistent GPU-memory and tool-environment state across alternating reasoning and acting steps. Existing systems treat these steps as independent requests, motivating an end-to-end program-level inference layer.

  • Workflow Properties: At each step, an agent receives an observation and emits a thought-action pair conditioned on its cumulative interaction history.The workflow alternates between reasoning and acting during generation.
  • Persistent State: KV caches preserve workflow history, while initialized tool environments must remain consistent and accessible throughout execution.Growing contexts can theoretically enable near-complete KV-cache reuse across steps.
  • System Gap: These dependencies require program-level coordination, but existing systems treat each thought and action as an independent stateless request.This prevents coordinated management of heterogeneous resources across long-running workflows.
  • Prior Approaches: Prior work optimizes individual inference or orchestration components, with few end-to-end approaches spanning GPU, CPU, and remote resources.The gap motivates a program-aware scheduling layer.
  • Prior Approaches: Autellix tracks GPU execution time but ignores workflow locality, allowing KV-cache eviction under heavy workloads.This can trigger KV-cache thrashing among concurrent workflows.
  • Prior Approaches: Continuum uses TTL-based KV-cache pinning but remains vulnerable to unpredictable tool durations and GPU-memory pressure.Incorrect TTL estimates can cause thrashing and stranded cache memory.

3 Challenges in Existing Agentic Inference Systems

Existing agentic inference systems expose three coupled inefficiencies: KV-cache thrashing, cross-node memory imbalance, and unsynchronized tool-resource lifecycles. These issues increase latency, waste capacity, and destabilize long-running workloads.

  • KV Cache Thrashing: Request-level scheduling evicts KV caches during tool execution, forcing repeated eviction and re-prefill despite high theoretical cache reuse.Thrashing intensifies as parallel workflow count increases.
  • KV Cache Thrashing: Re-prefill increases end-to-end request latency by up to 7.14× compared with a non-thrashing setting.The redundancy causes severe throughput degradation.
  • Memory Imbalance: Cache-locality routing can overload some nodes while leaving others lightly utilized because workflows have heterogeneous KV footprints and execution lifetimes.Stale router-side KV state further harms cache reuse and cross-node balance.
  • Memory Imbalance: Two data-parallel nodes diverged by more than 20% in memory usage for over 37 minutes, reaching a peak imbalance of 51%.This was observed during a 90-minute agentic RL rollout snapshot.
  • Tool Resources: Inference and tool orchestration lack synchronized resource lifecycles, causing silent resource wastage and latency overhead.The systems do not coordinate external tool resources with the inference engine.
  • Tool Resources: Unreclaimed resources such as Docker images increase disk usage linearly with processed workflows until system capacity is exceeded.The resulting resource leakage can cause fatal instability in long-running workloads.
  • Tool Resources: Environment preparation, including pulling containers and installing packages, becomes more costly as parallel workload count increases.Waiting for preparation to finish extends end-to-end inference latency.

4 ThunderAgent: A Program-Aware Agentic Inference System

ThunderAgent treats each multi-turn agent workflow as a persistent agentic program and coordinates KV-cache, scheduling, and tool resources end to end. Its cost-guided scheduler uses program-aware pausing, restoration, and resource lifecycle management to reduce recomputation, memory imbalance, and idle resource overhead.

  • Program abstraction: ThunderAgent models an agentic workflow as a persistent program that tracks identity, context size, tool environments, node placement, execution phase, and scheduling status.This abstraction exposes logical execution and system-level dependencies to the runtime.
  • System integration: ThunderAgent wraps existing inference engines and orchestrators through OpenAI-style endpoints while integrating program-aware scheduling and tool resource management.The system is designed to support new workflows without changing the execution backends.
  • Cost model: The system’s cost model separates productive decoding and prefilling from recomputation, unused capacity, and idle caching, then minimizes the non-productive components.The decomposition identifies KV-cache thrashing, cross-node imbalance, and tool-execution caching as distinct overheads.
  • Program-aware scheduler: A global program-aware waiting queue uses Restore and Pause operations to schedule programs according to context length and execution phase.Pause releases a program’s KV cache, while Restore admits a paused program when capacity permits.
  • Program-aware scheduler: Periodic monitoring detects KV-cache thrashing during execution, unlike arrival-only checks, and hysteresis watermarks control pausing and restoration under changing memory pressure.The scheduler monitors fixed intervals ∆t and uses λmax and λmin as high- and low-watermarks.
  • Program-aware scheduler: Time decay lowers the effective priority of long-idle acting programs, balancing caching against recomputation and enabling eviction as memory pressure rises.The scheduler also uses shortest-first eviction to minimize recomputation cost.

5 Experiments

ThunderAgent is evaluated across coding, routing, scientific, and distributed RL workloads using multiple models, hardware configurations, and baselines. It improves throughput through program-aware scheduling, KV-cache management, asynchronous environment preparation, and tool-resource management.

  • Experimental setup: Evaluations cover coding, routing, scientific discovery, and RL-rollout workloads across RTX5090, H100, and multi-node deployments.The experiments use OpenHands, mini-SWEAgent, ToolOrchestra, and multiple large language models.
  • Serving results: 1.48–3.58× throughput improvements over vLLM and 1.17–3.31× over Continuum occur at high concurrency across models and datasets.At 96 parallel programs, ThunderAgent maintains near-optimal KV-cache hit rates and supports asynchronous environment preparation.
  • Serving results: ThunderAgent maintains stable throughput beyond GPU-memory limits by automatically adapting to available capacity without manual tuning.Baseline systems experience severe throughput collapse when workload concurrency exceeds memory capacity.
  • Serving results: ThunderAgent trades KV-cache hit rate for higher compute utility when tool execution times are stochastic, balancing recomputation and caching costs.A time-decay function retains cache for short tool calls while pausing programs with long execution times.
  • RL rollout: 1.79–3.92× throughput improvement over the vLLM + Gateway baseline is achieved for distributed RL rollout on two 8×H100 nodes.The evaluation uses GLM-4.6 and reports effective scalability for memory-intensive workloads.
  • Ablation study: Tool-resource management contributes approximately 10% to latency improvement while providing 4.2× disk-memory savings.The overall throughput gain primarily comes from reducing prefill and decode latency.

6 Conclusion

ThunderAgent uses a program-level abstraction that tracks each agentic workflow throughout its lifecycle and supports runtime scheduling and resource management. The system dynamically schedules workflows across GPUs and manages tool resources, improving serving and RL-rollout performance.

  • Conclusion: ThunderAgent represents each agentic workflow with a program-level abstraction that tracks metadata throughout its lifecycle.The abstraction supports runtime scheduling and resource management.
  • Conclusion: Dynamic GPU scheduling mitigates KV-cache thrashing and memory imbalance, while tool-resource management prevents resource leakage.These mechanisms provide the system-level basis for the reported throughput improvements.
  • Conclusion: ThunderAgent improves throughput by 1.48–3.58× for serving and 1.79–3.92× for RL rollouts over previous systems.These are the reported aggregate performance ranges in the conclusion.

Impact Statement

ThunderAgent is presented as an efficiency-oriented system for agent inference and RL rollout. Its reduced memory footprint and hardware requirements are described as supporting more cost-efficient, energy-conscious, and accessible large-scale agent research.

  • Impact: ThunderAgent targets execution efficiency for agentic workflows by reducing the memory footprint and hardware requirements of inference and RL rollout.The stated goal is to improve cost-efficiency and energy sustainability for large-scale agent serving.
  • Impact: The approach can enable larger models and more complex environments to be trained and evaluated on limited hardware resources.The impact statement connects this efficiency to broader access to high-fidelity agent simulations.
  • Related methods: Existing KV-cache methods use GPU HBM, CPU DRAM, and NVMe SSD to offload inactive states and prefetch them when workflows resume.The passage describes this as a hardware-hierarchy approach to alleviating GPU-memory pressure.
  • Related methods: Distributed KV-cache management can be constrained by interconnect bandwidth and frequent state transfers caused by strong intra-program dependencies.The limitation concerns large-batch agentic serving and rollout without program-level management.
  • Related methods: KV-aware routing reduces cross-node transfers through locality or session-based pinning but cannot dynamically migrate active program states between nodes.The passage identifies workload transfer as a missing capability in these approaches.

A.2 Existing KV cache optimization methods

The appendix examines alternatives and extensions involving KV-cache offloading, PD disaggregation, multi-node routing, heterogeneous hardware, and framework integration. ThunderAgent remains effective by balancing cache locality with workload redistribution and by preventing capacity collapse under high concurrency.

  • KV cache optimization: KV-cache offloading with vLLM and LMCache is limited by insufficient PCIe bandwidth for frequent context switching and large data transfers.The experiment investigates offloading as a remedy for capacity constraints in agentic workloads.
  • KV cache optimization: PD disaggregation exacerbates thrashing for agentic workloads with continuous context growth by reducing the effective HBM pool available for prefill.The appendix contrasts this behavior with standard chatbot-serving assumptions.
  • Scaling: ThunderAgent routes paused workflows to nodes with available capacity, balancing KV-cache locality against multi-node load distribution.This global waiting-queue strategy replaces session-based static node pinning.
  • Scaling: From 2 to 8 nodes, ThunderAgent closely follows ideal linear scaling, while its speedup over SGLang’s Cache-Aware Routing Policy widens from 1.79× to 2.39×.The SGLang policy cannot redistribute paused programs, and ThunderAgent’s scheduler communication remains below per-step inference latency.
  • KV-cache integration: ThunderAgent prevents capacity collapse when integrated with SGLang HiCache, keeping P50 and P90 latency near 10 s and 20 s across batch sizes.Throughput grows monotonically with batch size in the reported sweep.
  • Hardware robustness: On 8×A100 GPUs, ThunderAgent reaches 1.71–2.08× speedup on mini-SWEAgent and 1.16–1.62× on OpenHands at higher concurrency.At low concurrency, both systems are similar because the working set fits in HBM; vLLM declines as thrashing begins.

B.1 Middleware Architecture and Unified Interfaces.

ThunderAgent exposes program-level workflow state to coordinate inference and tool execution through a unified runtime interface. This visibility supports resource-aware scheduling under unpredictable tool latency and preserves compatibility with existing request APIs.

  • Unified interfaces: ThunderAgent uses program IDs to expose workflow metadata and distinguish valid tool-wait periods from idle sessions.This enables preemption decisions that session-based routing cannot make.
  • Unified interfaces: Adopting ThunderAgent requires attaching a program ID to inference and tool requests and sending an explicit release signal when a program ends.Other request fields and the OpenAI-style API remain unchanged.
  • Program state: ProgramStatus distinguishes on-GPU reasoning, off-GPU tool execution, globally paused programs, and stopped programs whose resources are reclaimed.
  • Tool execution: Remote-service tools exhibit heavy-tailed latency, with p95 and p99 substantially above the median and tails extending to tens or hundreds of seconds.This makes tool-latency prediction brittle in practice.
  • Tool execution: ThunderAgent balances KV recomputation against idle caching because pinning cache during variable-duration tool calls can reduce recomputation while lowering throughput.A higher cache hit rate is therefore not always sufficient for tool-heavy workloads.

E End-to-End Latency Analysis

The latency analysis frames time decay as a response to unpredictable tool returns and derives the admissible decay forms under time-homogeneity and boundary conditions. The resulting forms are exponential in continuous time and geometric in discrete time.

  • End-to-end latency: ThunderAgent’s average per-step latency outperforms vLLM and Continuum for GLM4.6 and Qwen3 235B with mini-SWEAgent and OpenHands on a single H100.The comparison covers low- and high-parallelism workflow settings.
  • Time-decay derivation: Unpredictable tool returns motivate a decay function that depends only on elapsed acting time in a time-homogeneous manner.The relative decay after an additional interval depends on that interval rather than absolute elapsed time.
  • Time-decay derivation: The decay function equals 1 at zero tool time and approaches 0 as tool time becomes infinite.These boundary conditions connect multi-turn reasoning without tools to request-level scheduling for indefinitely waiting requests.
  • Time-decay derivation: Under the stated assumptions, admissible decay is exponential in continuous time, f(t) = e^-λt with λ > 0, or geometric in discrete time, f(k) = x^-k with x > 1.

F.2 Proof of recomputation STP cost

The recomputation-cost analysis models resumed programs’ rebuilding cost as super-linear in context length, then proves that evicting shortest contexts minimizes the resulting penalty under a reclaimed-memory constraint.

  • Cost model: STP recomputation cost grows quadratically with elapsed decoding time because instantaneous cost increases linearly with the decoding step.The resulting relationship is Costrecompute ∝ t^2.
  • Eviction optimization: The eviction problem selects paused programs whose total reclaimed memory satisfies the capacity requirement while minimizing a convex cost over context lengths.The relevant cost function is f(x) = x^2.
  • Shortest-first policy: The optimal strategy selects programs with the smallest context lengths.This follows from exchanging a selected longer program for an available shorter one.
  • Shortest-first policy: Replacing a large eviction target with smaller components reduces the sum of squared context lengths because the positive cross-term makes aggregation more costly.The exchange argument decreases the objective until only shortest available programs remain selected.
  • Shortest-first policy: Shortest-first is globally optimal because O(L^2) attention cost penalizes aggregation more than fragmentation.

G Additional Ablation Studies

The ablations show that ThunderAgent’s global queue, time-decay policy, shortest-first eviction, and local scheduling each address distinct sources of inefficiency. Their benefits are strongest when memory imbalance, tool-time uncertainty, or context length makes naive policies costly.

  • Global waiting queue: At four H100 nodes, the global waiting queue improves throughput by 1.28× over the local queue by redistributing paused programs across under-utilized peers.The gain widens with cluster size as cross-node memory imbalance grows.
  • Time-decay function: Exponential decay is most robust overall, reaching 1.08× on mini-SWE-Agent and 1.63× on ScienceAgent versus 1.39× for linear decay on ScienceAgent.On OSWorld, linear decay is better by about 6%, at 1.21× versus 1.14× for exponential decay.
  • Eviction policy: Shortest-first achieves the highest throughput and lowest average prefill time on both mini-SWE-Agent and OpenHands.On OpenHands, it is roughly 2.0× the throughput of longest-first and reduces average prefill time from 15.1 s to 7.5 s.
  • Component ablation: Local program-aware scheduling raises throughput from 375 to 602 steps/min, while adding the global queue produces a further 1.12× gain to 672 steps/min.The local component supplies the larger improvement in the component-wise ablation.
Loading 2602.13692v3…