Source-linked AI summary
CONCUR: High-Throughput Agentic Batch Inference of LLM via Congestion-Based Concurrency Control
Qiaoling Chen, Zhisheng Ye, Tian Tang, Peng Sun, Boyu Tian, Guoteng Wang, Shenggui Li, Yonggang Wen, Zhenhua Han, Tianwei Zhang
TL;DR
Agentic batch inference can suffer middle-phase thrashing as long-lived agents cumulatively pressure the GPU KV cache, degrading throughput before memory capacity is exhausted. CONCUR introduces feedback-driven, agent-level admission control inspired by congestion control, and reports throughput improvements of up to 4.09× on Qwen3-32B and 1.90× on DeepSeek-V3 while remaining compatible with existing serving systems.
Problem
Middle-phase thrashing is an under-characterized pathology in which cache efficiency collapses as long-lived agents accumulate state during batch inference.
Method
CONCUR is a lightweight middleware that uses cache-aware, AIMD-inspired feedback to regulate agent admission rather than individual generation requests.
Results
4.09× throughput improvement on Qwen3-32B and 1.90× on DeepSeek-V3 was achieved while remaining compatible with existing serving engines and agent frameworks.
Takeaways & Limitations
Agent-level concurrency control can stabilize KV-cache efficiency and improve throughput across large-model agent workloads.
Abstract
from arXiv · showhide
Batch inference for agentic workloads stresses the GPU key-value (KV) cache in a sustained and cumulative manner, often causing severe throughput degradation well before memory capacity is exhausted. We identify this phenomenon as middle-phase thrashing, a previously under-characterized pathology in which cache efficiency collapses as long-lived agents accumulate state over time. We argue that mitigating this pathology requires moving beyond reactive, request-level cache management to proactive, agent-level admission control. Drawing inspiration from congestion control in distributed systems, we view the KV cache as a shared resource whose efficient utilization depends on feedback-driven regulation. Based on this insight, we present CONCUR, a lightweight control layer that regulates agent admission to bound aggregate cache pressure while preserving execution continuity. CONCUR adapts a cache-aware control algorithm to dynamically adjust the number of active agents using runtime cache signals. Across large models and real-world agent workloads, CONCUR prevents middle-phase thrashing and improves batch inference throughput by up to 4.09x on Qwen3-32B and 1.9x on DeepSeek-V3, while remaining compatible with existing LLM serving systems.
1. Introduction
Agentic batch inference creates sustained KV-cache pressure as long-lived contexts grow and asynchronous agents make request-level cache management ineffective. CONCUR addresses this with proactive, AIMD-inspired agent admission control, preventing middle-phase thrashing and improving throughput.
- Agent contexts grow monotonically, turning GPU-resident KV cache into a contended, dynamically evolving shared resource.
- Standard LRU eviction can remove inactive but semantically critical prefixes, forcing recomputation or host-device transfer when agents resume.
- Middle-phase thrashing is a prolonged inefficiency period in which GPU memory remains saturated while cache hit rates collapse and throughput degrades.
- Adding more agents during the middle phase can reduce overall throughput despite unchanged model size and hardware resources.
- CONCUR regulates agent admission with an AIMD-inspired, cache-aware control loop that bounds aggregate KV-cache pressure while preserving active-agent continuity.
- 4.09× throughput improvement on Qwen3-32B and 1.90× on DeepSeek-V3 was achieved while remaining compatible with existing serving engines and agent frameworks.
2. Background
Agentic workloads create long-lived, dynamically growing KV-cache demand that exposes limits in recency-based caching and CPU offloading. The paper reframes KV-cache management as congestion control and motivates agent-level regulation.
- ReAct agents repeatedly append thoughts, tool outputs, and observations across long-horizon, multi-turn workflows.
- Agent context and KV-cache footprints grow monotonically, making KV cache a long-lived shared system resource rather than a short-lived optimization.
- Prefix caching works well for short independent requests with stable prefix reuse, but agentic workloads are long-lived and dynamic while eviction remains recency-based.
- CPU offloading can degrade under high concurrency because simultaneous transfers contend for PCIe bandwidth and increase synchronization overhead.
- AIMD cautiously increases a congestion window when congestion is absent and reduces it multiplicatively when congestion is detected.
- CONCUR adapts congestion-control principles to agent-level admission because KV-cache overcommitment produces degraded efficiency, fragmentation, and latency before capacity is exhausted.
3. Middle-Phase Thrashing in Agentic Batch Inference
Agentic batch inference develops middle-phase thrashing because asynchronous agents cause repeated KV-cache eviction and recomputation. The resulting low cache efficiency can degrade throughput even while GPU memory remains near saturation.
- Asynchronous tool pauses make inactive but important agent prefixes vulnerable to LRU eviction, forcing full-prefix recomputation when agents resume.
- Real-world traces show a three-phase execution pattern: warmup, prolonged middle-phase thrashing, and cooldown.
- Warmup Phase: KV-cache hit rates approach 90% during warmup, when shallow contexts and shared prefixes keep the working set within memory limits.
- Middle Phase with Thrashing: During the middle phase, cache usage stays near 80-100% while hit rates remain low, creating an eviction-recomputation cycle.The middle phase dominates over 90% of total execution time.
- Middle Phase with Thrashing: 49.1% of end-to-end latency is spent on extra recomputation during middle-phase thrashing.
- The primary bottleneck is reactive memory management during the volatile middle phase rather than peak memory capacity alone.This motivates shifting from request-level scheduling and reactive eviction toward proactive agent-level admission.
4. CONCUR
CONCUR inserts a lightweight agent-level controller between agent execution and the serving engine to regulate active-agent concurrency using runtime KV-cache signals. Its cache-aware AIMD control preserves agent continuity while limiting aggregate cache pressure.
- System Overview: CONCUR interposes a lightweight Agent-Level Controller between existing agent execution frameworks and LLM serving engines.
- Agent-Level Controller: Agent-level control regulates aggregate active concurrency using real-time KV-cache usage and hit rates to prevent middle-phase thrashing.
- Cache-Aware Admission Control Algorithm: The algorithm treats active-agent count as a congestion window and uses KV-cache usage as a proactive congestion signal and cache hit rate as a reactive failure signal.
- Cache-Aware Admission Control Algorithm: The U_low-to-U_high buffer absorbs discrete memory spikes and permits high concurrency at saturation when cache hit rates remain healthy.
- Cache-Aware Admission Control Algorithm: When usage is below U_low, the controller additively increases concurrency; when usage is high and hit rate is low, it multiplicatively cuts concurrency.
- Agent-Level Controller: The controller admits, pauses, and resumes agents rather than independently scheduling generation requests, preserving execution state across tool execution.
5. Evaluation
The evaluation examines throughput, KV-cache behavior, and fixed versus adaptive admission control under increasing concurrency. CONCUR achieves low latency by regulating agent admission and maintaining cache efficiency, while fixed policies trade off underutilization against thrashing.
- Evaluation Design: CONCUR is evaluated on throughput, KV-cache behavior, and fixed versus adaptive admission control under increasing concurrency.The experiments compare multiple serving and admission-control baselines across representative large models and workloads.
- End-to-End Performance: CONCUR consistently achieves the lowest latency across Qwen3-32B and DeepSeek-V3 configurations, with larger gains at high concurrency.For Qwen3-32B, it reduces latency by up to 4.09× versus SGLang and 2.8× versus request-level admission.
- KV-Cache Behavior: Agent-level admission bounds the aggregate KV-cache working set, preserving cache hit rates and stable batch efficiency during execution.This prevents middle-phase over-admission and explains CONCUR’s advantage over fixed-cap and cache-centric baselines.
- KV-Cache Behavior: At large batch sizes, SGLang’s KV-cache hit rate collapses to 35.41%, while request-level admission falls to 32.21%.HiCache retains high hit rates through CPU offloading, but cache retention alone does not establish the same end-to-end latency advantage.
- Static versus Adaptive Control: CONCUR reaches 846 ms latency, improving 1.5–2.9× over the best fixed levels and 2.99× over the baseline.Small fixed levels underutilize resources, whereas larger levels cause memory overcommitment and KV-cache thrashing.
6. Related Work
Related work addresses KV-cache efficiency through disaggregation, prefix caching, application-aware scheduling, offloading, and cache pinning. CONCUR differs by proactively regulating agent concurrency with congestion-control-inspired feedback to limit temporal overcommitment.
- Disaggregated Memory and Prefix Caching: Disaggregated memory and prefix-pooling approaches improve spatial KV-cache efficiency but do not address temporal overcommitment during concurrent context growth.The paper positions proactive flow control as complementary to these designs.
- Application-Centric Serving: Application-centric systems exploit DAGs, dependency awareness, or memory prediction to reorder requests and reduce blocking in multi-turn workloads.These approaches assume memory pressure can be handled through eviction or reordering.
- Agent-Native Inference and Concurrency: TokenCake and Continuum use reactive mechanisms such as offloading and TTL-based cache pinning for agent-native inference.CONCUR instead proactively regulates agent concurrency using AIMD to bound memory pressure and avoid middle-phase thrashing.
7. Conclusion
The conclusion identifies middle-phase thrashing as a bottleneck in offline agentic inference and presents CONCUR as a lightweight, practical agent-level control layer. The results support flow-control-inspired mechanisms for scalable LLM inference.
- Conclusion: Agentic batch inference creates sustained, cumulative GPU KV-cache pressure that existing request-level scheduling and reactive memory management do not address.The paper identifies this pressure as a source of performance pathology in offline agentic workloads.
- Conclusion: CONCUR regulates agent concurrency with cache feedback to stabilize memory efficiency and improve throughput.The system is presented as a lightweight system layer for proactive admission control.
- Conclusion: The results suggest a broader role for flow-control-inspired mechanisms in scalable LLM inference systems.This conclusion extends the paper’s supported implication beyond CONCUR’s specific implementation.
A.1. Sensitivity Analysis of Utilization Thresholds
Sensitivity analysis shows that Uhigh is robust across a moderate range, whereas Ulow requires careful calibration. The selected thresholds balance capacity exploration against timely congestion response.
- Impact of Uhigh: Moderate Uhigh values of 0.5–0.6 yield consistently low and stable latency across tensor-parallel configurations.Increasing Uhigh from 0.5 to 0.6 causes only modest latency increases of 24 ms, 18 ms, and 99 ms for TP8, TP4, and TP2.
- Impact of Ulow: Setting Ulow to 0.1 keeps the controller in a low-concurrency regime, producing 7–8× higher latency for TP8 and TP4.The threshold is too low for cache usage to fall below it, preventing window increases.
- Threshold Selection: The selected thresholds (Ulow, Uhigh) = (0.2, 0.5) balance capacity probing with prevention of premature window increases.The controller increases its window when cache usage is genuinely low while avoiding over-admission when usage is healthy.