Source-linked AI summary
From LLM Inference to Agentic Workloads: Characterization and Implications for Serving Systems
Chaokun Chang, Yukun Zhou, Kaihua Fu, Dakai An, Tianyu Feng, Hanfeng Lu, Sheng Yao, Pu Guo, Yinghao Yu, Yizhou Shan, Bo Li, Binhang Yuan, Wei Wang
TL;DR
Agentic workloads remain poorly characterized because existing serving assumptions target conventional inference rather than heterogeneous, stateful executions. AgentSysBench benchmarks ten applications using controlled experiments and production traces, finding dynamically shifting, cross-stack behavior and enabling optimizations such as 29–40% lower latency through task-disaggregated serving.
Problem
Agentic serving lacks unified evidence across broad applications and production traces about latency sources, state growth, shifting bottlenecks, and control-plane overhead.
Method
AgentSysBench combines ten representative agentic applications with unified systems-level instrumentation, controlled characterization, and production-trace analysis.
Results
Agentic workloads are heavyweight, heterogeneous across resource types, dynamically shifting, and marked by idle-but-live state, control-plane tax, and cross-request redundancy.
Takeaways & Limitations
Characterization-guided serving mechanisms improve efficiency, including 29–40% lower latency, up to 4.5× speedup, and 4.6× less memory.
Takeaways & Limitations
Controlled characterization uses specific single-server deployments, containerized modules, and selected model-serving configurations.
Abstract
from arXiv · showhide
Agentic applications are shifting AI serving from isolated model inference to long-running workloads in which LLMs coordinate tools, environments, and persistent state. However, the system behavior of these workloads---where latency, cost, and bottlenecks arise---remains poorly characterized, leaving serving systems to rely on assumptions built for conventional inference. We present AgentSysBench, a benchmark suite and measurement toolkit with ten representative agentic applications and unified systems-level instrumentation. Across controlled deployments and production traces, we identify six properties that distinguish agentic workloads from conventional LLM serving: (1) execution is heavyweight and stateful, with non-LLM components dominating latency in 5 of 10 applications and sandbox working-set memory peaking at 28 GB per session; (2) applications compose components with heterogeneous resource affinity---GPU-bound inference, memory-bound retrieval, CPU-bound sandboxes---whose task latencies diverge by up to 32x; (3) bottlenecks shift across requests, models, and deployments; (4) production sessions hold state idle for minutes to hours between active steps; (5) a control-plane tax---auxiliary LLM calls and context overhead from tool schemas and observations---crowds out productive compute and context; and (6) production traces from three applications reveal heavy cross-request redundancy in search queries and web fetches, exposing a large caching opportunity. Four design explorations demonstrate that these findings are actionable: task-aware serving reduces latency by 29--40%, communication-aware placement by up to 4.5x, state offloading reduces memory usage by 4.6x, and tool-result caching removes 35.2% of redundant search calls and saves 19.3% of aggregate search latency.
1 Introduction
Agentic workloads extend LLM serving into long-lived, stateful sessions that combine model inference, tool execution, environment interaction, and persistent state, yet their systems behavior remains poorly characterized. AgentSysBench addresses this gap with unified instrumentation, controlled experiments, and production traces that expose workload heterogeneity, shifting bottlenecks, idle state, control-plane overhead, redundancy, and actionable optimizations.
- Workload definition: Agentic workloads combine model inference, tool execution, environment interaction, and state management under LLM-driven control across long-lived sessions.A single request may trigger retrieval, code execution, browser or GUI interaction, API calls, and dozens of model invocations.
- Benchmark and methodology: AgentSysBench provides ten representative applications, unified instrumentation, and a modular serving stack for controlled, comparable characterization.It records latency, resource usage, data movement, live state footprint, token behavior, and cost across separately deployed or co-located components.
- Systems findings: AgentSysBench reveals heterogeneous task resources and shifting bottlenecks, showing that application-level averages cannot replace online, per-request adaptation.The study combines controlled experiments with production traces to expose behavior that synthetic workloads miss.
- Production findings: Production traces reveal sessions that remain idle but stateful for minutes to hours, while control-plane tokens for schemas, observations, and safety checks increase context use, latency, and cost.These findings motivate distinguishing waiting from done and using context-budget-aware interfaces and observation compression.
- Design implications: 29–40% lower latency, up to 4.5× gains, 4.6× less memory, and 35.2% fewer redundant search calls with 19.3% less aggregate search latency demonstrate actionable characterization-guided optimizations.The four explorations use task-disaggregated serving, agent-aware co-location, state offloading, and tool-result caching.
2 Background and Motivation
Agentic workloads transform a user request into a stateful execution graph that interleaves LLM calls, tools, environments, and state operations. Characterizing them therefore requires modeling workload distributions jointly with serving systems, instrumenting complete executions, and validating controlled findings against production traces.
- Agentic workload definition: An agentic execution interleaves LLM calls, tool invocations, environment interactions, and state operations under runtime control, forming graphs with chains, branches, loops, and parallel tasks.This differs from treating a chatbot request as a single stateless inference call.
- Agentic workload definition: An agentic workload is a distribution W=⟨R,T,M,O⟩ spanning request characteristics, tools and environments, model policies, and observed outcomes.These factors determine I/O behavior, state footprint, failure modes, latency, cost, and non-LLM bottlenecks.
- Benchmark motivation: Characterization must jointly control the workload and serving system because agentic behavior cannot be inferred from workload properties alone.The observed outcome also depends on how the workload is served.
- Benchmark motivation: Controlled experiments miss production phenomena including realistic arrivals, user think time, long-lived idle sessions, and cross-user redundancy.These behaviors emerge when sessions and requests interact in production rather than running tasks to completion in isolation.
- Benchmark requirements: The benchmark therefore requires representative workload selection, joint workload–system control, complete execution instrumentation, and production-trace validation.Instrumentation covers individual LLM calls, tool invocations, and state operations.
3 Benchmarking Suite
AgentSysBench is a systems-oriented benchmark suite that represents diverse agentic workloads, exposes serving configurations through modular deployments, and produces comparable execution and resource measurements. It also supports extensible application integration under standardized workload, serving-system, tracing, and deployment interfaces.
- Workload representativeness: The suite selects applications across distinct workload regions to preserve realistic task semantics while covering latency, resource demand, state footprint, and cost.Its task regimes span short document-QA requests, minute-scale browser and GUI interactions, and longer AutoML or research jobs with larger artifacts.
- Workload representativeness: Applications exercise heterogeneous tools, environments, orchestration patterns, and model choices, including retrieval, coding, browser, GUI, RAG, ReAct, DeepResearch, and multimodal workloads.These choices expose tool-driven I/O, stateful interfaces, non-LLM bottlenecks, control-flow variance, and inference-policy variance.
- Serving-system control: A modular serving stack separates arrivals, orchestration, tools, models, and deployment, while containerized experiments expose hardware allocation, serving mechanisms, and architecture choices.The stack supports controlled substitutions such as replacing SGLang with vLLM or moving components from embedded calls to isolated services.
- Measurement: Unified component-level instrumentation records comparable traces of latency, sizes, tokens, cost, and CPU, memory, disk, network, and GPU usage.White-box annotations and black-box proxies or sandbox hooks feed the same normalized per-operation trace schema.
- Extensibility and defaults: New applications join the suite by specifying the same workload and serving-system factors, reusing interfaces, tracing paths, arrival patterns, and deployment conversion mechanisms.The default controlled characterization uses containerized deployments, with each non-RAG workflow on one NVIDIA GPU server and RAG on one x86_64 server with 8× 4090D GPUs.
4 Heavyweight with Non-LLM Dominance
Agentic workloads are long-running, stateful executions in which non-LLM components frequently dominate latency, cost, and operational demands. Their large states, high I/O volumes, and resource idle times require serving systems to optimize the full tool-use lifecycle rather than model inference alone.
- Execution latency: Agentic requests span seconds to several hours, with Mini-SWE coding tasks frequently exceeding ten minutes and PiAutoResearch research tasks reaching several hours.These long-running, heavy-tailed executions increase exposure to node and network failures during a single request.
- Execution latency: Non-LLM components dominate the critical path in half of characterized applications, including over 70% of GUIAgent time and 90% of Pi-AutoR time.The decomposed components include sandboxes, search, databases, MCP calls, embedding, general model inference, and orchestration overhead.
- Token consumption: Prefix-cache hit rates reach up to 99% for iterative applications but fall to 1% or lower when context is dynamically restructured or fetched.The passage identifies prefix caching as necessary because cache-hit tokens dominate token consumption in these workloads.
- Data movement: A single agentic execution can generate and transfer tens of megabytes of data, driven especially by sandbox visualization and file synchronization, GUI screen frames, embeddings, and vector databases.GUIAgent produces the largest per-step data volume because its desktop sandbox captures and transmits high-resolution screen frames at each interaction step.
- Operating cost: Non-LLM infrastructure dominates pay-as-you-go cost in tool-heavy applications, with Pi-AutoR sandbox charges constituting over 99% of per-request cost.Sequential execution causes dedicated sandboxes and retrieval environments to continue incurring charges while sessions wait for GPU-bound model inference.
5 Cross-Stack Heterogeneity
Agentic applications combine components with fundamentally different resource demands, while task and invocation workloads vary sharply even within a single application. This cross-stack heterogeneity creates queueing, batching, and shared-resource interference, motivating task-aware serving policies.
- Component heterogeneity: Applications span GPU-bound LLMs, CPU-bound sandboxes, memory-bound vector databases, and network-bound search services; HuggingGPT models range from 16.8 to 6032 GFLOPs.These components have distinct memory footprints and latency profiles.
- Task heterogeneity: 32× separates Embed-Doc from Embed-Query, while W4 is 6.43× slower than W2 despite using the same model.W4 averages 26× more input tokens and 16× more output tokens than W2; Embed-Doc’s payload is 325× larger than Embed-Query’s.
- Invocation heterogeneity: 30× variation occurs for the same LLM task within one Mini-SWE trace, while sandbox latency varies up to 171× across invocations.LLM variation follows per-invocation input and output workload; the longest sandbox invocation is pip install, versus sed as the shortest.
- Performance interference: Heterogeneous tasks cause HOL blocking, co-batching interference, and shared-resource contention, requiring task-aware queueing, batching, and isolation.These mechanisms motivate the task-aware serving design explored in §8.1.
6 Shifting Bottlenecks
Agentic bottlenecks shift with workload factors—including request distribution, model choice, tool set, and orchestration—and serving-system factors such as hardware, serving mechanisms, and deployment architecture. Empirical results show that different request types, model speeds, and deployment choices can change which component dominates end-to-end latency.
- Systematic bottleneck shifts: Bottlenecks shift with request distribution, model choice, tool set, orchestration, hardware resources, serving mechanisms, and deployment architecture.The bottleneck location is therefore not intrinsic to the application alone.
- Request distribution: LLM execution dominates Movie, BI, and DB requests at up to 90% of total time, while tool execution dominates ETL, Wiki, and MAP at up to 84%.These differences arise from request categories using different problems and MCP tools.
- Model selection: 2.43× faster output-token throughput makes V4-Flash produce 51 output tokens/s versus 21 output tokens/s for V4-Pro.The comparison uses DeepResearch configurations with Qwen3.7-Max fixed for planning and Flash or Pro varied for writing and summarization.
- Model selection: The LLM share rises from 38% to 44% to 49% across Flash+Flash, Pro+Flash, and Pro+Pro, overtaking embedding at 49% versus 43% in Pro+Pro.The passage attributes the shift primarily to model-speed differences rather than token volume.
- Serving-system factors: Changing serving configuration—including GPU allocation, component-serving mechanisms, or co-located versus distributed deployment—shifts the bottleneck.Deployment choices alter component placement and communication, while sandbox performance depends strongly on allocated resources.
7 Production-Trace Analysis
Production traces reveal three serving-system properties hidden by isolated benchmarks: sessions retain state while idle, control-plane work consumes context and compute, and repeated tool requests create substantial caching opportunities.
- Idle-but-live sessions: 20% of median session lifetime is spent executing, while 70% of sessions execute for less than half their lifetime.Idle intervals range from seconds to hours, most commonly 1–10 minutes, while correctness-critical sandbox state remains allocated.
- Control-plane tax: 99.7% of initial input tokens are system messages, while history grows to 84.3% and precedes a 151-token action with 166,721 context tokens.Across 35,037 sessions, 3,170 compaction events reduce context by more than 70% in 99% of events.
- Control-plane tax: 156 s is the average compaction latency across 3,170 events, with p95 and p99 latencies of 161 s and 775 s.Each compaction event consumes an average of 176K input tokens and generates 5K output tokens.
- Control-plane tax: 2,684 auxiliary LLM calls add 6.5M input tokens and 50K output tokens across 35,037 production sessions.These calls support tasks including safety guardrails and loop detection outside the main reasoning-and-action loop.
- Cross-session tool redundancy: 67.3% of search invocations come from 27% of distinct queries, while 64% of URL fetches come from 24% of recurring URLs.These results expose shared caching and deduplication opportunities across sessions that single-request benchmarks cannot reveal.
8 Design Explorations
Four proof-of-concept explorations target heterogeneity, communication overhead, idle state, and cross-request redundancy in agentic serving. They improve latency, memory usage, or redundant tool execution through task-aware serving, communication-aware placement, state offloading, and tool-result caching.
- 8.1 Task-aware serving: Task-disaggregated serving isolates heterogeneous logical tasks on dedicated resources, reducing interference and enabling task-specific optimization.Co-serving tasks with different input sizes, latencies, and resource footprints causes head-of-line blocking and performance interference.
- 8.1 Task-aware serving: 40%, 38%, and 29% lower average latency result from task-disaggregated Dynamic RAG deployment at 0.5×, 0.7×, and 0.9× peak throughput.The comparison uses a component-sharing baseline with the same GPU count; improvements primarily come from mitigating queueing delays and interference.
- 8.2 Communication-aware placement: 2.8× and 4.5× lower average latency result from co-locating vector database and embedding tasks under low and high load.The co-vdb-embed strategy reduces transfer overhead, while the co-none baseline spends 67.5% of execution time on network communication.
- 8.3 State offloading: 4.6× lower average memory consumption and 2.1× lower peak memory consumption result from proactive sandbox offloading, with latency increasing within 0.5%.The trigger uses elapsed time of the in-flight planning call rather than context length, and the memory values are aggregate across concurrent sandbox processes.
- 8.4 Tool-result caching: 35.2% of redundant search calls are eliminated with a 10-minute TTL, saving 27 hours of aggregate search latency, or 19.3% of total latency.The two-tier cache uses exact query matching for searches and URL-level deduplication for web fetches within a configurable staleness window.
9 Conclusion
AgentSysBench enables systems-level study of agentic workloads through ten applications and a modular serving stack. Its experiments show that these workloads are heavyweight, heterogeneous, dynamically shifting, and shaped by production behaviors missed by model-centric benchmarks.
- Benchmark: AgentSysBench combines ten agentic applications with a modular serving stack for systems-level study.The benchmark is evaluated using controlled experiments and production-trace analysis.
- Workload characterization: Agentic workloads are heavyweight, cross-stack heterogeneous, and dynamically shifting.These properties emerge from controlled experiments and production-trace analysis.
- Production behavior: Production sessions exhibit long idle-but-live intervals, an LLM control-plane tax, and heavy cross-request redundancy.These production behaviors are invisible to model-centric benchmarks.