PaperlayerView exact revision on arXiv

Source-linked AI summary

Semantics, Workflows, and Infrastructure: Understanding Agent Serving at Production Scale

Yihao Zheng, Jingzhe Jiang, Dejiang Zhu, Zhiyuan Tan, Yang Tian, Tao Wang, Minchen Yu

arXiv:2609.34432v1cs.DC

TL;DR

Research has not yet connected task initiation and workflow execution to infrastructure-level agent-serving demand. This paper analyzes a two-week production trace across those levels, finding concentrated request volumes, rare sibling-request overlap, and context reuse across task boundaries.

  • Problem

    How task initiation and workflow execution shape inference demand remains insufficiently characterized, despite request-level workload measurements and studies of agent execution.

    Source: §1 · p. 1

  • Method

    The study analyzes a two-week production trace of 11.7 million requests and 948.4k sessions on a platform backed by over 10,000 GPUs.

    Source: §3 · p. 3

  • Results

    The top 5% of sessions account for 56.1% of requests, logical sibling requests rarely overlap, and context reuse can span task boundaries.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The findings motivate cross-layer abstractions, episode-based scheduling, and adaptive KV-cache management for agent serving.

    Source: §1 · p. 2

    Exact cluster sizes and total workload volumes are not disclosed for confidentiality reasons.

    Source: §3 · p. 3

Abstract

from arXiv · show

Large language model (LLM) agents execute applications through a workflow of inference requests with tool calls and user interactions. Serving these applications at production scale requires understanding how application behavior shapes inference demand and for guiding efficient execution. Recent characterization studies provide request-level workload measurements and agent execution analysis. However, an end-to-end view connecting task initiation, workflow execution, and inference infrastructure remains unexplored. In this paper, we analyze a two-week trace of 11.7 million requests from a large-scale production platform for general-purpose agents, backed by inference infrastructure comprising over 10k GPUs. We characterize the platform at three connected levels: task-level initiation semantics, workflow-level execution patterns, and infrastructure level serving demands. Our measurements reveal workload patterns such as highly skewed request volumes across sessions, rare execution overlap among logical sibling requests, and context reuse across task boundaries. Building on these observations, we analyze deployment implications and identify open problems to guide future research on agent serving systems.

1 Introduction

This paper connects task initiation, workflow execution, and infrastructure demand in a production agent platform. Its trace analysis finds concentrated request loads, rarely overlapping logical siblings, and context reuse that crosses task boundaries.

Study approach

  • The study analyzes a two-week production trace of 11.7M requests and 948.4k sessions, connecting task initiation, workflow DAGs, request execution, resource demand, and context reuse.It uses over 10k GPUs and examines placement and state-management implications.Source: §1 · p. 1

Task-level initiation semantics

  • Agent-triggered tasks comprise 35.8% of tasks but have lighter per-task workloads; serial tasks account for 77.6% of requests despite being less than half of tasks.Only 0.225% of logical sibling request pairs overlap in service time.Source: §1 · p. 2
  • The top 5% of sessions account for 56.1% of requests, while user-triggered workloads vary more in task size and arrival timing than agent-triggered activity.Recurring task groups with median recurrence intervals of at most 30 minutes contain 88.3% of recurring tasks.Source: §1 · p. 2

Infrastructure-level serving demands

  • Context reuse can cross task boundaries, while context changes can also occur within a task; episodes group requests continuing historical context within a retention horizon.The offline retention model finds that longer waiting-time tails do not necessarily warrant longer cache lifetimes.Source: §1 · p. 2

Implications and open questions

  • The findings motivate cross-layer abstractions, episode-based scheduling, and adaptive KV-cache management as directions for agent-serving research.The authors report that adaptive cache lifetimes can reduce recomputation compared with a uniform lifetime setup.Source: §1 · p. 2

2 Background

The background defines sessions, tasks, and requests as connected units of agent execution, while distinguishing workflow organization from request timing. Prior workload studies cover request-level or agent-execution aspects; this study connects them to task initiation and serving demand.

2.1 Agent Workflow Execution

  • A session preserves shared context across tasks, a task pursues a specific goal, and each inference request is one model invocation that may be interleaved with tools.Tasks can begin through user interaction or automated activation.Source: §2.1 · p. 2
  • Workflow DAGs describe serial continuations and branches, but tool execution and external waiting can separate successive requests, so structure alone does not determine execution timing.Model outputs, tool results, and user feedback can alter request relationships as work proceeds.Source: §2.1 · p. 2

2.2 Workload Characterization

  • Prior work measures request arrivals and demand [6], client heterogeneity [5], and coding-agent workflows, context behavior, or cache management [7], [8], [12].This study complements those efforts with an end-to-end view linking task initiation, workflow execution, and infrastructure demand.Source: §2.2 · p. 3

3 Workload Overview

The workload overview describes a large production platform and characterizes its model demand, temporal activity, and request profiles. It reports long-tailed input contexts, short outputs, and distinct timing patterns for user messages and automated tool loops.

Platform and model demand

  • The two-week sampled trace contains 11.69 million requests, 1.42 million tasks, and 948.4k sessions across a platform with over 1,000 model instances and 10,000 GPUs.Four model groups account for 88.58% of requests; exact cluster sizes and total workload volumes are not disclosed.Source: §3 · p. 3

Request characteristics

  • Median input and output lengths are 70,026 and 188 tokens, respectively, while median cached-input fraction is 98.1%.Both input and output length distributions have long tails; 6.34% of requests have no cached input.Source: §3 · pp. 3–4

Activity over time and request characteristics

  • User-message and auto-tool-loop start intervals have medians of 16.95 and 8.29 seconds, respectively; weekday-night arrivals fall to 17.6% of daytime levels.GPU power remains at 80.0% of its peak during weekday nights, while utilization drops from 74.8% to 61.4%.Source: §3 · p. 3, Figure 3 · p. 3

Task-level profiles

  • Figure 4 compares trigger prevalence, task request counts, DAG depth and width, request lengths, and task activity spans across user- and agent-triggered work.The distributions use task-weighted ECDFs for task metrics and request-weighted ECDFs for input/output lengths.Source: Figure 4 · p. 4

Session workload

  • Figure 5 presents requests per session and cumulative request share across sessions ranked by size and split by trigger composition.Panel (a) reports a session-weighted ECDF with P50 and P95 markers.Source: Figure 5 · p. 4

4 Task-level Initiation Semantics

Task initiation differs by trigger in prevalence, workload, and timing. User-triggered work has greater variation in task size and arrival patterns, while recurring automated work is concentrated in short-interval groups.

Trigger prevalence and workload profiles

  • User-triggered tasks are 64.17% of tasks versus 35.83% for agent-triggered tasks, and have a wider request-count tail: P90/P99 are 12/39 versus 5/16.Both categories have a P50 of two requests per task.Source: §4 · p. 4
  • User-triggered requests have median input/output lengths of 52.0k/206 tokens versus 27.4k/136, and task activity medians of 21.1 seconds versus 7.6 seconds.User-triggered input lengths also span a wider range: P25/P75 are 32.2k/76.5k tokens versus 23.3k/38.0k.Source: §4 · pp. 4–5

Work volume and concentration

  • The top 5% of sessions by request count account for 56.1% of requests, and user-only sessions contribute 80.1% of requests within that group.User-only sessions contribute 61.0% of requests overall.Source: §4 · p. 4

Activity rhythms and workload variability

  • User-triggered activity follows workday rhythms and has start-gap P50/P99 of 82.4 seconds/8.07 hours, while agent-triggered activity is more evenly distributed through the day.The figure reports within-trigger task-start shares and marks weekends.Source: Figure 6 · p. 4, §4 · p. 5

Recurring activity

  • Recurring groups with median gaps of at most 30 minutes contain 88.3% of recurring tasks, and task-weighting shifts the median gap from 30 to 5.5 minutes.Short-interval groups contribute more heavily to recurring activity.Source: §4 · p. 5

5 Workflow-level Execution Patterns

The section characterizes how workflow structure relates to request load and execution timing, from DAG organization to request transitions and tool-associated waiting. Serial tasks carry most request load, while long inter-request intervals and tool-category differences qualify how workflows consume time and reuse context.

5.1 Structural Organization and Volume

  • Singleton and serial structures at both workflow levels account for 89.46% of user-triggered tasks and 99.82% of agent-triggered tasks; request graphs have unit width in 82.10% of sessions.Session request and task DAGs have median depths of 2 and 1, respectively, and both have median width one.Source: §5.1 · p. 5, Figure 9 · p. 6
  • Serial tasks make up 45.82% of tasks but contribute 77.62% of requests and 70.38% of cumulative execution duration.Their execution-duration P50/P99 are 17.6/588 seconds, compared with 4.0/138 for singleton and 157/2,525 for branching tasks.Source: §5.1 · p. 6, Figure 9 · p. 6
  • Branching tasks are 1.63% of tasks yet account for 7.04% of requests and 11.21% of cumulative execution duration.They typically contain more requests than serial tasks: P50/P99 of 10/63 versus 4/39.Source: §5.1 · p. 6, Figure 9 · p. 6

5.2 Workflow Execution Dynamics

  • Across tasks, continuation-gap medians are near one second, but P99 reaches 25.72 minutes for user-triggered successors and 21.65 seconds for agent-triggered successors.Within serial tasks, median interval and execution times both rise with chain length, while execution retains a median span share above 50% in every group.Source: §5.2 · p. 7, Figure 10 · p. 7
  • A single inter-request interval exceeds half the span in 12.01% of serial tasks, which account for 41.18% of cumulative serial-task span.The pattern also occurs in 8.8% of tasks with 8–15 requests and 5.7% with at least 16 requests.Source: §5.2 · p. 7, Figure 10 · p. 7

5.3 Request Transition and Tool Calls

  • Request-transition gaps are typically around one second, and command plus search/read account for 81.25% of tool calls overall; command appears in 59.15% of intervals exceeding half a task span.These shares describe tool-call frequency and presence during long waits, not tool execution time.Source: §5.3 · p. 8, Figure 12 · p. 8, Figure 14 · p. 8
  • Ask-user single-tool transitions have P50/P99 gaps of 47.48/1,515.02 seconds versus 1.39/876.08 seconds for sub-agent transitions.Search/read and edit/write have shorter tails; search/read or sub-agent intervals have higher median tool-return totals than command or ask-user intervals.Source: §5.3 · p. 8, Figure 12 · p. 8
  • Ask-user and sub-agent successors have median cached-input fractions near 99%, but 22.82% and 10.68%, respectively, have less than half their input served from cache.Actual reuse reflects both context continuity and cached state available when execution resumes.Source: §5.3 · p. 8, Figure 12 · p. 8

6 Infrastructure-level Serving Demands

Serving demand concentrates in a common group of sessions, while cache affinity follows context and retention continuity rather than task boundaries. Retention decisions trade memory occupied during waits against recomputation on return.

6.2 Cache Affinity Episode Within Sessions

  • An episode groups requests continuing one context history within a retention horizon; context changes or longer idle intervals start new episodes, while subagent histories are tracked separately.The 300-second horizon shown is a grouping parameter, not a measured eviction time or optimal TTL.Source: §6.2 · p. 9

6.1 Concentrated Serving Demand

  • 14.3% of sessions account for 98.5% of context residency, 92.3% of new-prefill tokens, and 86.0% of decode tokens; decode is 76.1% of their request duration.The group is the intersection of the top 20% of sessions ranked independently by each demand measure.Source: Figure 15 · p. 9, §6.1 · p. 9

6.2 Cache Affinity Episode Within Sessions

  • Ordinary continuations reach 93% measured reuse, while model switches and session starts often have little reuse; subagent launches, compaction, and prompt rewrites retain high-reuse upper tails.For the latter events, P95 reuse exceeds 97%; reuse also weakens with longer idle intervals, though high-reuse continuations remain.Source: §6.2 · p. 9, Figure 16 · p. 10, Figure 17 · p. 10
  • Cross-task continuations within an episode retain 80.59% actual token reuse, versus 44.67% for same-task continuations across episodes.The reported upper bounds are 91.85% and 71.98%, respectively; actual reuse is distinct from potentially reusable prefix content.Source: §6.2 · pp. 9–10, Table 4 · p. 10

6.3 Dynamic State Retention

  • TTL decisions should adapt to waiting type, reusable context, and available resources because extending retention trades memory occupied during a wait against recomputation on return.A common retention horizon covers different fractions of observed returns across waiting types; single-tool gaps exceed 300 seconds for 0.7% of command transitions and about 7% each for sub-agent and ask-user.Source: §6.3 · p. 10

7 Implications and Open Questions

The section discusses cross-layer abstractions, episode-based scheduling, and adaptive KV-cache management as directions for agent serving, including an offline retention-allocation model. It frames unresolved questions around which signals to expose, when to reassign work, and how to allocate memory as estimates and resource pressure change.

7.1 Cross-Layer Abstraction

  • Cross-layer abstractions could expose task-initiation and workflow signals for capacity planning and scheduling, but their value depends on timeliness, informativeness, and what infrastructure already infers.The open question is which dynamic information to expose beyond request-level observations.Source: §7.1 · p. 11

7.2 Episode-Based Scheduling

  • Episode-based placement must weigh expected queueing and execution-time reductions against transferring state or rebuilding context, while uncertain remaining work makes reassignment timing difficult.Overestimating remaining work can trigger an unrecovered move; underestimating it can prolong imbalance.Source: §7.2 · p. 11

7.3 Adaptive KV Cache Management

  • At matched retention budgets, type-dependent TTLs reduce remaining recomputation by 6.4%, 1.0%, and 13.9% against the best feasible uniform TTL at the 5-, 15-, and 25-minute reference budgets.Gains are not uniformly large or monotonic; TTL allocation depends on recomputation avoided per added retention cost, not equalizing wait fractions beyond TTL.Source: Figure 18 · p. 11, §7.3 · p. 11
  • Online KV-cache management must balance waiting contexts against active requests and revise retention as estimates of return, timing, reusable state, and resource pressure change.The offline allocation model uses four waiting types and constrains cumulative retention cost, whereas online retention faces an instantaneous memory limit.Source: §7.3 · pp. 11–12

8 Other Related Work

Related work spans agent execution, evaluation, inference serving, and systems that exploit structure, dependencies, locality, or cache reuse.

8 Other Related Work

  • Agent methods and frameworks support reasoning, tool use, and collaboration.The paper situates its analysis alongside work on agent execution approaches.Source: §8 · p. 12
  • Benchmarks and experimentation frameworks evaluate execution costs and serving performance, while LLM serving systems optimize computation, batching, and resource use.Source: §8 · p. 12
  • Execution, scheduling, and KV-cache systems exploit program structure, request dependencies, context locality, and context reuse; this study complements them with production trace evidence.The stated goal is to inform serving optimizations such as request scheduling and state management.Source: §8 · p. 12

9 Conclusion

The conclusion presents a three-level characterization of agent workloads on a large production platform and connects its observations to future serving research.

9 Conclusion

  • The study uses a two-week trace of 11.7 million sampled requests across 948.4k sessions on a platform backed by over 10k GPUs.Source: §9 · p. 12
  • Its analysis connects task-level initiation semantics, workflow-level execution patterns, and infrastructure-level serving demands, examining request load, execution timing, and context reuse.Source: §9 · p. 12
  • The observations motivate further research on cross-layer abstractions, episode-based scheduling, and adaptive KV-cache management; sanitized traces are planned for public release.Source: §9 · p. 12

Related papers

Loading 2609.34432v1…