PaperlayerView exact revision on arXiv

Source-linked AI summary

TraceLab: Characterizing Coding Agent Workloads for LLM Serving

Kan Zhu, Mathew Jacob, Chenxi Ma, Yi Pan, Stephanie Wang, Arvind Krishnamurthy, Baris Kasikci

arXiv:2606.30560v2cs.LGcs.AIcs.PF

TL;DR

Serving coding agents efficiently is challenging, yet public traces and benchmarks do not capture their real-world, multi-step usage. TraceLab collects and analyzes cross-provider coding-agent traces, finding long autonomous loops, long contexts paired with short outputs, heavily tailed tool calls, and high but imperfect prefix-cache hit rates. These patterns point to serving opportunities in tool calling, prefill, and KV-cache management.

  • Problem

    Public traces and benchmarks do not capture real day-to-day coding-agent usage across agents and model families, while understanding workload patterns is needed to study efficient serving.

    Source: §1 · p. 1, §2.2 · p. 3

  • Method

    The authors extract, normalize, and anonymize coding-agent logs, then analyze a cross-provider trace of sessions, LLM steps, and tool calls.

    Source: §1 · p. 1

  • Results

    Coding-agent workloads feature long autonomous loops, long contexts with short outputs, strongly long-tailed tool calls, and high yet imperfect prefix-cache hit rates.

    Source: §10 · p. 16

  • Takeaways & Limitations

    The observations point to serving opportunities in lower-overhead tool calling, append-length-specific prefill optimization, and KV-cache eviction or prefetching around human-paced gaps.

    Source: §10 · p. 16

    Because the trace comes from the authors’ day-to-day use, results may not fully generalize to other organizations, workflows, or agent domains.

    Source: §9 · p. 16

Abstract

from arXiv · show

Coding agents are rapidly becoming a major application of agentic LLMs, but serving them efficiently remains challenging. Progress on this challenge requires understanding real workload patterns, yet the data needed for such analysis is largely absent. Existing public traces and benchmarks do not capture real, day-to-day coding-agent usage across multiple agents and model families for serving-system analysis. To help fill this gap, we collect and release a trace of roughly 4,300 coding-agent sessions, containing about 350,000 LLM steps and 430,000 tool calls from our own day-to-day use of Claude Code and Codex. Our analysis shows that coding-agent workloads feature long autonomous loops, long contexts with short outputs, diverse and heavily-tailed tool calls, and high but imperfect prefix cache hit rates. These findings point to concrete opportunities for optimizing serving, including lower-overhead tool calling, append-length-aware prefill, semantic-aware tool-latency prediction, and improved KV-cache management around human-paced gaps. We release the dataset, trace collection pipeline, and analysis code at https://github.com/uw-syfi/TraceLab.git the project website is https://tracelab.cs.washington.edu.

1 Introduction

Serving coding agents is challenging because their sessions accumulate context and repeatedly switch between LLMs and tools, while existing traces and benchmarks do not capture their real-world multi-step usage. TraceLab collects and analyzes cross-provider usage, finding long autonomous loops, long contexts with short outputs, skewed tool calls, and high but imperfect prefix-cache hit rates.

1 Introduction

  • The trace pipeline extracts, normalizes, and anonymizes logs from about 4,300 sessions, comprising roughly 350,000 LLM steps and 430,000 tool calls.The data spans 43 developers, Claude Code and Codex, and more than 20 model versions over roughly eight months.Source: §1 · p. 1
  • Requests average 8.8 LLM calls and 10.8 tool invocations, taking 4.3 minutes on average with a p90 above 6.4 minutes.Context usually grows over iterations, with compaction near the context limit and occasional micro context reductions in Codex.Source: §1 · p. 1
  • The median step contains about 119K prefix tokens, 875 append tokens, and 214 output tokens, while more than 80 observed tools are heavily skewed toward the top three per provider.Those top three tools account for more than 80% of all tool calls for both Claude and Codex.Source: §1 · p. 2
  • A 95.7% global prefix-cache hit rate still leaves costly misses, which cause 3.8× more prefill tokens than truly unique input tokens.Most misses fall between a request’s final output and the next user input, when human-paced delays can outlast cache eviction time.Source: §1 · p. 2
  • Frequent tool–LLM switching motivates denser or fused tool invocations and lower-overhead approval and runtime paths.The proposed directions also include prefill routing and kernel or serving-engine parallelism adapted to append length, plus tool-latency prediction using operation semantics and recent latency history.Source: §1 · p. 2
  • Long prefixes motivate sparse attention, while frequent idle gaps motivate cheaper KV storage, compression, eviction, or prefetching around those pauses.The cited opportunities address reducing decoding cost and managing KV cache around human-paced gaps.Source: §1 · p. 2
  • TraceLab presents a large-scale, cross-provider view of real coding-agent workloads and releases its dataset, trace-collection pipeline, and analysis code.The authors describe it as the first large-scale cross-provider look into real-world coding-agent workload characteristics.Source: §1 · p. 2

2 Background and Motivation

Coding agents use multi-step interactions and persistent session context, but existing datasets and benchmarks do not capture the real-world usage patterns needed to characterize their serving workloads. The paper addresses this gap with a large cross-provider trace collected from day-to-day workflows.

2.1 Agentic LLMs and Coding Agents

  • Coding agents are a rapidly adopted application of agentic LLMs, which use tool interaction and multi-step reasoning for complex tasks such as code generation, debugging, and testing.The passage reports that 84% of developers use or plan to use AI tools in their workflow.Source: §2.1 · p. 2
  • A coding-agent request forms a loop of LLM steps and tool calls: tool results join the context for the next step until the agent returns a final answer.Sessions retain accumulated context across requests, and each request inherits context from earlier requests in that session.Source: §2.1 · p. 2

2.2 Existing Datasets and Benchmarks

  • Existing serving traces lack long-horizon tool-calling structure, while coding-agent benchmarks evaluate isolated tasks and may underspecify or restrict tools, limiting workload fidelity and scale.The passage notes that benchmark scale can be too small for modeling large serving fleets, particularly with PD or AF disaggregation.Source: §2.2 · p. 3
  • TraceLab collects a large cross-provider trace from daily research workflows to capture persistent, tool-using coding-agent behavior and inform serving-system design.The passage identifies long sessions, repeated tool calls, accumulated context, and human-paced gaps as workload characteristics that shape serving cost.Source: §2.2 · p. 3

3 Trace Format and Data Collection

Claude Code and Codex session logs are normalized into a unified, step-level trace, then sanitized to protect user privacy. The trace preserves token usage and timing information for workload analysis.

3.1 Raw Agent Logs

  • Both systems persist session logs with the same logical event types despite using different raw schemas.Source: §3.1 · p. 3
  • Each event is timestamped, and each LLM invocation reports token usage.Source: §3.1 · p. 3

3.2 Normalization

  • The logs are normalized into a unified step-level schema, with each row representing one LLM invocation and the tool calls it produces.Source: §3.2 · p. 3
  • Step input is decomposed into prefix tokens and append tokens, while output-token counts include reasoning tokens.Source: §3.2 · p. 3
  • A user-initiated step is classified by its first input event; user messages sent during ongoing work may instead arrive with a tool result, and messages that trigger no model call are not steps.Source: §3.2 · p. 3
  • Table 2 summarizes per-session, per-request, and per-step count distributions; Table 1 summarizes the collected trace, with token counts in B/M and price as estimated API list-price equivalent.Source: Table 2 · p. 4, Table 1 · p. 4
  • Tool latency is reported directly by Codex and derived for Claude from tool-call and return timestamps.Source: §3.2 · p. 4

3.3 Anonymization

  • The released trace replaces session, tool-call, project, and user identifiers with stable pseudonyms and drops raw messages and tool text, retaining character counts, token usage, timestamps, and pseudonymous user identifiers.Source: §3.3 · p. 4

4 Session and Context Management

This section characterizes session loops, context changes, cost, and wall-clock time, finding autonomous multi-step work, prefix-dominated spend, and human-dominated session elapsed time.

4.1 Autonomous Loop

  • Sessions average more than 9 requests (p99 137), and each request takes around 8 steps and 11 tool calls; most steps are tool-initiated.Each step averages slightly more than one tool call, suggesting some parallelization, though it is not pervasive.Source: §4.1 · p. 4

4.2 Context Growth and Compaction

  • Most major reductions are genuine compactions (1,519 of 1,630); context reductions are rare for Claude but more frequent and usually smaller for Codex.Context generally accumulates; compaction is triggered near the context limit.Source: §4.2 · p. 5

4.3 Cost Distribution

  • 59.5% of total cost is prefix tokens, versus 29.2% append and 11.2% output, despite prefix billing at roughly one-tenth the fresh-input rate.The accumulating context replayed on each step makes re-reading context the dominant cost.Source: §4.3 · p. 5, Table 5 · p. 6

4.4 Timing Distribution

  • 92.3% of session wall-clock is human thinking, while within requests tools account for 59.8% of response time versus 41.0% for generation.When idle gaps are capped at one hour, human share falls to 64.3%, with generation at 14.5% and tool execution at 21.2%.Source: Table 7 · p. 6, §4.4 · p. 6

Takeaways

  • More tool calls per step or fusion could amortize LLM-tool scheduling overhead and expose parallelism; cheaper KV-cache storage and sparse attention target prefix-loading costs.These proposals address frequent tool calls and the dominant cost of re-reading context.Source: §4.3 · p. 5, Takeaways · p. 7

5 LLM generation

Coding-agent steps combine very long prefixes with short appends and outputs, while rare long-appends account for most append tokens. Output reuse and observed generation speed also vary across models and context lengths.

5.1 Input length distribution

  • A median step reads 126k prefix tokens and appends 857 for Claude, versus 116k and 886 for Codex; Claude’s prefix p99 reaches 918k tokens.Both providers therefore read roughly two orders of magnitude more prefix than append tokens, while Codex prefixes saturate near 231k.Source: §5.1 · p. 7
  • Most steps cluster at 32k–128k prefix and 256–8k append tokens; below 1k prefix, median appends reach 78k for Claude and 124k for Codex, but above 32k they fall below 1k.The short-prefix group mainly reflects cache misses or initial prefills; longer-prefix steps typically add an incremental tool result or user turn.Source: §5.1 · p. 7, Figure 1 · p. 7
  • Over 90% of steps append fewer than 1k tokens, yet steps appending at least 10k contribute over 70% of all append tokens.The rare long-append steps therefore account for most of the append-token prefill workload.Source: §5.1 · p. 7, Figure 2 · p. 8

5.2 Output length distribution

  • Median outputs are 252 tokens for Claude and 184 for Codex, with p90 below 1.7k; frequent tool calls split full responses into short generations.Codex also has a pronounced ∼40-token output spike associated with write_stdin calls that wait for commands or send Ctrl+C.Source: §5.2 · p. 7, Figure 4 · p. 9

5.3 Output token attribution

  • Claude primarily resends prior outputs in the next step’s append, while Codex behavior varies by version: GPT-5.4 mostly caches outputs and GPT-5.5 mostly resends them.Output-caching places prior output in the next prefix; output-resend includes it in the next append.Source: §5.3 · p. 8, Figure 3 · p. 9

5.4 Timing

  • Median normalized generation speed is 46.8 tokens/s for Claude and 33.9 for Codex; longer contexts correlate with slower observed generation, especially for Codex, but variability remains large.Codex’s binned median falls from about 43 tokens/s around 12k–23k input tokens to about 29 near 185k; Claude’s binned median drops mainly at its longest contexts.Source: §5.4 · pp. 8, 10, Figure 6 · p. 10

Systems Opportunities

  • Append-aware prefill can target throughput for rare long appends and latency for common short appends; output-cache policy trades KV transfers against re-prefilling prior outputs.The section also proposes dedicated TTFT optimization for Codex, where residual TTFT is significant.Source: Systems Opportunities · p. 10

6 Tool Calls

Tool calls are frequent and diverse, but a small number of tools dominate usage and a small number of slow calls dominate tool time. Latency also varies by tool type and includes measurable non-execution overhead for Codex.

6.1 Tool Call Count Distribution

  • Claude exposes 54 tools and Codex 31; command execution is most common for both, and their top three tools account for over 80% and 95% of calls, respectively.File operations such as Read and Edit are the next most common tools.Source: §6.1 · p. 10, Figure 7 · p. 11

6.2 Tool Call Latency Distribution

  • Average tool-call latency is 16.8s, but varies by type: Read and Edit usually take milliseconds to seconds, while Agent and AskUserQuestion can take minutes.Claude Bash latency ranges from milliseconds to minutes despite a median below one second.Source: §6.2 · p. 10
  • Calls longer than 1 minute make up 4.9% of Claude calls but 92% of its tool time; for Codex, calls over 1 minute are 3.1% of calls and 61% of time.Conversely, Claude calls under 1 second account for 70% of calls but less than 1% of tool time; Codex calls under 10 seconds account for 88% of calls and 12% of time.Source: §6.2 · pp. 10–11, Figure 8 · p. 11

6.3 Tool Call Overhead

  • For 253,391 Codex calls covering 87.3% of its tool calls, residual latency was 77.8h (18.6%); its average was 1.11s and p99 was 10.0s.The residual is end-to-end time beyond runner-reported internal execution; Claude lacks comparable internal timing coverage for this decomposition.Source: §6.3 · p. 11, Table 10 · p. 11

System Opportunities

  • Tool type can guide latency prediction, but within-type variation motivates adding operation semantics and recent latency history; lower-overhead tool calling is another proposed opportunity.The paper also suggests more tool calls per step or fusing them to amortize LLM–tool scheduling overhead and expose tool-level parallelism.Source: Takeaways · p. 11, System Opportunities · p. 11

7 Prefix Cache

Prefix caching serves about 96% of prompt tokens, but misses cluster around user-initiated steps after human-paced idle gaps. The trace also shows substantial prefill amplification beyond truly fresh input tokens.

7.1 Per-Step Prefix Cache Hit Rate

  • Both Claude and Codex serve about 96% of prompt tokens from the prefix cache; user-initiated steps are the main misses, while tool-result steps remain near-perfect hits.Human thinking can make the idle interval before a user step long enough to trigger eviction.Source: §7.1 · p. 12, Table 11 · p. 12
  • In an example session, a 10-minute human pause before step 28 coincides with a cache miss and a large fresh prefill.Most closer-spaced steps retain their cached context.Source: §7.1 · p. 12, Figure 10 · p. 13

7.2 Prefix Cache Eviction Time

  • Low-hit-rate steps begin appearing after gaps exceed 5 minutes, and almost all steps miss after 1 hour; long gaps are more common before user-initiated than tool-result steps.Codex tool gaps rarely exceed 5 minutes because long-running tasks move into the background.Source: §7.2 · p. 12, Figure 11 · p. 13

7.3 Gap to Optimal and Redundant Prefill

  • Truly fresh tokens are only 19.0% of append tokens overall, while deployed caching amplifies prefill 5.3× relative to those fresh tokens.Fresh-token shares are 12.3% for Claude and 25.8% for Codex; the amplification is 8.1× and 3.9×, respectively.Source: §7.3 · pp. 12–13

7.4 Storage Trade-off

Longer cache retention improves achievable hit rate but requires more suspended KV storage and reduces prefill amplification. An upper-bound estimate also quantifies append-prefill and cost savings if user-thinking-induced misses were eliminated.

7.4 Storage Trade-off

  • Raising eviction timeout from 1 minute to 1 hour increases achievable hit rate from 85.4% to 98.6%, while storage ratio rises from 0.74 to 5.07.At 5 minutes, hit rate is already about 94% at R ≈1.9; reaching 1 hour costs about 2.7× more storage for roughly 4 additional hit-rate points. R compares suspended-request KV storage with active-decoding KV storage.Source: §7.4 · pp. 13–14, Figure 12 · p. 14
  • Prefill amplification is 18.9× at a 1-minute timeout, 7.4× at 5 minutes, and 1.8× at 1 hour, against a 1× floor for a cache that never evicts.The sweep assumes a complete miss when the preceding idle gap exceeds the timeout and a full hit otherwise.Source: §7.4 · pp. 13–14

7.5 Cost of Human Thinking Time

  • The upper-bound estimate reduces append prefill by 1.07 billion tokens (45.9%) and priced-round cost by $5,189 (12.8%) if user-step prefixes survived human thinking time.This is an upper bound, not an achievable policy guarantee, because it assumes shifted tokens can be served at the cache-read rate.Source: §7.5 · p. 14, Table 13 · p. 14

7.6 Takeaways and System Opportunities

Cache misses cluster around human-paced gaps despite a high overall hit rate, while only a minority of append tokens are fresh. The takeaways identify cache eviction, prefetching, and retention strategies for serving these workloads.

Takeaways

  • User-initiated steps are more likely to miss the prefix cache because their idle times are longer.Misses increase when idle time exceeds 5 minutes and almost never hit after 1 hour.Source: Takeaways · pp. 14–15
  • Only 19% of append tokens are fresh, indicating substantial redundant prefill and a trade-off between cache hit rate and storage space.In a retained-cache upper-bound estimate, misses after human thinking account for 45.9% of append tokens and 12.8% of total priced cost.Source: Takeaways · p. 15

System Opportunities

  • Eviction timing depends on the ratio of cache storage cost to prefill cost, and lower storage costs can reduce prefill workload.The takeaways point to KV compression as one way to reduce storage cost.Source: System Opportunities · p. 15
  • Predicting gap duration and prefetching KV cache could support more aggressive eviction while maintaining similar user TTFT.This is presented as a serving-system opportunity.Source: System Opportunities · p. 15
  • Agent harnesses could periodically refresh cache during long human-thinking gaps or reduce user interventions through better automation.Both strategies are proposed to avoid eviction and reduce prefill cost.Source: System Opportunities · p. 15

8 Related Work

Related work covers real-world coding-agent traces, production serving traces, benchmarks, serving systems, KV management, and long-context methods. TraceLab distinguishes its focus on whole-session coding-agent workloads and their serving implications from these adjacent efforts.

8.1 Real-world coding-agent traces and developer usage

  • TraceLab analyzes a much broader cross-provider trace than CacheWise, with several orders of magnitude more token counts.Other cited work studies code-change outcomes, while TraceLab focuses on system implications of agentic workloads.Source: §8.1 · p. 15

8.2 Production LLM serving traces and conversation datasets

  • TraceLab measures real coding-agent traffic at the systems detail needed for serving decisions, unlike production traces and conversation datasets that lack agent session structure.Those structures include accumulated context, tool-triggered invocations, tool latency, and human-paced gaps.Source: §8.2 · p. 15

8.3 Coding-agent benchmarks and tool-use evaluation

  • Capability benchmarks and frameworks are task-centric, whereas TraceLab captures whole day-to-day sessions, including human delays, repeated tool calls, cache retention, and serving cost.Benchmark instances are usually isolated requests with constrained tooling and limited session history.Source: §8.3 · pp. 15–16

8.4 LLM serving systems and agent-aware scheduling

  • Existing serving systems provide mechanisms for coding-agent traffic, and TraceLab offers a benchmark to guide serving-engine development for real coding agents.The related systems include agent-aware scheduling and prefix-reuse mechanisms.Source: §8.4 · p. 16

8.5 Prefix-cache and KV-state management

  • TraceLab identifies noisy tool latency and human thinking time in coding-agent traces as issues for prefix-cache management.Prior KV-state work considers retention across idle periods and choices among GPU retention, CPU swapping, and recomputation.Source: §8.5 · p. 16

8.6 Long-context attention and KV compression

  • TraceLab finds that long-context pressure also appears in coding agents, where sessions replay large prefixes while appending small tool results or user turns.This connects coding-agent workloads to attention-cost and KV-compression techniques commonly evaluated on long documents or long-context benchmarks.Source: §8.6 · p. 16

9 Limitations and Future Work

The study’s trace comes from the authors’ day-to-day coding-agent use, and its externally visible logs constrain both generalizability and the mechanisms its analysis can identify.

9 Limitations and Future Work

  • The trace comes from the authors’ day-to-day use, so results may not fully generalize to other organizations, workflows, or agent domains.It spans multiple developers, agents, and model versions.Source: §9 · p. 16
  • Because the analysis uses externally visible logs without provider internal telemetry, it identifies workload patterns but only infers mechanisms such as scheduling, cache behavior, and backend routing.The logs include prompts, model responses, tool calls, timestamps, and token metadata.Source: §9 · p. 16
  • The study focuses on coding agents, while future trace studies should extend the method to computer-use and automation domains and compare their workload characteristics.The proposed comparisons include context growth, tool use, latency, and cache behavior.Source: §9 · p. 16

10 Conclusion

TraceLab presents a trace-based analysis of coding-agent workloads and finds long autonomous loops, long contexts with short outputs, long-tailed tool calls, and high but imperfect prefix-cache hit rates. These observations point to serving opportunities and a reference for future systems research.

10 Conclusion

  • Coding-agent workloads feature long autonomous loops, long contexts paired with short outputs, strongly long-tailed tool calls, and high yet imperfect prefix-cache hit rates.These are the study’s headline workload characteristics.Source: §10 · p. 16
  • The findings point to denser or lower-overhead tool calling, append-length-specific prefill optimization, and better KV-cache eviction or prefetching around human-paced gaps.These are identified as serving opportunities for coding-agent workloads.Source: §10 · p. 16
  • The authors hope the trace-based analysis will serve as a reference for future system research on coding agents and broader LLM-based agents.They also call for extending the methodology to larger-scale and more diverse agent workloads.Source: §10 · pp. 16–17
Loading 2606.30560v2…