PaperlayerView exact revision on arXiv

Source-linked AI summary

Agentic Coding in the Wild: Characterizing GitHub Copilot Traces at Production Scale

Banruo Liu, Haoran Qiu, Íñigo Goiri, Rodrigo Fonseca, Ricardo Bianchini, Esha Choukse

arXiv:2608.00101v1cs.AIcs.LG

TL;DR

Coding agents create production workloads whose behavior is not captured by the limited empirical understanding of agent execution and the request-oriented assumptions of existing serving systems. This paper characterizes sampled GitHub Copilot traces at scale, finding distinctive cache and idle-time patterns. Its results inform session-aware resource and cache management, although the client-side traces lack server-side metrics.

  • Problem

    Production-scale behavior of coding agents is not empirically understood, while serving systems designed around independent, short-lived requests are structurally mismatched to these workflows.

    Source: §2.1 · p. 2

  • Method

    The paper characterizes sampled GitHub Copilot coding-agent traces spanning 13M sessions, 3.2M users, 761M LLM calls, and 95T tokens.

    Source: §1 · p. 1

  • Results

    90% average within-turn KV-cache hit rates fall to 55% across turn boundaries, and the idle-time predictor captures 86–90% of total idle time.

    Source: §1 · p. 2, §12 · p. 19

  • Takeaways & Limitations

    The findings inform session-aware scheduling, cache lifecycle management, adaptive compaction, and archetype-aware service-level objectives.

    Source: §12 · p. 19

    The client-side traces lack server-side metrics such as GPU utilization, queue depth, batch size, and memory pressure, limiting end-to-end optimization.

    Source: §11 · p. 19

Abstract

from arXiv · show

AI coding agents like GitHub Copilot, Claude Code, and Codex interleave multi-step LLM inference with tool execution, creating a workload different from chatbots. We present the first production-scale characterization of this workload using sampled GitHub Copilot traces from June 2026, comprising 3.2M users, 13M sessions, 761M LLM calls, and 95T tokens. Our analysis reveals distinctive workload properties with important systems implications. For example, agentic coding sessions consist of sparse user-initiated turns, each unfolding into an autonomous agent loop of LLM calls almost always coupled with tool execution. This structure yields KV cache hit rates averaging 90% within a turn, but falling to 55\% across turn boundaries and drastically invalidated after events like model switches or context compaction. Diverse workflows and user behaviors are observed with variable and long-tailed token consumption, time span, and tool calls. We highlight the difference between quick agentic turnaround times and the minutes-long user idle periods at turn boundaries, and design a lightweight idle-time predictor that captures 86-90\% of total idle time, enabling proactive decisions for efficient resource orchestration. These findings challenge assumptions underlying current LLM-serving systems and provide an empirical foundation for agent-native infrastructure.

1 Introduction

The paper addresses the lack of production-scale evidence about coding-agent behavior with a large GitHub Copilot trace study. Its findings characterize agent execution, cache lifecycle, failures, user diversity, and idle time as workload properties relevant to serving systems.

Motivation

  • Production-scale traces reveal behavior largely invisible in synthetic evaluations, including temporal and structural properties, tool-call failures, context compaction, and developer-population heterogeneity.The characterization is presented as an empirical foundation for evaluating system designs, identifying bottlenecks, and developing optimizations grounded in production deployments.Source: §1 · p. 1

Characterizing Production Workloads

  • The first production-scale characterization covers 13M sessions from over 3.2M users, with 761M LLM calls, 95T tokens, and 775M tool invocations.The traces are sampled from GitHub Copilot’s coding agent.Source: §1 · p. 1

Key findings

  • 87% of LLM invocations are agent-initiated, while prefix caching averages 90% of tokens within a session but falls to 55% hit rates at turn boundaries and 8% after model switches.Tool outputs drive subsequent LLM calls, shifting optimization and scheduling from individual requests to entire sessions.Source: §1 · p. 2
  • Context compaction affects 44% of total tokens in 7.8% of sessions, while tool failures in 9% of turns can amplify compute by up to 4×.Compaction drops over 70% of prompt tokens and resets cache state as severely as a model switch; failures trigger autonomous retry loops with growing context windows.Source: §1 · p. 2
  • Five user archetypes span 23K to 1.1M tokens per turn, indicating that uniform resource policies are suboptimal.The paper also identifies seconds-long idle periods within turns and minutes-long periods across turns, with turn boundaries offering actionable signals for resource reclamation and cache management.Source: §1 · p. 2

2 Background

Coding agents iteratively combine LLM reasoning with tool actions, creating sequential workflows that differ from chat workloads. The paper contrasts these workflows with request-oriented serving infrastructure and discusses systems that begin to account for agent structure.

2.1 AI Coding Agents

  • A user request can trigger dozens of LLM calls and tool invocations, forming a tightly coupled workflow across GPU-backed inference and CPU/IO-intensive tool execution.Agents iteratively select actions, execute tools, observe results, and repeat until the task is complete.Source: §2.1 · p. 2
  • Coding agents have stronger sequential dependencies than other agentic applications because later actions often depend on exact tool outputs, limiting parallel execution and making end-to-end latency sensitive to individual steps.The paper says request-level scheduling, LRU cache eviction, and independent request batching are structurally mismatched to coding-agent workloads.Source: §2.1 · pp. 2–3

2.2 Serving Infrastructure

  • Existing agent-serving systems use workflow-aware scheduling, cache retention, or co-design, but are largely evaluated with synthetic benchmarks or small-scale workloads.The paper notes that this leaves uncertain whether their assumptions and optimizations reflect production coding-agent behavior.Source: §2.2 · p. 3

3 Dataset and Methodology

The study analyzes anonymized structural telemetry from GitHub Copilot’s coding agent and organizes execution as sessions, user turns, and individual LLM or tool steps.

3.1 At-Scale Production Data Source

  • The sampled telemetry covers 13.5M sessions from 3.2M users during the first week of June 2026, across more than 27 models and 45 tools.It includes Visual Studio and VS Code traces from multiple US regions, representing at most three timezones.Source: §3.1 · p. 3
  • The traces record structural metadata such as timestamps, durations, token counts, model and tool names, and success or failure status, but omit prompt text, outputs, source code, and tool arguments.Anonymized identifiers allow related events to be linked and workflow behavior reconstructed without exposing user identities or content.Source: §3.1 · p. 3

3.2 Session Hierarchy

  • A session contains user turns, each comprising a user-prompt-triggered turn and its autonomous response chain; each step is one LLM invocation or tool call.Every turn begins with exactly one user-initiated LLM call, and subsequent calls or tools are agent-initiated; the final step is always an LLM invocation.Source: §3.2 · p. 3

4 Agent Execution Pattern

This section characterizes production coding-agent execution, finding strong time-of-day and session-level variation alongside tightly coupled, mostly autonomous LLM–tool workflows. Diverse turn archetypes and retry loops further produce substantial variation in resource demand.

4.1 Time Series and Diurnal Patterns

  • Daytime activity reaches approximately 4–5× baseline, while medium-length prompts drive most diurnal volume swings and weekend sessions use more resources on average.Extremely long and long prompts remain comparatively stable across peaks and troughs; weekend sessions also have higher average prompt-token usage than weekday sessions.Source: §4.1 · p. 4, Figure 1 · p. 4, Figure 2 · p. 4

4.2 Session-Level Overview

  • Session and turn metrics are right-skewed: median sessions have 15 LLM calls and last 4.2 minutes, while median turns have 3 LLM calls and 3 tool invocations.Mean session duration is 14.9× the median, and P90 turns require nearly 16 LLM calls, 21 tool invocations, and over 1M prompt tokens.Source: §4.2 · pp. 4–5, Figure 4 · p. 5
  • Weekend sessions are fewer but longer, with more LLM and tool calls per turn than weekdays; this differs from chatbot observations of longer weekday sessions.The authors suggest developers may undertake more ambitious tasks when uninterrupted.Source: §4.2 · p. 5, Figure 5 · p. 5

4.3 Execution Structure

  • LLM calls and tool invocations track at roughly 1:1 across sessions and turns, reflecting an observe–decide–act loop in which reasoning-only and action-only steps are rare.The per-turn call-count CDFs closely track across the distribution, except for LLM-only turns.Source: §4.3 · p. 5, Figure 6 · p. 5
  • 87% of LLM calls are agent-initiated; after a user message, the agent averages 6.6 calls before returning control.A small fraction of requests trigger long chains that account for a disproportionate share of calls and serving load.Source: §4.3 · pp. 5–6
  • LLM execution is mostly serial: median concurrency is 1.15 and P90 is 1.4, with overlap concentrated in the middle of turns.Parallel branches can create straggler dependencies and same-session KV-cache contention.Source: §4.3 · p. 6, Figure 7 · p. 6

4.4 Workflow Archetypes

  • Six recurring workflow archetypes emerge: Deep-loop read is largest (30.5%), averaging 9 LLM calls and 7 tool batches; LLM-only turns comprise 20.2%, while Multi-cycle edit comprises 19.0%.Source: §4.4 · p. 6
  • Failure-driven retry turns (9.1%) can reach 36 LLM calls—four times the median—with 34 tool batches, amplifying compute as errors trigger autonomous recovery.Across workflows, calls, tool use, and token consumption can vary by an order of magnitude.Source: §4.4 · pp. 6–7, Figure 8 · p. 7

5 LLM Footprints

Agentic coding calls are input-heavy, and KV-cache effectiveness varies across turns and model switches. Within-turn reuse is strong, but idle gaps and model changes sharply reduce cache hits.

5.1 LLM Call Basics

  • A median 68K prompt tokens versus 247 output tokens yields an input-to-output ratio above 275:1, with cached prompt tokens at a median 63K.Conversation history supplies 48% of input tokens and function-call messages 28%; together they exceed 75%.Source: §5.1 · p. 8, Figure 10 · p. 8
  • 85.4% of wall-clock time in sessions is LLM execution, versus 4.7% in tool execution; multi-turn sessions spend a median 80.1% of wall-clock time idle between turns.For sessions with both LLM and tool calls, LLM inference takes a median 87.7% of runtime.Source: §5.1 · pp. 8–9

5.2 KV Cache

  • 98% median cache hit rate masks a bimodal distribution: roughly 10% of calls have below 20% reuse, while over 80% exceed 90%.Within a turn, cache hit rate rises from about 45% on call one to 86% on call two, then plateaus at 92–94%.Source: §5.2 · p. 9, Figure 13 · p. 9

5.3 Cache Degradation at Turn Boundaries

  • A same-model turn boundary causes an average 26% cache-hit-rate drop, and hit rates fall sharply once inter-turn idle gaps exceed roughly 2 minutes.Beyond 10 minutes, median hit rates are near 0–5%.Source: §5.3 · p. 9, Figure 14 · p. 10, Figure 15 · p. 10

5.4 Model Switches

  • Model switches affect about 6.4% of sessions and reduce average cache hit rate to 8%, a 67% drop versus 26% at same-model turn boundaries.Switches are predominantly reactive, associated with errors or rate limiting; manual-to-auto switches are 52% downgrades, while auto-to-manual switches are 51% upgrades.Source: §5.4 · p. 10, Figure 16 · p. 10

6 Context Compaction

Context compaction rewrites accumulated prompts when sessions approach context limits, adding latency and disrupting cache reuse. It is uncommon overall but concentrated in resource-intensive sessions.

6 Context Compaction

  • 7.8% of sessions undergo compaction, but those sessions account for 44.2% of total tokens, 37.1% of LLM calls, and 38.9% of tool calls.Compaction occurs in 0.5% of calls and in 22.6% of sessions with prompts over 100K tokens; most affected sessions compact once.Source: §6 · pp. 10–11
  • Compaction triggers at heterogeneous context utilizations, with peaks near 50%, 65–66%, 80%, and 100%, including multiple trigger points within a single model.The overall median trigger point is 65–66%.Source: §6 · p. 11, Figure 17 · p. 11
  • Compaction takes a median 22% of total turn execution time, with P90 at approximately 34%, because it requires a long-prefill and decode LLM call on the critical path.Incremental compaction or overlapping strategies are suggested as possible optimizations.Source: §6 · p. 11, Figure 18 · p. 11
  • Compaction drops a median 72.8% of prompt tokens and causes a median 66.1% cache-hit-rate drop; 34.3% of events erase at least 90% of cache hit rate.The rewritten prompt shares little prefix with the prior prompt, often effectively cold-starting the cache.Source: §6 · p. 11, Figure 19 · p. 11, Figure 20 · p. 11
  • Compaction is a hidden cost of long sessions: after deep exploration fills context, the agent loses cached state and must rebuild it, increasing latency and cost.The paper suggests incremental, prefix-preserving compaction to maintain partial cache continuity.Source: §6 · p. 12

7 Tools

Tool usage is concentrated in a small set of tools, but execution times and failure behavior vary widely. Most tool batches are serial; short batches overlap with LLM execution, while long-running tools remain exposed on the critical path.

7.1 Tool Usage Statistics

  • The 40+ tools are heavily concentrated: get_file accounts for 35.0% of invocations, and the top 11 tools account for over 90%.Median latency varies roughly 320× across tools; read-oriented tools are consistently fast, while execution tools dominate the latency tail.Source: §7.1 · p. 12, Figure 23 · p. 12
  • Tool execution has a median duration of 166 ms but a mean of 16.7 s, with P90 4.4 s and P99 79 s, reflecting a long tail dominated by commands and builds.Execution-oriented tools can have mean latencies of 68 s for run_command_in_terminal and 78 s for run_build.Source: §7.1 · pp. 12–13, Figure 22 · p. 12
  • Read and search tools succeed nearly universally, while run_command, run_build, and edit_file fall to roughly 73% success; failed calls generally take longer and vary more.A failed run_cmd_in_terminal invocation takes 48× longer at P95 than a successful one.Source: §7.1 · p. 13, Figure 24 · p. 13
  • Failed run_build calls return 7–8× more tokens at the median than successful builds, whose outputs have a median of roughly 60 tokens.By contrast, successful run_tests calls return richer token deltas than failures.Source: §7.1 · p. 13, Figure 25 · p. 13

7.2 Runtime Parallelism

  • 93% of tool batches contain a single invocation; parallel batches have a median width of two, though the tail reaches 108 concurrent tools.Read-only tools are frequently batched, while write and execution tools are rarely parallelized because of shared state or side effects.Source: §7.2 · pp. 13–14, Figure 26 · p. 14, Figure 27 · p. 14
  • More than 99% of tool time for batches under 500 ms overlaps an active LLM call, compared with 5% for batches lasting 30 seconds or more.For 0.5–5 second batches, 76% overlaps; for 5–30 second batches, 65% occurs independently of LLM execution.Source: §7.2 · p. 14, Figure 28 · p. 14
  • 97% of batches overlap an LLM window, yet only 7.7% of aggregate tool wall-clock time is hidden; long-running tools account for the remaining critical-path time.Builds, test executions, and external commands contribute to the long tail.Source: §7.2 · p. 14

8 Users

Users fall into distinct workflow archetypes with sharply different activity and resource footprints. Usage is concentrated among intensive users, motivating archetype-aware serving policies rather than uniform resource allocation.

8 Users

  • Linking anonymized user identifiers enables analysis of how user behavior maps to resource footprints and serving-system SLOs.Users are grouped into five archetypes using per-turn tool/LLM-call mix and token consumption.Source: §8 · p. 15, §8.1 · p. 15

8.1 User Archetypes

  • Deep-loop users consume 1.1M tokens per turn, compared with 203K for Readers and 417K for Coders, despite having only 2 median sessions and 6 turns per user.Deep-loop users also average 20 tools per turn; Chat-only users consume 23K tokens per turn and never invoke tools.Source: §8.1 · p. 15

8.2 Resource Concentration

  • Per-user usage is strongly right-skewed: users have a median of 2 sessions and 11 turns, while P90 values are 8 sessions and 74 turns.Median prompt-token consumption is 3.2M per user versus 33K completion tokens, and a minority of users account for a disproportionate share of token volume.Source: §8.2 · pp. 15–16

8.3 Implications for Serving Infrastructure

  • A uniform cache timeout creates a 50× per-miss cost gap between Deep-loop and Chat-only users, supporting higher retention for intensive sessions and more aggressive eviction for Readers and Chat-only users.Coders and Terminal users also accumulate container state, making checkpointing preferable to termination.Source: §8.3 · p. 16

9 Idleness and Resource Reclamation

Idle time differs sharply within and across turns: autonomous execution produces brief gaps, while user pauses create long idle periods. Turn boundaries offer reclamation opportunities, and a predictor estimates idle duration to guide resource management.

9.1 Idle Time Distributions

  • Cross-turn idle intervals are only 8–9% of intervals but have median durations of 243 seconds for containers and 172 seconds for KV caches, versus 5.8 and 1.2 seconds intra-turn.User idle time has a median of 1,512 seconds (25.2 minutes), with a tail extending beyond one day.Source: Table 8 · p. 16, §9.1 · p. 17

9.2 Turn Boundaries as Reclamation Signals

  • The predictor models a survival curve for idle duration, allowing resource managers to choose an operating threshold and update the conditional estimate as idle time elapses.Evaluation uses 150K sessions from one week for training and 50K sessions from the following week.Source: §9.2 · p. 17
  • Captured idle time remains 86–90% across evaluation windows, although accuracy and F1 decline from roughly 80–88% at 30 seconds to 25–43% at 30 minutes.The predictor estimates idle duration at turn boundaries using turn-, session-, and temporal-level features; average accumulated idle duration and turn index together account for over half of feature importance.Source: §9.2 · p. 17

9.3 Multiplexing Opportunity based on Predictions

  • Within a turn, retain KV cache and keep tool containers warm; across turns, offload cache or sleep containers when the longer idle gap can amortize migration and restoration costs.A turn boundary identifies a possible reclamation point, while predicted idle duration helps estimate how long the session will remain inactive.Source: §9.2 · p. 17, §9.1 · p. 17, §9.3 · p. 18

10 Related Work

Prior coding-agent studies use benchmarks, controlled settings, or smaller trace collections, while this work extends production workload characterization to GitHub Copilot at much larger scale.

Workload Characterization Studies

  • This study extends production workload-characterization research, previously applied to Azure Functions, deep-learning clusters, and GPU power workloads, to coding assistants.The cited studies include Shahrad et al. [33], Philly [15], Helios [13], and POLCA.Source: §10 · p. 18

Coding Agent Benchmarks

  • SWE-bench evaluates agents on 2,294 fixed GitHub issues rather than characterizing production workloads, and OpenHands and SWE-agent are studied in controlled settings.Majgaonkar et al. [24] analyze SWE-bench trajectories and report longer, more variable failed trajectories.Source: §10 · p. 18

Coding Agent Characterization

  • Production-scale GitHub Copilot traces spanning 13M sessions and 3.2M users complement prior coding-agent characterization efforts.TraceLab released 4.3K Claude Code and Codex traces from 43 developers; it reported autonomous loops, prefix-token cost, idle gaps, and heavy-tailed tool latency.Source: §10 · p. 18

11 Limitations and Future Work

The study’s conclusions are bounded by missing quality signals, client-side-only traces, and a rapidly evolving workload that may change over time.

11 Limitations and Future Work

  • Without prompt and response content, the traces cannot correlate infrastructure efficiency with task completion quality.The authors identify joint analysis of resource consumption and task success rate as a path toward quality-aware scheduling.Source: §11 · p. 19
  • Client-side traces omit server-side metrics such as GPU utilization, queue depth, batch size, and memory pressure, limiting end-to-end optimization.The authors note that collecting these details could erode model-level and serving-engine confidentiality.Source: §11 · p. 19
  • The workload evolves as tools, models, and autonomy strategies change; longitudinal tracking is needed to understand possible drift in characteristics such as LLM-call parallelism.The reported results remained consistent from January to June 2026.Source: §11 · p. 19

12 Conclusion

The paper characterizes GitHub Copilot coding-agent workloads and proposes a lightweight predictor that captures 86–90% of idle time for turn-boundary resource reclamation.

12 Conclusion

  • 86–90% of idle time is captured by a lightweight predictor using session-level features for turn-boundary resource reclamation.The paper also discusses session-aware scheduling, cache lifecycle management, adaptive compaction, and archetype-aware SLOs as systems design implications.Source: §12 · p. 19
  • The characterization identifies observations about LLM–tool execution loops, agentic workflow types, KV-cache lifecycle, and resource idleness in GitHub Copilot workloads.The traces are from GitHub Copilot’s coding-agent workloads.Source: §12 · p. 19
  • The authors plan to release sanitized traces from the characterization data to the public.Source: §12 · p. 19

Related papers

Loading 2608.00101v1…