Source-linked AI summary
How Do AI Agents Spend Your Money? Analyzing and Predicting Token Consumption in Agentic Coding Tasks
Longju Bai, Zhemin Huang, Xingyao Wang, Jiao Sun, Rada Mihalcea, Erik Brynjolfsson, Alex Pentland, Jiaxin Pei
TL;DR
AI agents create a need to understand and predict token costs in complex coding workflows, but existing evidence on these consumption patterns is limited. This paper studies trajectories from eight frontier LLMs on SWE-bench Verified and evaluates pre-execution self-prediction, finding that agentic costs are high, variable, input-token dominated, and difficult for models to estimate precisely.
Problem
The paper addresses the need to understand where AI agents spend tokens and whether they can predict their usage before execution, for more transparent and sustainable pricing.
Method
The authors analyze OpenHands trajectories from eight frontier LLMs solving SWE-bench-Verified tasks and benchmark agents’ ability to predict their own token costs before execution.
Results
Agentic coding tasks are unusually expensive and variable, input tokens dominate cost, and frontier models systematically underestimate usage while achieving only weak-to-moderate prediction correlations.
Takeaways & Limitations
Self-prediction may provide a coarse signal for identifying relatively costly tasks, but precise instance-level estimates and fully predictable pricing remain difficult.
Takeaways & Limitations
The study evaluates only eight frontier models, so its qualitative patterns require validation across a wider range of architectures and agent designs.
Abstract
from arXiv · showhide
The wide adoption of AI agents in complex human workflows is driving rapid growth in LLM token consumption. When agents are deployed on tasks that require a significant amount of tokens, three questions naturally arise: (1) Where do AI agents spend the tokens? (2) Which models are more token-efficient? and (3) Can agents predict their token usage before task execution? In this paper, we present the first systematic study of token consumption patterns in agentic coding tasks. We analyze trajectories from eight frontier LLMs on SWE-bench Verified and evaluate models' ability to predict their own token costs before task execution. We find that: (1) agentic tasks are uniquely expensive, consuming 1000x more tokens than code reasoning and code chat, with input tokens rather than output tokens driving the overall cost; (2) token usage is highly variable and inherently stochastic: runs on the same task can differ by up to 30x in total tokens, and higher token usage does not translate into higher accuracy; instead, accuracy often peaks at intermediate cost and saturates at higher costs; (3) models vary substantially in token efficiency: on the same tasks, Kimi-K2 and Claude-Sonnet-4.5, on average, consume over 1.5 million more tokens than GPT-5; (4) task difficulty rated by human experts only weakly aligns with actual token costs, revealing a fundamental gap between human-perceived complexity and the computational effort agents actually expend; and (5) frontier models fail to accurately predict their own token usage (with weak-to-moderate correlations, up to 0.39) and systematically underestimate real token costs. Our study offers new insights into the economics of AI agents and can inspire future research in this direction.
1 Introduction
This paper presents a systematic study of token consumption in agentic coding tasks and examines whether agents can predict their costs before execution. It identifies token-consumption patterns relevant to agent pricing, model development, and user-aligned cost estimation.
- The study analyzes token consumption in agentic coding tasks using trajectories from frontier language models.It is presented as the first large-scale empirical study of this behavior, with trajectories released to support future research.
- Agentic coding tasks consume orders of magnitude more tokens than chat and reasoning tasks, with input tokens dominating overall cost.This finding motivates closer analysis of where tokens are spent and how agent pricing should account for them.
- Token usage varies substantially across runs, making agent costs difficult to anticipate before execution.The study reports that usage is inherently stochastic even when tasks are similar in complexity.
- The paper benchmarks models’ ability to predict their own token usage before task execution.The prediction task is framed as a capability gap with implications for estimating costs in advance.
- The findings provide insights for future research and practice on agent pricing, model development, and more predictable execution costs.The authors position these insights as steps toward transparent and user-aligned pricing.
2 Data and Method
The study collects end-to-end coding-agent trajectories across real-world repository tasks and extracts token- and behavior-level measurements. It evaluates both overall execution and fine-grained interaction patterns across diverse language models.
- The experiments use OpenHands on SWE-Bench-Verified, with four independent runs per problem across eight diverse language models.The benchmark pairs real-world GitHub issues with corresponding repositories and tests.
- Each agent autonomously solves a task through repeated language-model responses, tool calls, and executions without human intervention.The process begins from an initial task description and proceeds through multiple interaction rounds.
- The analysis extracts token costs, monetary costs, action types, tool usage, and file-access patterns from structured completion histories.Usage information records language-model interactions at each round, enabling token-level and behavioral analysis.
3 Overall Agent Token Consumption Patterns
Agentic coding consumes unusually large and variable token budgets, with input-token accumulation driving cost. Higher spending does not reliably improve accuracy, expensive runs often involve repeated file actions, and human difficulty ratings only weakly predict token use.
- Cost scale and drivers: 3500x more tokens than a typical single-round reasoning task and 1200x more than a multi-round chatting task are consumed by agentic coding, primarily through input-token growth.Repeatedly feeding accumulated context into the model produces a dramatically higher input/output ratio and higher costs even with token caching.
- Variation across tasks and runs: ∼7 million more tokens separate the most expensive problem from the cheapest on average, while repeated runs of the same problem also show large cost variation.The paper characterizes token costs as highly variable across both problems and independent runs.
- Cost and accuracy: Accuracy increases modestly from MinCost to LowerCost, then saturates in UpperCost and MaxCost runs rather than rising monotonically with token cost.At the problem level, higher input-token costs are associated with lower overall accuracy across models.
- Behavior behind high costs: Repeated file viewing and editing sharply increase in more expensive runs, indicating redundant back-and-forth access that inflates context length without proportional progress.The paper presents this behavior as a concrete explanation for the inverse accuracy–cost relationship, while noting that not all high-cost runs are redundant.
- Human difficulty versus agent cost: Kendall τb = 0.32 measures the modest rank-monotonic association between human-rated difficulty and token consumption, with substantial overlap between difficulty groups.6.7% of tasks labeled “<15-minute” required more total tokens than the average “>1-hour” task.
4 Which Models are More Token Efficient
Frontier models differ substantially in how efficiently they trade token usage for accuracy. GPT-5 and GPT-5.2 achieve strong accuracy at low cost, whereas several other models operate at higher cost, with Kimi-K2 the clearest outlier.
- Accuracy–cost trade-off: GPT-5 and GPT5.2 achieve strong accuracy at low cost, while Claude Sonnet 4.5, Claude Sonnet 4, and Qwen3-Coder-480B operate in a higher-cost regime.The comparison covers eight frontier models and examines the accuracy–cost trade-off.
- Accuracy–cost trade-off: Kimi-K2 remains an outlier with both the highest cost and the lowest accuracy among the compared models.This places Kimi-K2 at the unfavorable end of the observed accuracy–cost trade-off.
- Task-controlled efficiency: Models’ relative token-usage ranking persists on shared success and shared failure subsets, indicating that the efficiency gap is not driven by task difficulty.The shared success subset contains 230 tasks solved by all models, and the shared failure subset contains 100 tasks failed by all models.
- Action-level behavior: GPT-5 and GPT-5.2 perform fewer file views, modifications, and repeated file actions than token-expensive models such as Qwen3-Coder-480B.These fine-grained action differences provide a behavioral view of how token efficiency varies across models.
5 Token–Cost Dynamics Across Phases and Rounds
Agent costs accumulate through long, multi-phase trajectories in which input and cached-context tokens dominate overall usage and cost. Round-level spikes arise when actions introduce new content, while repeated file interactions contribute to model-level differences.
- Model interaction patterns: Token-efficient models perform fewer file views, modifications, and repeated file actions than higher-cost models.The comparison focuses on fine-grained file interactions on the shared success subset.
- Phase-level dynamics: The five phases are Setup, Explore, Fix, Validate, and Closeout, with Explore and Fix comprising roughly two-thirds of all rounds.The analysis aggregates token counts, costs, and correlations across 500 problem instances.
- Phase-level dynamics: Input tokens dominate both raw usage and dollar cost across agent trajectory phases.Cache-read input tokens are the largest category in every phase, reflecting cumulative reuse of prior context.
- Round-level dynamics: Cache-read costs form a stable baseline, whereas repository exploration, file creation, test execution, and final summarization produce cost spikes.These actions introduce new content into the context, making individual rounds expensive.
- Phase-level dynamics: Output tokens dominate planning-heavy Setup rounds, while input tokens take over during repository inspection and code reading in Explore.Output tokens reappear during later script-generation and editing activities, while new artifacts raise non-cached input costs.
6 Predicting Agent Token Consumption before Execution
The paper tests whether coding agents can estimate their own token costs before execution using their available tools and environment access. Self-prediction captures broad cost trends but remains noisy, systematically biased downward, and costly enough to limit precise instance-level estimates.
- Prediction setup: Agents are repurposed to inspect the environment and estimate input and output token usage before attempting a task.The setup preserves tool-calling and interaction capabilities while instructing the agent to produce an estimate instead of a fix.
- Prediction accuracy: Self-prediction captures coarse token-usage trends but remains noisy at the instance level, with correlations reaching 0.39.Output-token prediction peaks at 0.39 for Sonnet 4.5, while Kimi K2 reaches 0.38 for input-token prediction.
- Prediction overhead: Prediction overhead is usually below half the task cost, but accuracy and overhead are not monotonically related.Sonnet 4.5 reaches its highest correlation at 0.32× task cost, while GPT-5.2 reduces overhead below 6% with moderate correlations.
- Calibration: Models systematically underestimate both input and output token usage, with the bias especially pronounced for input tokens.Predicted values generally fall below the perfect-calibration diagonal.
- Limitations: Current prediction quality is insufficient for precise instance-level cost estimates and remains an open problem for reliable, efficient integration.Self-prediction may still provide a coarse signal of relative cost and task difficulty.
7 Discussion
The discussion connects token-prediction challenges to transparency and sustainable pricing for agentic systems. It also notes that the evaluation covers only a subset of the broader agentic model landscape.
- Limitations: The study evaluates eight frontier models, which represent a broad sample but only a slice of the agentic model landscape.Wider validation across architectures and agent designs would strengthen the generality of the observed qualitative patterns.
- User transparency: Reliable pre-execution token predictions could improve pricing transparency and help users make informed decisions about task costs.The paper states that current models are not yet accurate enough for exact cost estimates.
- Agent pricing: Agentic tasks complicate sustainable pricing because even simple problems can consume large token budgets through multi-step reasoning and tool use.Accurate cost prediction is presented as important for designing predictable pricing strategies.
8 Conclusion
The paper studies token consumption in agentic coding and tests whether models can predict their own usage before execution. It finds complex token dynamics and persistent difficulty in forecasting consumption, motivating further work on transparent agent pricing.
- Conclusion: Agentic coding produces complex token-usage dynamics, and predicting consumption before execution remains fundamentally challenging for frontier models.The study links these findings to future work on more controllable and transparent agent-pricing schemes.
A Output Token Analyses
Output-token analyses show that higher-cost agent runs become less accurate and increasingly repeat file operations, indicating redundant behavior at higher costs.
- Accuracy decreases as output-token cost increases, while repeated file-view and modify actions rise sharply.The analysis reports this pattern across settings.
- Higher costs reduce accuracy and sharply increase redundant file operations.Mixed-effects analyses examine repeated modify and view actions across cost levels.
B Cost Calculation Details
The cost calculation combines separately priced token categories, using distinct rates for base input, output, cache creation, and cache reads.
- Token cost adds output, cache-creation, and cache-read charges to the base input cost.The formula uses separately priced token categories.
- The rates distinguish base input, output, cache creation, and cache reads.The cache-creation rate corresponds to five-minute writes in the stated setting.
B.2 Implicit Cache (GPT5 and alike)
GPT-5 token accounting uses implicit caching: cached input is reported automatically, while the estimation procedure requires agents to predict input, output, total tokens, confidence, and phase-level breakdowns.
- GPT-5 pricing was $1.250 per 1M input tokens, $0.125 per 1M cached input tokens, and $10.000 per 1M output tokens.These were the official rates at the time of the experiments.
- The agent must estimate predicted input, output, and total tokens, plus confidence and a phase breakdown.The required JSON breakdown covers phases such as repository cloning, reading, debugging, coding, verification, and cleanup.
- Token estimation is the agent’s only goal; it must not implement fixes or modify files.The instructions explicitly distinguish cost estimation from problem solving.
- The required output is a JSON object containing predicted token fields and a breakdown by phase.The schema specifies fields for predicted input, output, and total tokens, confidence, and phase details.
D Self-Prediction Without In-Context Example
Without an in-context example, instruction compliance narrows to Sonnet 4.5 and GPT-5.2, whose token estimates remain biased downward and show weaker correlations with actual usage.
- Most models failed to consistently follow token-estimation instructions without a demonstration.Results are therefore reported for Sonnet 4.5 and GPT-5.2, which remained compliant.
- Underestimation persists without demonstrations and becomes more severe, particularly for input tokens.This behavior is shown in Figure 13.
- Correlation with real token usage degrades substantially without the in-context example.Table 3 reports the corresponding self-prediction performance decline.
- The demonstration improves calibration, while anticipating long-horizon token growth remains difficult.The results indicate that example-induced anchoring does not cause the downward bias.