Source-linked AI summary
Can your AI agent be cheaper? Investigating the effects of task specifications on token spend in agentic coding tasks
Jakub Smékal
TL;DR
The paper investigates how task-description detail shapes agentic coding token spend and whether cost can be predicted for unseen tasks. It evaluates structured specification variants and thinking efforts across repeated Kimi K3 runs, finding that prompt changes shift mean cost without changing variance and that one cheap probe predicts the broader cost distribution.
Problem
How task specifications affect agentic token spend, and whether spend is predictable on unseen tasks, remain largely unmeasured despite token usage being important for deploying long-horizon coding agents.
Method
The study varies ten structured specification variants and two anchor prompts across five SWE-bench Verified tasks, three thinking efforts, and repeated Kimi K3 runs, then fits a predictor using held-out tasks and a cheap probe.
Results
29.7%: reducing a full specification to a bare user story raises token spend; a single $0.11 probe predicts spend across configurations to typically 36%, versus 161% without measurement.
Takeaways & Limitations
Measuring a prompt distribution can help practitioners evaluate and predict task-specific token cost, including which usage patterns are expensive and how much expense may be recoverable through prompt choice.
Takeaways & Limitations
The study uses a sparse set of coding tasks and a single model, limiting generalization to broader models and real-world tasks; its shared-cost method omits task-specific prompt sensitivity.
Abstract
from arXiv · showhide
Agentic coding workflows are now widely deployed in real-world systems. With long-horizon reasoning and tool use, token usage has become an important consideration for both cost and efficiency. Two engineers using AI will solve the same problem differently. How the specification of a task shapes an agent's token spend, and whether that spend can be predicted in advance, are open questions. Here, we study the effects of different task specifications on agentic token spend with the Kimi K3 model at three thinking efforts. Across $2,700$ runs, we show that reducing a full task specification to a bare user story raises token spend by $29.7\%$, while run-to-run variance remains unaffected by any prompt changes. We show that prompt-sensitivity is task-dependent, running from $13\%$ to $115\%$. We fit a simple predictor that can price a full distribution of task specifications and thinking effort configurations from a single cheap probe on an unseen task within $36\%$, improving over prior work in predicting token spend. Our work provides initial results quantifying the effects of task specification on agentic token spend and introduces a method that can be used to systematically evaluate the cost of AI coding workflows.
1 Introduction
The paper asks how task specifications and thinking effort affect agentic token spend, and whether spend is predictable on unseen tasks. Across a controlled specification distribution, prompt changes shift mean cost but not run-to-run variance, while a cheap probe predicts broader cost configurations.
- 29.7%: reducing a full task specification to a plain prose problem description raises token spend and turns to success by 16.4%.The direction is consistent on every measured task.
- ×1.34: rerunning an identical specification produces this spread in token spend, and no tested specification widens or narrows it.
- 36%: a single $0.11 probe run on an unseen task typically predicts spend across other specifications and thinking efforts, versus 161% without measurement.
2 Related work
Prior work varied models or prompt surface form but largely left the effect of task-description detail on token cost unmeasured. This paper treats task detail as a deliberate variable alongside thinking effort and measures its cost consequences.
- Prior spend studies varied models while holding each task’s problem statement fixed, whereas this paper varies the prompt with the model fixed.
- Earlier prompt studies changed surface form while holding meaning constant and did not repeatedly measure token cost as an outcome.
- Task descriptions contain valued bug-report elements that are difficult to supply, while fix suggestions, reproduction scripts, and localization correlate with higher resolution odds.
- Reasoning-budget research generally takes the query as given, although how much thinking effort buys may depend on task description.
- No prior work known to the authors varies task-description detail while measuring token cost and run-to-run variance.
3 Methods
The study constructs structured prompt variants for five SWE-bench Verified tasks, evaluates them across thinking efforts and repeated runs, and models both effects and prediction on held-out tasks.
- Tasks and prompts: Five SWE-bench Verified tasks are represented by ten specification variations spanning different levels of task-relevant information.The design aims to attribute cost changes to removing specific task sections.
- Tasks and prompts: Ten variations include a full specification, seven single-section removals, and two partial specifications, built from eight structured sections.The sections include user story, acceptance scenarios, requirements, success criteria, and other template components.
- Tasks and prompts: The oracle anchor supplies the fix and is excluded from analyses, while the raw anchor uses failing-test output; analyses therefore use eleven specifications.The oracle costs 0.05 to 0.19 times the full specification and is retained only as a sanity check.
- Experimental setup: 2,700 runs: each task-specification pairing is tested at low, high, and max thinking effort with fifteen repeats using Kimi K3.Runs use mini-swe-agent in the standard SWE-bench Docker image without network access.
- Analysis: The analysis fits Bayesian hierarchical models that estimate typical effects across tasks and task disagreement with posterior medians and 90% credible intervals.
- Prediction model: For held-out tasks, the predictor learns relative configuration costs from four training tasks and calibrates the new task’s overall level using probe-run log costs.The probe is configured as a full specification at low thinking effort.
4 Results
Across specification variants and thinking efforts, prompt content shifts mean cost and turns but not run-to-run variance, with effects depending on task and effort. A single inexpensive probe also predicts an unseen task’s cost across configurations substantially better than no measurement.
- Prompt content and token spend: 29.7%: reducing a full specification to a bare user story raises token spend, while turns to success rise 16.4%.The increases occur in the same direction on all five measured tasks.
- Prompt content and token spend: Prompt sensitivity is task-dependent, with the same bare-user-story reduction raising cost by 115% on xarray-7393 versus 29.7% pooled.The xarray-7393 result is identified as an outlier, showing that task-specific measurement matters.
- Prompt content and token spend: Acceptance scenarios matter more than success criteria: removing the concrete Given/When/Then cases costs turns, whereas removing the abstract prose assertions has no measurable outcome effect.The concrete form costs turns on all five tasks, while the abstract form has no measurable effect on any outcome.
- Thinking effort: At low effort, the most-to-least expensive specification ratio is ×2.13; it narrows to ×1.67 at high effort and ×1.61 at max effort.The acceptance-scenario removal adds 20.1% turns at low effort but 2.1% at max effort.
- Run-to-run spread: Prompt variation does not measurably change variance: repeated runs have median geometric standard deviation ×1.34, while absolute spread scales with mean cost.Across specifications, the median geometric standard deviation ranges from ×1.29 to ×1.40; the cost-spread relationship has log-log slope 1.08 and r = 0.95.
- Cost structure: At a 96.3% cache hit rate, output tokens are 2.7% of processed tokens but 51.1% of dollars, making turn reduction the larger cost target.The passage cautions that these percentages depend on the price schedule and cache hit rate.
- Predicting the cost of an unseen task: 36%: one 0.11 USD probe predicts an unseen task’s cost across other specifications and efforts, versus 161% median error without measurement.Ten probes reduce median error to 25%, and one probe brings two thirds of settings within ±50%.
- Predicting the cost of an unseen task: One probe raises the within-thinking-level cost correlation from r = 0.08 without measurement to r = 0.72.The comparison uses the metric reported in prior work and requires measuring the unseen task rather than self-estimating before execution.
5 Discussion
Task specifications shift mean token spend, but prompt sensitivity varies by task while run-to-run stochasticity remains largely unchanged. A single cheap probe can calibrate cost across specifications and efforts, though generalization is limited by the sparse task set, single model, and omitted task-specific sensitivity.
- 13% to 115%: the same specification cut changes mean token spend by different amounts across tasks.
- Prompt leverage falls as thinking effort rises, from ×2.13 to ×1.61, while removing acceptance scenarios falls from 20.1% to 2.1% additional turns.This narrows the potential benefit of packaged instructions in high- and max-effort configurations.
- A single 0.11 USD probe reduces median prediction error for a new task from 161% to 36% across specifications and thinking efforts.Without running the new task, inferring its cost shape from four other tasks is a median of 161% off.
- At a 96.3% cache hit rate, output tokens are 2.7% of processed tokens but 51.1% of spending, making turn reduction more consequential than input shortening.These percentages depend on the price schedule and cache hit rate, so they do not directly transfer to other models or inference stacks.
- The study’s generalizability is constrained by its sparse task set, single model, and cost predictor’s omission of task-specific prompt sensitivity.
6 Conclusion
Because token usage is a primary concern in agentic coding, the paper evaluates how task specifications affect spend and whether a cheap probe can predict new-task cost. It finds that careful specification can substantially improve token spend for some tasks and motivates systematic evaluation of usage patterns.
- Token usage is a primary consideration for practitioners adopting agentic coding workflows.
- The study evaluates token spend across task specifications and shows that a cheap probe can predict the cost of a new task.
- Some tasks show substantial token-spend improvement through careful task specification.
- The method motivates systematic evaluation of how usage patterns affect token spend in agentic coding workflows.
A Experimental parameters and task selection
The experiments use Kimi K3 with controlled endpoint and scaffold settings, bounded runs, and pooled task-level reporting. The appendix table summarizes five tasks across specifications and thinking efforts using solve rate, cost, and turns.
- All runs use Kimi K3 through a Modal endpoint with temperature 1.0 and low, high, or max reasoning effort.The mini-swe-agent scaffold runs tasks in the standard SWE-bench Verified Docker image.
- Runs are bounded at 120 agent turns and 4.00 USD; 108 of 112 previously capped runs completed after re-execution with the longer turn limit.
- Table A1 pools solve rate, cost, and turns over five tasks, eleven analyzed specifications, and three thinking efforts.Costs are reported in USD at list prices.
B The prompt set
The study constructs structured task specifications from eight sections, alongside raw and oracle anchors, and emits controlled variants by removing whole sections. The prompt set is evaluated on a concrete xarray task whose failing tests expose dtype-preservation errors.
- Specification design: The specification template contains eight sections, including user story, acceptance scenarios, functional requirements, and success criteria.The full list also includes header, edge cases, key entities, and assumptions.
- Task content: The xarray task tests whether stacking preserves coordinate dtypes through MultiIndex creation and unstacking.Its requirements prohibit changing values and define success as retaining dtype i4 after stacking without regressions.
- Task content: The raw test anchor contains 483 tokens and shows two failed dtype-restoration tests alongside 71 passing tests.The failures report float64 where float32 was expected for int32- and float32-parameterized cases.
- Design qualification: Specification and anchor lengths are not controlled: raw transcripts can exceed full specifications, while oracle prompts are usually shorter except on django-15503.Raw transcripts include session headers, tracebacks, and repeated assertions; the oracle is long when the fix itself is substantial.
C Statistical model
The analysis estimates section-removal effects with a hierarchical model over task-level differences and sampling variance. Weakly informative priors, proportion adjustments, and 90% intervals govern the reported evidence and its limitations.
- Model specification: For each section and outcome, the model estimates task-level effects as differences in mean log cost or log turns from the full specification.Effects are averaged over the three thinking efforts and paired with task-level sampling variance.
- Model specification: The model uses θ ∼Normal(0, 0.52) and σ ∼HalfNormal(0.3), with task-level effects integrated out on a 1201 × 130 grid.The priors are weakly informative and stated in log units, so their scales are multiplicative.
- Model specification: For solve rate, prior scales are halved and per-task proportions receive the Agresti-Coull adjustment to avoid zero variance at 100%.This adapts the model to proportions rather than log quantities.
- Inference: The paper reports 90% credible intervals and posterior probabilities that effects exceed zero because conclusions are sensitive to the interval threshold.At 95%, some described effects include zero, while the reported probabilities are 0.97 or 0.98 for the highlighted effects.
- Inference: Specification contrasts are supported, but stable rankings are not: split-half repeats yield mean Spearman ρ = 0.41, projected to ρ = 0.58 at fifteen repeats.The authors estimate that ranking would require an order of magnitude more repeats.
D Per-section and per-task results
Removing specification sections changes cost unevenly across tasks, with the largest effects concentrated in reducing prompts to a bare user story. Token-based estimates closely track cost estimates across solved runs.
- Per-task effects: Reducing a full specification to a bare user story raises cost by 13% to 115% across the five tasks.The task-specific increases are 13%, 16%, 21%, 24%, and 115%, with xarray-7393 highest.
- Per-task effects: The acceptance-scenario effect is positive on four of five tasks, ranging from −8% on scikit-learn-14053 to +53% on xarray-7393.The direction is therefore not uniform across tasks for this section removal.
- Per-task effects: Five other section removals change cost by no more than 18% in absolute value on any task, with signs differing between tasks.The table attributes this pattern to small between-task disagreement for those effects.
E Token-denominated and supplementary figures
Supplementary analyses show that token counts closely reproduce cost patterns, while thinking effort modestly changes spend composition. Solve rates are near ceiling for most tasks but reveal substantial headroom for selected cases.
- Token measures: Cost and tokens are near-equivalent under one price schedule, although their mapping is provider-specific.Figure E1 reproduces the main cost figure in token units.
- Supplementary figures: Figure E1 reports median tokens per run for xarray-7393 across eleven specifications at each thinking effort.Figure E2 covers cost and turns for the four tasks omitted from the main supplementary comparison.
- Token measures: Higher thinking effort buys more output tokens at the most expensive rate, but this composition shift is small relative to the price-schedule effect.The supplementary token and spend table breaks composition out by thinking effort.
- Solve rates: Four of five tasks sit at or near the solve-rate ceiling across specifications and thinking efforts.django-15503 retains headroom under every specification.
- Solve rates: On astropy-14365, raw issue text solves 47% of runs at low effort versus 99% at high and max effort.Across the ten structured specifications, pooled solve rate reaches 99.6% for that task.