PaperlayerView exact revision on arXiv

Source-linked AI summary

TokenEconomizer: Experience-Driven Black-Box Cost Control for Coding Agents

Aaron J. Li, Weijia Xu, Chandan Singh, Zixiao Chen, Jeevana Priya Inala, Charlotte Siska, Acar Tamersoy, Bin Yu, Ion Stoica, Michel Galley, Jianfeng Gao

arXiv:2610.22732v1cs.CL

TL;DR

Coding agents incur substantial token costs over long task trajectories, while token efficiency is commonly evaluated one task at a time. TokenEconomizer learns environment-specific cost-control policies from completed executions; on SWE-Bench it reduces mean per-task cost by 18.4–30.9% while maintaining comparable task quality.

  • Problem

    Token efficiency is typically optimized on isolated tasks, although software development proceeds across sequences of tasks in evolving repositories and agent trajectories repeatedly bill accumulated context.

    Source: §1 · p. 1

  • Method

    TokenEconomizer is an external controller that learns environment-specific policies for micro-skill activation, tool pruning, and failure-loop intervention from completed execution traces.

    Source: §1 · p. 2, §2.2 · p. 4

  • Results

    18.4–30.9% lower mean per-task cost on SWE-Bench, with Pass@1 improving in four of five agent–model settings and savings across every repository under each tested setting.

    Source: §1 · p. 2, §4.2 · p. 7

  • Takeaways & Limitations

    An external controller can learn from completed work to make an unchanged agent cheaper while largely preserving task success across software engineering, web development, and security auditing.

    Source: §6 · p. 10

    Evaluation covers three coding agents, not more minimal harnesses such as Pi; supporting such a harness requires an adapter.

    Source: §6 · p. 10

Abstract

from arXiv · show

Advances in frontier coding agents have sharply increased the token costs of AI-assisted software engineering. Their token efficiency is typically evaluated on isolated tasks, whereas real-world developers use them to continuously build, update, and maintain large-scale projects. This mismatch motivates us to evaluate agents over sequences of realistic coding tasks, where we find that agents exhibit recurring environment-dependent token costs that existing methods struggle to reduce. To minimize these costs, we introduce TokenEconomizer, an external controller that cuts environment-specific token costs over a sequence of tasks. These cuts come from determining (i) which operational micro-skills to inject, (ii) which persistently irrelevant tools to prune, and (iii) when to intervene in unproductive failure loops. Across Claude Code, Codex, and GitHub Copilot CLI paired with frontier models, TokenEconomizer reduces the average token cost by 18.4-30.9% with comparable task success. Savings hold in every repository under every tested agent-model setting and generalize to diverse domains such as sequential web development and security auditing, where costs fall by 6.4-21.2% depending on task structure. Our work establishes adaptive black-box cost control as a promising direction for reducing token costs of coding agents in realistic settings.

1 Introduction

Coding agents incur repeated context costs across long trajectories, while token efficiency is commonly studied on isolated tasks. The introduction motivates environment-adaptive control after fixed strategies show mixed repository effects, then presents TokenEconomizer and its cross-agent evaluation.

1 Introduction

  • Billed context includes persistent instructions and an expanding interaction history, so earlier content is charged repeatedly across agent turns.Caching discounts replayed context but does not eliminate its cost.Source: §1 · p. 1, Figure 1 · p. 2
  • Task-local optimization overlooks software-development continuity, where completed executions in a shared repository can provide reusable experience for later tasks.The paper therefore targets cumulative cost across task sequences rather than isolated executions.Source: §1 · pp. 1–2
  • Fixed strategies have small pooled effects and mixed repository-level cost changes, motivating adaptation rather than uniform configuration.RTK, Serena, and RepoMap each help some repositories and raise costs in others.Source: §1 · p. 2, Figure 2 · p. 3
  • TokenEconomizer reduces mean per-task cost on SWE-Bench by 18.4–30.9%, with savings in every repository under each tested agent–model setting.Pass@1 improves in four of five settings; on Web-Bench and security benchmarks, costs fall by 16.0–21.2% and 6.4–19.6%, respectively, with comparable task quality.Source: §1 · p. 2
  • TokenEconomizer is an external black-box controller that learns environment-specific policies for activating micro-skills, pruning irrelevant tools, and intervening in failure loops.It acts through prompt and tool settings and execution events without modifying model weights or agent-loop code.Source: §1 · pp. 2–3

2 Cost Profiling and Analysis of Fixed Cost-control Strategies

Profiling shows that fixed strategies vary in effect across repositories, while execution traces reveal recurring operational actions, underused tools, and costly failures. These patterns motivate learning repository-specific policies from completed tasks.

2 Cost Profiling and Analysis of Fixed Cost-control Strategies

  • The profiling analyzes 60 verified SWE-Bench tasks, comparing default execution with three fixed strategies under the same configuration in every repository.The three strategies are RTK command-output compression, Serena semantic navigation, and Aider RepoMap structural context injection; none learns from completed executions or activates selectively from prior experience.Source: §2 · p. 3, §2.1 · p. 3

2.1 Repository-specific Effects of Fixed Strategies

  • RTK, Serena, and RepoMap produce mixed costs because their fixed configurations do not adapt selectively from prior repository experience.The strategies respectively compress command outputs, add semantic code navigation, and inject a structural repository map.Source: §2.1 · p. 3
  • Every fixed strategy has mixed repository-level effects: all three lower costs on Django but raise them on SymPy, while scikit-learn and Sphinx favor different strategies.On scikit-learn only RTK helps; on Sphinx only RepoMap does.Source: §2.1 · p. 3

2.2 Three Sources of Removable Cost

  • Within-repository action reuse grows with task history: 30% of calls match an earlier task after at least five tasks, versus 7% against another repository’s complete history.Repeated actions account for 12–28% of tool-call bytes in repositories where they recur in at least half of tasks.Source: §2.2 · pp. 3–4, Figure 3 · p. 4
  • Although agents see 23 tool schemas each turn, no repository invokes more than seven; 16 tools are never called and three account for 98% of calls.This motivates evidence-based pruning that can recover a tool when needed, rather than a fixed reduced interface.Source: §2.2 · p. 4, Figure 3 · p. 4
  • Ten failed executions are 17% of tasks but 32% of spend, yet repeated failure signatures also occur in successful runs, so intervention must distinguish loops from recoverable work.The authors motivate comparing trajectory length with repository successes and redirecting before termination; the three adaptive decisions are learned separately within each environment.Source: §2.2 · p. 4, Figure 3 · p. 4

3 TokenEconomizer

TokenEconomizer uses shared, repository-agnostic meta-strategies to learn environment-specific policies from completed traces. These policies shape which micro-skills and tools are used and when the controller intervenes in a failure loop.

3 TokenEconomizer

  • TokenEconomizer learns repository-specific policies from completed traces and applies them through instructions, loaded tools, and live intervention.The same three meta-strategies are applied across repositories.Source: §3 · p. 4, Figure 4 · p. 5
  • For each task in a chronological repository stream, the controller uses prior task state to select policies and updates that state from the resulting trajectory.Repository state accumulates micro-skills and supporting evidence, tool-use history, and turn statistics from completed tasks.Source: §3 · p. 5

3.1 Meta-strategy I: Experience-grounded Micro-skills

  • Micro-skills are proposed from completed-trajectory digests and activated only when evidence supports both repository-level reliability and repeated need.Contradictions can retire a confirmed skill, while a confirmed skill below the need threshold is withheld rather than retired.Source: §3.1 · pp. 5–6

3.2 Meta-strategy II: Tool Pruning

  • Tool pruning removes non-core tools after sustained non-use or an irrelevance judgment, while periodic full-catalog re-audits can recover mistaken exclusions.Core shell and file tools are always retained, and a tool called during re-audit is never masked again.Source: §3.2 · p. 6

3.3 Meta-strategy III: Failure-loop Intervention

  • Failure-loop intervention uses prior successful-task turn counts to set a reference budget, then checks whether unusually long runs are unproductive.A judge can redirect a stuck agent and, if the pattern recurs, terminate it; a strict stall rule also governs the forced stop.Source: §3.3 · pp. 6–7

4 Experiments

The experiments compare TokenEconomizer with default execution, reduced reasoning effort, fixed strategies, and combinations across SWE-Bench, then test transfer to web development and security auditing. TokenEconomizer lowers cost across settings and repositories, with savings also observed on the additional benchmarks.

4.1 Setup

  • The evaluation covers five agent–model settings on the same 277 SWE-Bench tasks across six repositories, plus 102 Web-Bench and 200 security tasks.Efficiency is mean cache-aware cost, including auxiliary controller calls; quality is Pass@1 on SWE-Bench, cumulative Pass@2 on Web-Bench, and detection F1 on security benchmarks.Source: §4.1 · p. 7

4.2 SWE-Bench Results

  • 18.4–30.9% lower mean cache-aware cost across five SWE-Bench settings; Pass@1 improves in four and falls by 1.0 percentage point only on Opus 4.8.Both combinations with fixed strategies reduce cost in all five settings; adding RTK on Opus 4.8 raises savings from 30.9% to 33.2% with Pass@1 above default.Source: §4.2 · p. 7, Table 1 · p. 8
  • TokenEconomizer cuts cost in all 30 repository–setting cells by 5.2–40.1%, while RTK and RepoMap alone increase cost in 12 and 15 cells, respectively.Figure 5a shows repository–setting cost changes; TokenEconomizer+RTK also reduces cost in all 30 cells, and TokenEconomizer+RepoMap in 29.Source: §4.2 · p. 7, Figure 5 · p. 8
  • TokenEconomizer variants account for every Pareto-frontier point that maintains or improves default Pass@1, while reduced reasoning effort saves cost with substantial Pass@1 losses.RTK and RepoMap stay closer to default and move in mixed directions.Source: §4.2 · pp. 7–8
  • After starting without prior traces, TokenEconomizer becomes cheaper than default by the sixth task in every repository and ends with 16.9–32.0% cumulative savings.Figure 5b plots cumulative cost changes within each repository, averaged over the two Claude Code settings.Source: §4.2 · p. 8, Figure 5 · p. 8
  • Tool pruning provides the largest single ablation savings on Claude Code, at 13.2% for Sonnet 5 and 17.7% for Opus 4.8.Each single strategy keeps Pass@1 within 3 points of default, but none alone matches the full stack’s savings.Source: §4.2 · pp. 8–9

4.3 Generalization to Diverse Task Structures

  • Across Web-Bench and the security benchmarks, cost falls by 16.0–21.2% on Web-Bench, 16.5–19.6% on CredData, and 6.4–9.3% on ReposVul.Security tasks use different repositories, so repository-specific micro-skill injection is disabled; shorter ReposVul trajectories leave less removable cost.Source: §4.3 · p. 9, Table 2 · p. 9

5 Related Work

Prior cost-reduction work addresses model choice or within-trajectory context, interface, and runtime controls, while cross-task work reuses executions or builds reusable capabilities. TokenEconomizer differs by adapting cost-control policies from completed traces through exposed agent interfaces.

5 Related Work

  • Existing LLM cost methods use cascades or routers to select models, trading answer quality for cost on each query, whereas TokenEconomizer keeps the model and effort fixed.For coding agents, prior approaches control context, interfaces, or rollout length; interface-deployable methods are typically fixed and task-local, with effects that vary across repositories.Source: §5 · p. 9
  • TokenEconomizer uses completed traces to adapt instruction prefixes, tool sets, and stopping policies to each environment through exposed interfaces.It builds on cross-task experience reuse while addressing the repository-varying effects of fixed strategies.Source: §5 · pp. 9–10
  • Cross-task systems reuse completed executions as verbal lessons, workflows, repository knowledge, or reusable skills and tool libraries.Because indiscriminate repository context can increase cost without generally improving success, experience must be reused selectively.Source: §5 · p. 10

6 Limitations and Conclusion

The paper identifies decision errors, limited harness coverage, and the use of related but independent SWE-Bench fixes as limitations. It concludes that an external controller can learn from completed work to reduce coding-agent costs while largely preserving task success across several domains.

6 Limitations and Conclusion

  • A micro-skill may become stale, a pruned tool may become necessary, and a productive difficult task may be mistaken for a failure loop.Recovery paths retire contradicted skills, restore tools at the next re-audit, and redirect failure-loop interventions before stopping uncertain cases.Source: §6 · p. 10
  • Experiments cover three coding agents, not more minimal harnesses such as Pi, and SWE-Bench streams use related independent fixes rather than actual user sessions.Supporting a more minimal harness requires an adapter; continuous sessions are proposed as a natural next test.Source: §6 · p. 10
  • TokenEconomizer learns from completed work to make an unchanged agent cheaper while largely preserving task success across software engineering, web development, and security auditing.Source: §6 · p. 10

A Extended Discussion

TokenEconomizer’s savings generalize across repositories and benchmarks without per-environment tuning, while the authors frame adaptive black-box cost control as promising but not yet fully explored. Micro-skills and fixed strategies can supply complementary information, though combining them trades savings against quality.

Extended discussion

  • The authors present TokenEconomizer as one instance of external controllers that use completed traces to set environment-specific policies, not as the ceiling of this direction.They describe possible extensions including task-specific model or reasoning choices, learned context compaction, and environment setup reuse.Source: A Extended Discussion · p. 13

Generalization without per-environment tuning

  • Savings hold in all 30 SWE-Bench repository–setting cells without tuning, and TokenEconomizer saves 16.0–21.2% on workspace-inheriting Web-Bench streams.SWE-Bench tasks use fresh checkouts across histories spanning 1.9 to 8.5 years; the authors describe this as a conservative test of experience reuse.Source: A Extended Discussion · p. 13

Complementarity with fixed strategies

  • On Opus 4.8, adding RepoMap or RTK raises Pass@1 above default, but combinations trade some savings for quality and vary by setting.Micro-skills convey repository operation, RepoMap supplies ranked symbol signatures for localization, and RTK keeps failure messages salient in shorter context.Source: A Extended Discussion · p. 14

Why micro-skills

  • Micro-skills encode single environment facts that can be detector-verified across later traces, unlike broader reusable lessons or task-solving procedures.Their brevity matters because the manual is billed every turn; fact-based entries avoid carrying task-specific content between tasks.Source: A Extended Discussion · p. 14

B Cost Profiling Protocol and Detailed Results

The cost-profiling protocol compares fixed strategies across repositories and estimates exposure to removable cost from default traces. Controller settings are shared across projects, with benchmark-specific adjustments only where stream structure requires them.

B Cost Profiling Protocol and Detailed Results

  • Table 3 reports, by repository, tasks passed out of 10 and each strategy’s cost change relative to default execution on the same tasks.The protocol runs Claude Code with Opus 4.8 at high reasoning effort on 10 tasks from each of six repositories.Source: Table 3 · p. 14, B Cost Profiling Protocol and Detailed… · p. 14
  • The three fixed strategies are RTK output compression, Serena semantic code navigation, and RepoMap structural context injection; TokenEconomizer combinations test complementarity with RTK and RepoMap.Serena is excluded from the combination comparison because tool pruning could remove its server and confound the methods.Source: B Cost Profiling Protocol and Detailed… · pp. 14–15
  • Repeated actions, unused tools, and failed-task spending estimate exposure to removable cost from default traces, so these measurements are upper bounds rather than achievable savings.Repeated actions match normalized tool-call keys across tasks, while unused tools compare the initial 23-tool catalog with actual calls.Source: B Cost Profiling Protocol and Detailed… · p. 15

C Controller Configuration

  • One controller configuration is shared across repositories and projects, with values fixed before main runs rather than tuned per repository, project, or stream.Web-Bench uses a 30-turn initial guard budget until five tasks have passed; security-benchmark micro-skill injection is disabled because tasks use different repositories.Source: C Controller Configuration · p. 15

D Task and Stream Selection

The evaluation uses ordered task streams across software engineering, sequential web development, and security auditing, with task construction and accounting designed to preserve stream order and include controller overhead.

D Task and Stream Selection

  • The SWE-Bench streams contain 277 tasks across six repositories, mixing verified and full-split tasks; added difficult tasks were checked by applying gold patches and running official evaluation.Tasks run in base-commit order, and an answer-key guard blocks direct access to public fix commits.Source: D Task and Stream Selection · p. 16
  • Web-Bench uses five projects with 102 dependent tasks, and reports Pass@2 because retrying with test errors as context prevents one failed attempt from ending the stream.Projects span web standards and frameworks; projects where default execution fails early are excluded under the benchmark’s stopping protocol.Source: D Task and Stream Selection · pp. 16–17
  • CredData and ReposVul select 100 repositories or instances each for line-level security audits; ReposVul excludes instances with no vulnerable line to detect or outdated flags.ReposVul counts a reported location only when its file and line match a line removed by the fixing commit.Source: D Task and Stream Selection · p. 17

E Task Stream Formulation and Cost Accounting

  • The task stream updates from prior state, but each controller decision depends only on earlier tasks and the current trajectory prefix, not later tasks.A trajectory alternates observations and actions, and success is recorded as a binary label.Source: E Task Stream Formulation and Cost… · p. 17
  • Cost accounting includes task-agent usage and attributable auxiliary calls, priced using provider-reported uncached-input, cache-read, and output token classes.Each auxiliary call is charged once to the task boundary that invokes it, and the shared execution record prevents controller work from being omitted.Source: E Task Stream Formulation and Cost… · pp. 17–18
  • The design objective is to reduce amortized stream cost while preserving default-execution success; experiments compare observed cost and success against matched defaults.The objective is not an evaluation-time tuning rule: the tolerance parameter is not selected after observing results.Source: E Task Stream Formulation and Cost… · p. 18

F Additional Meta-strategy Details

The appendix details how the controller proposes and retires micro-skills, prunes tools, and intervenes in possible failure loops, alongside the prompts used for those decisions.

F Additional Meta-strategy Details

  • Micro-skills are proposed from recent task digests, require detector-confirmed evidence across cited tasks, activate only when needed, and retire when support turns negative.Contradictions are checked before confirmations, and high-stakes skills retire at the first contradiction after confirmation.Source: F Additional Meta-strategy Details · p. 18, Table 5 · p. 18
  • Unused tools are masked after three uncalled tasks or an irrelevance judgment, while re-audits restore the full catalog without another judge call.The mask remains fixed for the task; a failed judge call leaves tools loaded.Source: F Additional Meta-strategy Details · p. 18
  • The failure-loop guard combines repeated errors, edit churn, and stagnation signals; a strict stall at 2.5 times the turn budget ends the rollout without another redirect.When consulted, the judge may prompt a redirect that summarizes the agent’s actions and asks it to try a materially different hypothesis.Source: F Additional Meta-strategy Details · pp. 18–19

G.1 Micro-skill Injection

  • The proposer prompt mines durable repository facts from recent task digests, allows at most four skills, and requires citations and working detectors rather than task-specific fixes.Prompt guidance emphasizes canonical commands, setup facts, and other information useful to future tasks.Source: §G.1 · p. 19

G.2 Tool Pruning

  • The tool-judge prompt evaluates whether candidate tools are relevant to one issue and directs conservative dropping only when irrelevance is clear.Issue text and candidate tool descriptions are supplied, while repository links in the task description are replaced by a placeholder.Source: §G.2 · pp. 19–20

G.3 Failure-loop Intervention

  • The trace-judge prompt evaluates whether recent actions show a repetitive dead end or ongoing progress, favoring sparing the agent when progress is visible.The redirect writer names a pattern in the digest and proposes a materially different next step without telling the agent to stop.Source: §G.3 · p. 20

H Ablations of Meta-strategies

The ablations compare single-meta-strategy variants with the combined stack using shared parameters, while activity counts show selective decisions and little interference. Appendix examples illustrate the three strategies’ decisions.

H Ablations of Meta-strategies

  • Single-strategy variants share the full stack’s parameters, avoiding variant-specific tuning; Table 2a reports cost and Pass@1, while activity counts are per-stream means with differing units.Needed counts task events, Masked counts tools remaining masked, and Re-audits counts tasks run with the full catalog, so Masked and Re-audits are not comparable.Source: H Ablations of Meta-strategies · p. 20, Table 6 · p. 21, Table 7 · p. 21
  • Combined-stack activity counts stay close to those of corresponding single-strategy variants, and the strategies seldom interfere; skills are retired, while redirects and aborts are selective.About one in five proposed skills is retired; the trace judge is consulted 10–13 times per stream, redirects follow on roughly one task in ten, and aborts on fewer than one in twenty.Source: H Ablations of Meta-strategies · p. 21

I Examples of Meta-strategy Decisions

  • The appendix organizes examples as micro-skills, tool masks, and failure-loop interventions; several compare TokenEconomizer with default execution on the same task.The cited passage specifies example categories, but not the outcomes of those comparisons.Source: I Examples of Meta-strategy Decisions · p. 21

I.1 Micro-skill Injection

Micro-skill examples show how confirmation, contradiction, and task need shape whether operational instructions are injected or retired. Tool-pruning examples compare masks and first-request prompt tokens across the first four Django tasks.

I.1 Micro-skill Injection

  • A pytest instruction was activated because its detector matched the command, but one contradictory module check outweighed decayed confirmations and retired it after one injection.All 57 traces invoking pytest failed because pytest was unavailable; the skill’s confirming detector did not check for a passing result.Source: §I.1 · p. 23
  • A confirmed reinstall warning never cleared the need gate because the agent never attempted a reinstall, so injection would add tokens without preventing waste.This example distinguishes confirmation of a skill from evidence that the task needs it.Source: §I.1 · p. 24
  • A sphinx skill was never activated because all but one confirmation came from edits to python.py, indicating recurrence in one module rather than a repository-wide practice.The examples distinguish retirement after contradiction, confirmation without need, and failure to meet the confirmation condition.Source: §I.1 · pp. 22, 24

I.2 Tool Pruning

  • The Django tool-pruning examples compare masks and first-request prompt-token counts for the first four tasks under Claude Code and Copilot CLI.The passages specify what is compared but do not state the comparison’s outcome.Source: §I.2 · pp. 24–25

I.3 Failure-loop Intervention

Failure-loop examples show both redirection and forced stopping, while harness-specific implementations adapt controller actions to each CLI’s exposed interfaces. The examples include a redirect followed by testing and a forced stop on a task that default execution also fails.

I.3 Failure-loop Intervention

  • The other example forcibly stops an xarray task that TokenEconomizer fails, and default execution also fails on that task.By contrast, the sphinx redirect example ends in fewer turns than default execution.Source: §I.3 · p. 25
  • In the redirect example, 11 edits across one file and three repetitions of the same failure trigger a prompt to try a materially different hypothesis and verify it.The example reports a stuck verdict with confidence 0.72; afterward, the agent runs the relevant test.Source: §I.3 · p. 25

J Harness-specific Customization

  • Across harnesses, adapters use exposed prompts, tool settings, and event streams with a shared normalized trace representation; Copilot CLI reads token usage from its debug log.The normalized representation includes tool names, arguments, observations, model outputs, usage, and grading outcomes.Source: J Harness-specific Customization · pp. 25–26
  • Tool pruning differs by harness: Codex switches capability packages rather than individual tools, while Claude Code removes masked tool schemas from requests.Codex’s packages include sub-agent collaboration, goal tracking, image input, bundled skills, and headless-unused chat guidance.Source: J Harness-specific Customization · p. 26

Related papers

Loading 2610.22732v1…