Source-linked AI summary
The Reasoning Tax: Token Economics of LLM Reasoning Across Task Types and Deployment Contexts
Sachin Gopal Wani, Ajay Dholakia, David Ellison
TL;DR
Accuracy-only benchmarks do not reveal when extended reasoning tokens justify their cost. The paper introduces TES and evaluates it across tasks, effort levels, and deployment contexts, finding task-structured efficiency, diminishing returns from greater effort, and altered economics under on-premises deployment.
Problem
Accuracy-only benchmarking does not answer whether extended reasoning tokens provide sufficient accuracy gains for their cost in different deployment contexts.
Method
The paper introduces TES, paired and approximated baseline variants, and analyzes 151 model-benchmark runs alongside RCS and DCM deployment measures.
Results
Reasoning efficiency is task-structured: sequential inference tasks yield positive TES, recall-heavy tasks do not, higher effort has diminishing returns, and deployment context changes economic viability.
Takeaways & Limitations
Reasoning should be enabled selectively according to task type, effort level, and deployment context rather than treated as universally beneficial.
Takeaways & Limitations
TES-A conflates reasoning mechanisms with general capability differences between model families, while token-count means can be influenced by outliers.
Abstract
from arXiv · showhide
Accuracy-only benchmarking of reasoning-capable large language models misses a central deployment question: when do extended thinking tokens earn their cost? We introduce the Token Economy Score (TES), a marginal benchmarking metric that measures the accuracy gain of a reasoning model over a non-reasoning baseline, normalized by the generated-token multiplier. We define paired and approximated TES variants for model families with reasoning toggles and frontier models without direct non-reasoning counterparts. We then conduct an empirical benchmarking analysis across 151 model-benchmark evaluation runs on seven benchmarks spanning mathematics, code generation, science reasoning, instruction following, expert knowledge, knowledge recall, and research-level physics. The analysis examines three deployment-facing dimensions: which task structures yield positive marginal reasoning efficiency, how increasing reasoning effort changes TES within model families, and how deployment context changes economic viability. Results show that task structure predicts reasoning efficiency better than nominal difficulty: sequential inferencechain tasks such as AIME 2025 and LiveCodeBench show high TES, while knowledge-recall tasks such as MMLU-Pro show low TES despite their difficulty. We also find systematic diminishing returns at higher reasoning effort levels, including cases where additional thinking reduces accuracy. Finally, Reasoning Cost Share (RCS) shows that inference spend is often dominated by internal thinking, while Deployment Cost Multiplier (DCM) shows how on-premises deployment can change the economics of otherwise costly reasoning workloads. These findings support a benchmarking-driven model-selection rule: enable reasoning selectively by task type, effort level, and deployment context rather than treating it as a universally beneficial mode.
1 Introduction
Reasoning-enabled models can improve performance but may incur substantially higher and unpredictable token costs, making deployment context part of model selection. The paper addresses this with Token Economy Score and an empirical analysis organized around task structure, reasoning effort, and deployment economics.
- Reasoning models generate extended thinking chains before answering, unlike instruction-following models.
- Reasoning gains on mathematics, coding, and scientific reasoning can require orders of magnitude more tokens than instruction-following alternatives.
- Token Economy Score measures whether reasoning accuracy gains justify generated-token overhead.
- The empirical analysis covers 151 model-benchmark runs across seven benchmarks and examines task structure, reasoning effort, and deployment context.
2 Related Work
Prior work measures absolute accuracy efficiency, reasoning verbosity, logical depth, or deployment cost, while this paper positions TES as a marginal, deployment-aware complement. TES focuses on gains over non-reasoning alternatives using generated tokens to isolate reasoning overhead.
- OckBench measures absolute accuracy per decoding token and identifies an Overthinking Tax in smaller reasoning models.
- LLMThinkBench finds that extended reasoning can substantially lengthen outputs without monotonic accuracy improvement in basic mathematics.
- TES differs by measuring marginal accuracy gain over a non-reasoning baseline rather than absolute accuracy per token.
- This paper combines reasoning-efficiency analysis with deployment cost across seven task-stratified benchmarks, extending prior cost and efficiency framings.
3 Token Economy Score
TES evaluates the marginal accuracy return from enabling reasoning relative to a non-reasoning baseline, while RCS and DCM separately characterize inference-cost composition and deployment differences. The framework includes paired and approximated baselines, but the approximated variant has an explicit capability-confounding limitation.
- Token Economy Score: TES compares reasoning accuracy gains with the generated-token multiplier over a non-reasoning baseline.
- Token Economy Score: Generated tokens include reasoning tokens and final output tokens, while input tokens are excluded because prompts and harnesses primarily determine them.
- Token Economy Score: TES > 1 indicates highly efficient reasoning; 0 < TES ≤ 1 indicates marginal reasoning; TES ≤ 0 indicates wasteful or harmful reasoning.
- Token Economy Score: Negative TES occurs empirically when reasoning models overthink problems that instruction-following models solve correctly without extended deliberation.
- TES Variants: TES-Δ pairs reasoning and non-reasoning variants within a model family, whereas TES-A uses the strongest available non-reasoning model as its baseline.
- TES Variants: TES-A conflates reasoning effects with general capability differences between model families, a deliberate limitation for frontier models lacking reasoning-disabled counterparts.
- Deployment Metrics: Reasoning Cost Share measures the fraction of total inference cost consumed by the thinking chain and is reported descriptively, not as part of TES.
- Deployment Metrics: Deployment Cost Multiplier compares total workload cost across cloud and on-premises deployment and is reported separately because TES is token-ratio based.
4 Experimental Setup
The study evaluates TES across diverse benchmarks, model families, reasoning settings, and deployment environments, while documenting assumptions and validity boundaries for cost and token measurements.
- Benchmark selection: Seven benchmarks span instruction following, knowledge recall, sequential inference, and frontier-domain reasoning, including both partially saturated and low-ceiling tasks.This design separates task structure from nominal difficulty.
- Model selection: 27 model configurations from eight families provide paired, reasoning-effort, and scale diversity for TES comparisons.Paired coverage supports within-family comparisons, while effort diversity enables analysis across available reasoning settings.
- Data collection: Cloud token, accuracy, and pricing data come from Artificial Analysis, while selected open-weight models receive independent on-premises evaluations.The self-run evaluations cover four benchmarks and validate reported accuracy values within expected variance.
- On-premises evaluation: On-premises per-token cost is derived from amortized hardware ownership and operating cost divided by measured throughput, including prefill and decode time.The system uses 8x NVIDIA B300 GPUs in FP16 precision; the amortized running cost is $0.00633 per second.
- Cost assumptions: Low-utilization or bursty deployments can have higher effective per-token costs than the full-utilization assumptions used for on-premises comparisons.Under those conditions, cloud APIs may remain preferable despite a large DCM under full-utilization assumptions.
- Validity and limitations: Mean token counts can be distorted by outlier problems, especially when hard benchmarks trigger exceptionally long reasoning chains.TES means and standard deviations are reported to reflect this variance.
5 Empirical Analysis
Across 151 evaluation runs, task structure predicts reasoning efficiency better than nominal difficulty, with sequential inference tasks producing stronger TES than recall-heavy or saturated tasks. Higher reasoning effort shows diminishing returns, while internal thinking dominates generated-token cost in most reasoning-enabled runs.
- 5.1 TES Across Benchmarks: Task structure governs TES more strongly than nominal difficulty: sequential inference-chain tasks outperform recall-heavy and saturated tasks.AIME 2025 has the highest mean TES; MMLU-Pro and GPQA Diamond show lower TES despite high conventional difficulty.
- 5.1 TES Across Benchmarks: AIME 2025 has the highest mean TES, while LiveCodeBench and IFBench exceed the TES > 1 threshold on average.These tasks require intermediate steps whose outputs constrain later actions, including proof construction, program synthesis, debugging, or multi-constraint instruction following.
- 5.1 TES Across Benchmarks: Gemini results use TES-A and therefore represent approximated cross-family comparisons.This comparison boundary should be considered when interpreting aggregate TES patterns.
- 5.2 Diminishing Returns Within Reasoning Effort Levels: Reasoning effort shows sharply diminishing marginal TES across model families, with some maximum-effort settings producing negative marginal accuracy.TES declines when additional tokens grow faster than the accuracy gain; maximum effort reduced accuracy for DeepSeek V4 Pro on GPQA Diamond and GPT-5.2 on LiveCodeBench.
- 5.2 Diminishing Returns Within Reasoning Effort Levels: The recommended effort workflow starts at medium or high effort and increases only when validation shows that added accuracy justifies token cost and latency.Maximum effort is treated as task- and model-specific rather than the default for difficult tasks.
- 5.3 Reasoning Cost Share: Reasoning tokens account for a median 94.7% of generated-token cost across reasoning-enabled entries.Several frontier-model runs exceed 99% RCS, while GPT-5.5 on GPQA Diamond rises from 76.7% RCS at high effort to 85.5% at xhigh effort.
6 Deployment Analysis
Deployment economics depend on both token efficiency and where inference runs. On-premises deployment can substantially reduce costs and alter which reasoning workloads are viable.
- On-Premises vs Cloud API: 2x to 26x lower cost on owned hardware than cloud APIs can materially change reasoning-model deployment recommendations.The reduction depends on model architecture.
- On-Premises vs Cloud API: DCM is primarily determined by model architecture, hardware configuration, and provider pricing rather than benchmark identity.It can therefore approximate the deployment-cost multiplier for comparable workloads on the same hardware.
- On-Premises vs Cloud API: TES remains unchanged by DCM, but a high DCM can make a cloud-marginal workload economically attractive on owned hardware.DCM changes dollar cost rather than token-efficiency measurement.
- On-Premises vs Cloud API: Qwen3.5-397B-A17B on GPQA Diamond has TES 1.233, while on-premises cost falls from $4.889 to $0.192 per run.This illustrates how high token efficiency and lower owned-hardware cost can coincide.
- Model Selection Implications: The model-selection framework combines TES with cloud inference cost across four quadrants using TES > 1 and $19.69 as thresholds.Figure 4 organizes 77 reasoning/non-reasoning comparisons in this space.
- Model Selection Implications: Q4 favors disabling reasoning, reducing effort, or replacing the model, whereas Q2 identifies high-TES workloads that may become practical on-premises.Q1 primarily contains AIME 2025 and selected moderate-effort LiveCodeBench cases; Q3 decisions depend more on application accuracy requirements.
7 Discussion: Implications for Agentic Systems
Single-turn TES does not capture how reasoning tokens compound across multi-step systems. Agentic deployment therefore requires matching reasoning to step type, managing context growth, and controlling effort per step.
- Token Compounding in Multi-Turn Systems: Single-turn TES counts one prompt, one response, and one generated-token total, whereas agentic systems can carry outputs into later inputs.Reasoning cost can therefore shift from per-call accounting to pipeline-level accounting.
- Token Compounding in Multi-Turn Systems: Retained intermediate outputs make context grow approximately as I_k = I_0 + ∑ G_j, while TES omits this downstream context pressure.High-RCS reasoning residue may not provide compact task state or reusable memory.
- Agentic Deployment Guidelines: High-RCS orchestrators and verbose subagents can crowd out instructions, evidence, tool outputs, and later planning state.A strategy acceptable in a single-turn benchmark can become inefficient at system level.
- Agentic Deployment Guidelines: Reasoning should be matched to step type, reserving it for planning, computation, code generation, verification, and other sequential inference processes.Retrieval, routing, formatting, summarization, and routine tool calls resemble low-TES tasks.
- Agentic Deployment Guidelines: The context window should be treated as a shared reasoning budget across instructions, evidence, tool outputs, intermediate state, and reasoning residue.Deployments should estimate reasoning-enabled steps multiplied by expected generated tokens per step.
- Agentic Deployment Guidelines: Medium or high effort is generally safer for intermediate steps than maximum effort, which should be reserved for terminal or high-stakes decisions.The recommendation follows sharply diminishing marginal TES at higher effort levels.
- Toward Agentic TES: Existing agentic benchmarks generally omit step-level token, reasoning-contribution, and context-growth accounting.An agentic TES extension would need pipeline-level accounting beyond the paper’s single-turn metric.
8 Conclusion
The paper presents TES as a deployment-oriented alternative to accuracy-only evaluation. Its results show that reasoning efficiency varies by task, effort level, and deployment context, while several methodological limits remain.
- Conclusion: Token Economy Score measures whether reasoning accuracy gains justify generated-token overhead relative to non-reasoning baselines.It reframes reasoning as a deployment decision rather than an accuracy-only leaderboard outcome.
- Conclusion: Sequential inference-chain tasks yield positive TES, whereas recall-heavy tasks such as MMLU-Pro generally do not.The conclusion identifies task structure as the central pattern in reasoning efficiency.
- Conclusion: Increasing reasoning effort produces sharply diminishing marginal TES, with confirmed maximum-effort accuracy declines in two model families.Effort level should therefore be selected explicitly rather than treated as a monotonic quality control.
- Conclusion: On-premises Mixture-of-Experts deployment can make economically marginal workloads viable through DCM values of 13–26x.RCS also indicates that reasoning tokens dominate inference cost in many frontier-model runs.
- Limitations: The main limitations are benchmark-level mean token counts, approximated baselines, pricing-epoch dependence, and TES’s single-turn scope.Future work includes uncertainty estimates, headroom-normalized TES, and agentic extensions for multi-step context growth.