Source-linked AI summary
Thinking Costs Tokens: When More Structure is Worth the Price
Thomas Nolasque, John Grey, Calista Pham, Ankit Vani
TL;DR
The paper asks whether inference structure pays for itself only above a token-budget threshold, addressing the trade-off between planning and verification overhead and answer-generation capacity. It evaluates monolithic and verified-search systems across financial reasoning cases and budget tiers, finding a crossover between 1,000 and 1,500 output-equivalent tokens, with verified search ahead from 1,500 onward.
Problem
The paper asks whether extra inference structure pays for itself only above a token-budget threshold in financial reasoning tasks.
Method
The study compares a monolith with verified search across 1,000 cases, 2 systems, and 14 budget tiers from 250 to 42,000 output-equivalent tokens.
Results
Verified search surpasses the monolith from 1,500 tokens onward, with a 3-5 percentage-point advantage at every tier through 42,000 tokens and a crossover confirmed at p ≤0.001 at both endpoints.
Takeaways & Limitations
Architecture selection should be conditional on available token budget: the monolith is more efficient below approximately 1,500 tokens, while verified search is better at 1,500 tokens or more.
Takeaways & Limitations
Results may shift with the underlying model, decoding strategy, prompt wording, scoring metric, case difficulty, and the absence of a human expert baseline.
Abstract
from arXiv · showhide
Adding inference structure to a language model lets it search, verify, and revise, but these actions consume the very budget they are supposed to use well. In this paper, we investigate whether there exists a token-budget threshold, below which the overhead of planning and verification hurts performance and above which it helps. We evaluate two systems on FinQA and TAT-QA financial reasoning tasks, using GPT-5.4 mini across 14 budget tiers ranging from 250 to 42,000 output-equivalent tokens. The first system is a monolith, which is a single LLM call. The second is a verified search architecture that adds planning, label-blind checking, and repair capabilities. We run 1,000 cases for a total of 28,000 completed cells. Both systems score 0% at the two lowest tiers, where neither can fit a complete prompt. At 1,000 tokens, the monolith reaches 18% accuracy while verified search scores near 0%, since the planning overhead leaves no room for an answer. From 1,500 tokens onward, verified search surpasses the monolith and maintains a consistent advantage, reaching approximately 44% at the highest tiers while the monolith reaches approximately 40%. The crossover occurs between 1,000 and 1,500 output-equivalent tokens, confirmed by a strict intersection-union test ($p \le 0.001$ at both endpoints).
1 Introduction
The paper asks whether added inference structure becomes worthwhile only above a token-budget threshold. It predicts a crossover in which the monolith is more efficient at low budgets, while verified search benefits once its planning and verification pipeline fits.
- Motivation: Planning, retrieval, verification, and repair can improve answer quality but consume tokens that may otherwise support reasoning.Repeated instructions, candidate generation, and coordination overhead can crowd out the answer.
- Research question: The study tests whether extra inference structure pays for itself only above a token-budget threshold on financial reasoning tasks.The experiment varies architecture and token-budget tier, measuring answer correctness.
- Hypotheses: The monolith is expected to achieve reasonable accuracy at a lower minimum budget because it spends its calls directly on retrieval and answering.Verified search must first allocate tokens to planning, retrieval, candidates, and checking.
- Hypotheses: Verified search is expected to outperform the monolith once the budget supports planning, multiple candidates, and label-blind checking.These mechanisms are hypothesized to improve evidence selection, draft quality, and rejection of unsupported arithmetic.
- Crossover definition: The predicted crossover means the monolith has a better return below a budget region, while verified search has a better return above it.The confirmatory hypothesis formalizes this as D_low < 0 and D_high > 0, where D_b is verified-search accuracy minus monolith accuracy.
2 Related work
Prior work finds that additional test-time computation can improve reasoning, but its value depends on task difficulty and informative external feedback. These findings motivate allocating resources across budget tiers and using grounded checking rather than unsupported self-correction.
- Test-time computation: Additional test-time computation can outperform scaling model parameters, but its optimal allocation depends on task difficulty.Easy problems benefit less from additional compute, whereas hard problems can improve meaningfully.
- Self-correction: Self-correction is ineffective without external feedback signals and can degrade performance when models rely on their own confidence.The paper therefore uses a checker that supplies grounded validation signals.
2.3 Chain-of-thought and structured reasoning
Chain-of-thought, branching reasoning, and retrieval augment language models with intermediate structure or external evidence. This paper extends that line by examining how retrieval and structured reasoning interact with a fixed token budget.
- Chain-of-thought: Intermediate reasoning steps improve performance on arithmetic, commonsense, and symbolic reasoning tasks.The cited work establishes chain-of-thought prompting as a structured-reasoning approach.
- Retrieval augmentation: Retrieval-augmented generation combines a parametric language model with a dense retrieval index to ground generation in external evidence.The paper’s systems use retrieval to supply financial evidence during inference.
- Budget interaction: The paper studies retrieval under a fixed token budget because each retrieval round consumes tokens that could otherwise support answer generation.This frames retrieval as a resource-allocation trade-off rather than an unbounded capability addition.
2.5 Agentic reasoning and tool use
Agentic reasoning interleaves reasoning with actions such as search, enabling models to plan, retrieve, and revise during inference. The verified-search architecture adopts this pattern while focusing on its cost under a constrained token budget.
- Agentic reasoning: ReAct interleaves reasoning traces with actions such as search queries within a single inference trajectory.This design lets language models plan, retrieve, and revise rather than relying on reasoning or acting alone.
- Verified search: The verified-search architecture plans queries, retrieves evidence, generates candidates, and repairs failures within one budget-constrained execution.Its key focus is the token cost of this interleaved design.
2.6 Iterative refinement and repair
Prior iterative refinement methods improve outputs through feedback and revision, while this paper emphasizes grounded test-time verification and repair as related mechanisms.
- Iterative refinement methods generate an output, provide feedback, and revise it without additional training.
- Process supervision supports checking intermediate reasoning steps rather than evaluating only the final answer.
- The verified search checker validates arithmetic and citation provenance without access to gold labels.
2.8 Financial question answering
FinQA and TAT-QA evaluate financial question answering over combined tabular and textual evidence, including multi-step numerical reasoning.
- FinQA pairs financial-report questions with text, tables, and executable reasoning programs.
- TAT-QA combines tabular and textual financial evidence with arithmetic and counting questions.
- Both datasets require multi-step numerical reasoning over structured evidence.
3 Methodology
The study compares two fixed inference architectures on rigorously selected financial reasoning cases under hard, price-adjusted token budgets, with confirmatory crossover testing.
- Datasets and selection criteria: 1,000 balanced cases comprise 500 FinQA and 500 TAT-QA questions requiring at least two derivation operations.
- System comparison: Both systems use the same model, retriever, instructions, evidence serialization, and candidate schema; only inference structure varies.
- Verification: The verified-search checker validates citations, arithmetic, units, scales, entities, periods, and division safety without seeing correctness or a gold value.
- Budget intervention: The hard budget sums prompt and completion tokens across all calls in each case-system-tier cell.
- Budget intervention: Input tokens are converted to output-equivalent tokens using a 0.17 input-to-output price ratio for GPT-5.4 Mini.
- Confirmatory inference: Crossover confirmation requires both directional one-sided exact McNemar tests to reject at α = .05.
4 Results
Across 1,000 cases and 14 budget tiers, verified search is initially constrained by planning overhead but overtakes the monolith at 1,500 tokens and remains ahead thereafter.
- 28,000 scheduled cells cover 1,000 cases, 2 systems, and 14 budget tiers using gpt-5.4-mini-2026-03-17-eastus-dz.
- 0% accuracy occurs for both systems at t250 and t500 because neither fits a complete prompt and valid answer.
- 18% versus 0.1% accuracy at t1000 favors the monolith because verified search exhausts its budget on mandatory planning.
- 24% versus 20.6% accuracy at t1500 marks verified search’s first advantage and the minimum tier fitting its full pipeline.
- From t1500 onward, verified search maintains an advantage over the monolith at every tier.
- Approximately 44% versus 40% accuracy at t42000 shows continued improvement for verified search and the monolith, respectively.
- p ≤ 0.001 at both t1000 and t1500 confirms a strict crossover under the intersection-union test.
5 Discussion
Verified search is worthwhile only once the token budget supports its full pipeline: the monolith is better below approximately 1,500 tokens, while verified search leads from t1500 onward. This result is promising but bounded by fixed architectures, hand-designed allocations, shared retrieval, unexamined cost measures, aggregate scoring, and financial-task scope.
- Budget-dependent architecture choice: At t1000, the monolith is the better investment, while verified search cannot produce a candidate.The crossover occurs when verified search can fit its planning, retrieval, candidate-generation, and checking pipeline within the allocation.
- Budget-dependent architecture choice: From t1500 onward, verified search remains ahead, with a 3-5 percentage-point gap at every higher tier.Its advantage is attributed to planning, broader retrieval, multiple candidates, checking, and repair becoming affordable.
- Practical implication: Below approximately 1,500 output-equivalent tokens, the monolith is more efficient; at 1,500 or more, verified search is recommended.The paper frames architecture selection as conditional on the available token budget.
- Limitations: The comparison uses two fixed architectures, excluding hybrid or adaptive strategies that switch according to available budget.The budget-to-action mapping was hand-designed, and a learned or adaptive allocation policy might perform better.
- Limitations: Both systems share a deterministic BM25 retriever, so a learned dense retriever could shift the crossover or narrow the architectural gap.
- Limitations: The study measures gateway-reported prompt and completion tokens but does not analyze tool calls, CPU time, wall time, latency, or monetary cost.It also uses exact numeric match without partial credit and reports aggregate accuracy without stratifying case difficulty.
- Limitations: The findings are limited to financial reasoning tasks from FinQA and TAT-QA, leaving generalization to other domains open.
6 Conclusion
The paper asks whether inference structure pays for itself only beyond a token-budget threshold and concludes that it does conditionally. Verified search is less efficient below approximately 1,500 tokens but provides the better return at and above that threshold, with the crossover statistically confirmed.
- Conclusion: Below approximately 1,500 output-equivalent tokens, the monolith provides a better return because verified search cannot fit its full pipeline.
- Conclusion: At 1,500 tokens and above, verified search provides a better return through broader evidence coverage, multiple candidate attempts, and label-blind verification.
- Conclusion: The crossover at 1,000-1,500 tokens is confirmed by a strict intersection-union test with p ≤0.001 at both endpoints over 1,000 cases.