Source-linked AI summary
Not All Tokens Are Equal: Inflation-Aware Routing for Agentic LLM Systems
Heming Fu, Shan Lin, Qianqian Xie, Guojun Xiong
TL;DR
Agentic retries make single-call token prices underestimate true workflow costs, complicating model routing. InflationAgent predicts retry-driven cost and routes by expected accuracy per true cost, achieving 94.7% accuracy versus 91.0% for FrugalGPT with 31% fewer tokens on GSM8K.
Problem
Agentic retries make single-call cost estimates diverge from true workflow costs, limiting cost-aware model routing.
Method
InflationAgent measures token inflation, predicts it using CoT Branching Entropy and a lightweight MLP, then routes by Semantic Exchange Rate with fresh escalation.
Results
94.7% accuracy versus 91.0% for FrugalGPT with 31% fewer tokens on GSM8K under a fixed token budget.
Takeaways & Limitations
Routing should account for retry-driven true cost rather than nominal per-token price when selecting models for agentic workflows.
Takeaways & Limitations
The evaluation covers two reasoning benchmarks and does not extend to open-ended generation, multi-turn dialogue, or tool-use agents.
Abstract
from arXiv · showhide
When a language model fails to answer a query on the first attempt, an agentic system retries, consuming additional tokens each time. This retry overhead creates a gap between what a model's per-token price implies and what a full workflow actually costs. We call this gap \emph{token inflation} and define it as the ratio of true workflow cost to single-call cost. Systems like FrugalGPT route based on the latter, which can underestimate real cost by more than $2\times$ on difficult tasks. We address this with InflationAgent, a four-stage router that (1) measures token inflation systematically across model tiers and task types, finding inflation as high as $4.25\times$ for a 7B model on multi-hop question answering; (2) introduces CoT Branching Entropy (CBE), a pre-execution difficulty signal computed entirely from local inference, which predicts high inflation with AUROC 0.887; and (3) selects models by maximizing a Semantic Exchange Rate (SER) that divides expected accuracy by predicted true cost, with a fresh-escalation policy that discards failed chains before routing to a stronger model. On GSM8K under a fixed budget, InflationAgent achieves 94.7\% accuracy versus 91.0\% for FrugalGPT while using 31\% fewer tokens, and we show that forwarding a failed reasoning chain to GPT-4o reduces its accuracy by up to 34.8 percentage points, validating the fresh-escalation design.
1 Introduction
Agentic retries create token inflation because single-call routing estimates can ignore substantial workflow cost. InflationAgent addresses this gap by measuring inflation across models and tasks and introducing a no-API-cost signal to predict retry likelihood.
- Motivation: Retrying wrong or incomplete responses, tool failures, self-consistency sampling, and code-test loops consume tokens that routing decisions often overlook.The hidden retry cost occurs across multiple agentic patterns, including ReAct, self-consistency, and code-generation agents.
- Motivation: A 7B local model may retry five times and still fail, while GPT-4o answers correctly on its first attempt, making the nominally cheap option expensive in workflow cost.The example concerns a multi-hop question about a film soundtrack artist who collaborated with Michael Jackson on Thriller.
- Problem Definition: 2–5× error can arise when routing estimates multiply per-token price by expected single-call output length despite first-attempt failures.Token inflation is defined as actual workflow cost divided by the predicted single-call cost.
- Measurement: 4.25× inflation occurs for a 7B model on multi-hop QA, versus 1.31× for GPT-4o.InflationAgent instruments a retry harness across two reasoning tasks and three model tiers to measure this variation.
- Prediction: CoT Branching Entropy predicts retry likelihood from a small sample of local reasoning chains with no API cost.CBE is introduced as a pre-execution difficulty signal for routing.
2 Related Work
Prior work addresses inference-cost routing, uncertainty estimation, and multi-step agentic inference, but the supplied literature frames retry loops as standard practice without addressing their cost implications.
- LLM cost routing and cascades: FrugalGPT and related routing systems reduce inference cost by sending easy queries to smaller models or escalating across increasingly capable models.The related work also includes RouteLLM, Hybrid LLM, RouterBench, LLM-Blender, and language model cascades.
- Uncertainty estimation for language models: Self-consistency, semantic uncertainty, and prompted self-assessment estimate language-model uncertainty through agreement, semantic clustering, or calibrated confidence.CBE is described as related to self-consistency but serving a different purpose; the supplied passage truncates before specifying that purpose.
- Agentic systems and multi-step inference: Chain-of-thought prompting, ReAct, Reflexion, and AgentBench support or evaluate multi-step agentic inference, while retry loops remain unaddressed in cost terms.The passage characterizes retries as standard practice and states that their cost implications are not addressed.
3 Methodology
InflationAgent models true retry-inclusive workflow cost rather than relying on single-call cost, then routes by expected accuracy per predicted true cost. It estimates inflation from local reasoning behavior and uses that prediction to select the highest-SER model.
- Cost modeling: True workflow cost sums token use across retry attempts, whereas existing routers approximate cost with the first call alone.Token inflation is the ratio between these quantities.
- Cost modeling: Inflation ranges from 1× for first-attempt success to R× when all retry attempts are exhausted.Optimizing single-call cost therefore assumes an inflation ratio of 1, which fails on difficult multi-hop queries.
- Routing objective: The Semantic Exchange Rate maximizes expected accuracy per unit of true workflow cost.SER can prefer a slightly less accurate model when a higher-accuracy alternative has severe token inflation.
- Difficulty estimation: CoT Branching Entropy estimates query difficulty from answer disagreement across K independent local-model reasoning chains.The method computes the empirical answer distribution and its Shannon entropy without API calls, using local inference regardless of the deployed tier.
- Inflation prediction: A lightweight MLP combines CBE, agreement score, average chain length, and prompt token count to predict inflation and estimate each model’s true cost.Stages 1 and 2 run locally at no API cost, while Stage 3 selects the model with the highest predicted SER.
4 Evaluation
Evaluation across GSM8K and HotpotQA shows that token inflation varies sharply by task and model tier, while InflationAgent’s inflation-aware routing improves accuracy and token efficiency. Held-out prediction and fresh-escalation experiments support using CBE and discarding failed reasoning chains before escalation.
- Experimental setup: Evaluation uses Qwen2.5-7B-Instruct, GPT-4o-mini, and GPT-4o on GSM8K and HotpotQA under a retry-until-correct protocol with R = 5 maximum attempts.HotpotQA queries provide three candidate passages rather than the full ten.
- Token inflation: 4.25× mean inflation occurs for the small model on HotpotQA, compared with 3.15× for the medium tier and 2.92× for the large tier.GSM8K inflation is modest across tiers at 1.31–1.42×.
- Inflation prediction: 0.887 AUROC is achieved by the held-out inflation predictor, while CBE alone reaches approximately 0.81 AUROC.The predictor also achieves Pearson r = 0.714; chain length and prompt complexity contribute the remaining 8 points beyond CBE.
- Routing comparison: 94.7% accuracy is achieved by InflationAgent versus 91.0% for FrugalGPT under a fixed GSM8K token budget, while using 31% fewer tokens to reach FrugalGPT’s accuracy.Confidence Escalation achieves only 88.7%, indicating that CBE without an inflation model or fresh-call semantics is insufficient.
5 Conclusion
The conclusion argues that token inflation can reverse routing decisions and motivates InflationAgent’s CBE, inflation prediction, and SER-based fresh-escalation design. It also identifies evaluation and modeling limitations and outlines extensions to broader agent settings and adaptive routing.
- Conclusion: 4.25× inflation with 21.4% accuracy makes a nominally cheap model neither cheap nor effective, demonstrating how token inflation can invert routing decisions.Token inflation is defined as the gap between single-call cost estimates and true agentic workflow cost.
- Conclusion: InflationAgent combines zero-cost CoT Branching Entropy, a lightweight MLP inflation predictor, and SER-based routing with fresh escalation.The inflation predictor achieves AUROC 0.887.
- Limitations: The evaluation covers only two reasoning benchmarks, excludes open-ended, multi-turn, and tool-use settings, and leaves cross-task generalization and dynamic pricing unaddressed.The contamination estimate also uses a small stratum of 23 failure queries.
- Conclusion: 34.8 pp contamination reduces GPT-4o accuracy from 73.9% to 39.1% on 23 failure queries when the failed Qwen2.5-7B reasoning chain is prepended.The comparison is between a fresh prompt and forwarding the full failed reasoning chain.
- Future work: Future work targets CBE for multi-turn and tool-use settings, richer SER utility beyond binary accuracy, and online predictor adaptation to new task distributions.These extensions would reduce reliance on offline measurement campaigns and support more nuanced routing.