Source-linked AI summary
Towards Optimizing the Costs of LLM Usage
Shivanshu Shekhar, Tanishq Dubey, Koyel Mukherjee, Apoorv Saxena, Atharv Tyagi, Nishanth Kotla
TL;DR
LLMs differ in cost and task- and context-specific quality, making cost-effective document processing a model-selection challenge. QC-Opt predicts quality and optimizes model choice and input length; the reported framework reduces costs while improving observed quality.
Problem
Method
Results
40–90% lower costs and 4–7% improved observed quality compared with closest baselines.
Source: §1 · p. 2Takeaways & Limitations
The framework reports significant cost savings with comparable quality in most cases, and sometimes improved quality from context-based model selection.
Source: §6 · p. 8The LLM-choice optimization discussion assumes quantitative quality predictions are available for each input.
Source: §4.1 · p. 4
Abstract
from arXiv · showhide
Generative AI and LLMs in particular are heavily used nowadays for various document processing tasks such as question answering and summarization. However, different LLMs come with different capabilities for different tasks as well as with different costs, tokenization, and latency. In fact, enterprises are already incurring huge costs of operating or using LLMs for their respective use cases. In this work, we propose optimizing the usage costs of LLMs by estimating their output quality (without actually invoking the LLMs), and then solving an optimization routine for the LLM selection to either keep costs under a budget, or minimize the costs, in a quality and latency aware manner. We propose a model to predict the output quality of LLMs on document processing tasks like summarization, followed by an LP rounding algorithm to optimize the selection of LLMs. We study optimization problems trading off the quality and costs, both theoretically and empirically. We further propose a sentence simplification model for reducing the number of tokens in a controlled manner. Additionally, we propose several deterministic heuristics for reducing tokens in a quality aware manner, and study the related optimization problem of applying the heuristics optimizing the quality and cost trade-off. We perform extensive empirical validation of our methods on not only enterprise datasets but also on open-source datasets, annotated by us, and show that we perform much better compared to closest baselines. Our methods reduce costs by 40%- 90% while improving quality by 4%-7%. We will release the annotated open source datasets to the community for further research and exploration.
1 INTRODUCTION
LLMs vary in cost and task- and document-specific capability, making quality-aware model selection difficult. QC-Opt combines quality prediction, constrained model selection, and token reduction; the paper reports lower costs and improved observed quality.
Motivation
- LLMs have diverse costs and capabilities, and their relative performance can vary by task and document format, making quality difficult to estimate without invoking models.The passage notes that existing output-based evaluation can add cost and latency, while common text-quality metrics may not align well with human perceptions.Source: §1 · p. 1
Contributions
- QC-Opt jointly optimizes LLM choice and input token count to reduce costs while maintaining output quality.Its proposed components include a quality predictor, cost-and-latency-constrained selection, and quality-aware token reduction.Source: §1 · p. 2
- The paper studies constrained cost, latency, and quality optimization, including problem hardness and polynomial-time algorithms for practically important special cases.The contribution list also describes an LP-rounding algorithm for selection.Source: §1 · p. 2
- For summarization, the paper proposes predicting LLM output quality without runtime queries and simplifying sentences with controlled token length and quality.It also proposes generalized token-reduction heuristics and optimized ways to apply them while controlling quality loss.Source: §1 · p. 2
Results
- Across public and enterprise datasets, the framework reduced costs by 40–90% and improved observed quality by 4–7% compared with the closest literature and industry baselines.Source: §1 · p. 2
Contributions
- The paper reports releasing annotated open-source training datasets for further community exploration.The introduction also describes extensive evaluation on public and enterprise datasets against literature and industry baselines.Source: §1 · p. 2
2 RELATED WORK
Prior approaches address model selection, prompt reduction, or caching, but the paper describes limitations in their runtime overhead, compression flexibility, or fit to context-aware routing.
Model Selection and Cascade
Prompt Length Reduction
- GPTrim applies token-reduction heuristics unconditionally and without tokenizer awareness, which can yield suboptimal reduction or even increase token count.The paper says GPTrim's heuristic set is not exhaustive.Source: §2 · p. 2
Sentence Compression
- SCRL drops words and is tokenizer-agnostic, limiting reordering and compression efficiency; changing its target length at inference requires retraining.The paper contrasts this with its own paraphrasing-based, tokenizer-aware module with controllable length and information loss at inference.Source: §2 · p. 2
Caching Based Approaches
- GPTCache caches API responses rather than selecting models by input context, and its usefulness depends on repeated queries from a shared distribution.The paper says that condition does not apply in its case.Source: §2 · p. 2
3 PROBLEM DESCRIPTION AND PROPOSED FRAMEWORK
QC-Opt targets high response quality with low cost and latency through predicted-quality model routing and quality-aware token reduction. The paper describes theoretical analysis and extensive empirical validation of the framework.
3 PROBLEM DESCRIPTION AND PROPOSED FRAMEWORK
- QC-Opt predicts model performance without making model queries, selects LLMs under performance, cost, latency, and system constraints, and reduces input tokens in a quality-aware manner.The framework’s three components are a quality predictor, budget-aware optimization, and token optimization.Source: §3 · p. 3, Figure 1 · p. 3
- The Smart Router combines quality prediction with budget-aware optimization to select an LLM for each document section.The paper motivates prediction because model performance varies by task and context, and runtime queries to evaluate candidate models add cost and latency.Source: §3 · p. 3
- The budget-aware quality optimization problem is NP-hard; the paper presents LP rounding, while cost minimization has polynomial algorithms for special cases.The framework also studies cost, latency, and quality optimization theoretically.Source: §3 · p. 3
- Input-token optimization uses both controlled text simplification and generic heuristics, with optimization to control quality loss from token reduction.The paper reports extensive validation of these approaches individually and in sequence.Source: §3 · p. 3
- The paper reports extensive validation of QC-Opt and its components, individually and in sequence, and says the full pipeline reduces costs while maintaining reasonable quality.The evaluation compares the approaches with closely related baselines.Source: §3 · p. 3
4 SMART ROUTER
The Smart Router estimates output quality for a task and context, then routes inputs among LLMs to maximize quality subject to budget and latency constraints.
4.1 Optimizing the Choice of LLMs
- The model-selection problem assigns one LLM to each document section using estimated quality scores, while accounting for model-specific costs and token lengths.Source: §4.1 · p. 4
4.1.1 Budget Aware Optimizer.
- Budget-Opt maximizes aggregate summary quality under monetary and latency constraints; after NP-hardness, LP relaxation and rounding empirically violate the budget by <0.2%.Source: §4.1.1 · p. 4
4.1.2 Quality Aware Cost Minimizer.
- Cost-Min minimizes total cost while maintaining a per-instance quality threshold, and the paper proves this optimization problem NP-hard.Source: §4.1.2 · p. 4
4.1.3 Polynomial Special Cases.
- Two special cases admit polynomial solutions: an O(K) greedy algorithm without latency constraints, and a minimum-cost maximum-flow solution when all sections have equal token length.Source: §4.1.3 · pp. 4–5
4.2 Estimating the Output Quality of LLMs
- A BERT-backbone regressor predicts each cascade model’s summarization quality against a gold standard from input text, avoiding model invocations at inference; training targets are reference BERTScores from candidate summaries.Dataset-I comprises approximately 1,000 text sections extracted from real-world PDF documents obtained from Adobe Inc. Dataset-II contains around 3,000 samples from bigpatent[35], samsum[18], and wiki bio[21].Source: §4.2 · p. 5
4.3 Experiments on Smart Router
- In the cost-versus-average-BERTScore comparison, our method achieves the same BERTScores as FrugalGPT-inspired baselines at significantly lower cost.Source: §4.3 · p. 6
- On Dataset I, our method cost 550.12 dollars and achieved 0.770 performance, yielding 84.50% lower cost and 3.2% higher performance than the Only Use Text-Davinci-003 baseline.Compared with the Only Use GPT-3.5-Turbo baseline, cost was reduced by 22.55% and performance improved by 1.2%.Source: §4.3 · p. 6
4.4 User Study
- When the predictor estimates a significant performance gap, its recommendation correlates strongly with human summary preferences; correlation is low when predicted scores are close.Source: §4.4 · pp. 6–7
5 OPTIMIZING TOKEN LENGTH
The token-length module reduces input tokens while controlling quality loss, addressing both usage costs and context-window limits. It combines token-aware sentence simplification with heuristics, then evaluates their compression and quality trade-offs.
5 OPTIMIZING TOKEN LENGTH
- Token reduction can lower LLM usage costs, but may lose information and meaning; token counts also depend on the model’s tokenizer.The paper notes limited prior work and no dedicated dataset for token-count reduction.Source: §5 · p. 7
5.1 Token Optimized Text Simplification
- The simplification model builds on controllable sentence simplification, using NUM_TOKENS_RATIO and BERT_SCORE tags to control token count and information loss.Training pairs come from WikiLarge; the token ratio uses OpenAI’s cl100k-base tokenizer, and BERT_SCORE is the BERTScore between the paired sentences.Source: §5.1 · p. 7
5.2 Token Optimization Heuristics
- The heuristic module devises eight text-editing rules and selects them per sentence by approximately solving a knapsack problem under a tolerated performance-loss capacity.It brute-forces the application order of the selected heuristics because there are at most eight.Source: §5.2 · pp. 7–8
5.3 Experiments on Token Optimization
- Our method and SCRL achieve comparable performance loss, while the token optimization module achieves more compression; GPTrim compresses most but incurs much higher performance loss.Figures 7 and 8 show quality loss and compression achieved, respectively.Source: §5.3 · p. 8, Figure 8 · p. 8, Figure 7 · p. 8
- Setting a quality-loss threshold reduces loss in a controlled manner while achieving similar token reduction to fixed-order brute-force application.The comparison evaluates optimized heuristic application against applying all heuristics in a fixed order.Source: §5.3 · p. 8
5.4 Validation of entire pipeline QC-Opt
- On Dataset II at budget 891, the full pipeline costs 579.429 with average BertScore 0.654, versus 891.08 and 0.773 without token optimization.At budget 550, estimated API-call wait time falls by about 13% against a GPT-3.5-Turbo latency baseline.Source: §5.4 · p. 8
6 CONCLUSION AND FUTURE WORK
The paper presents QC-opt and studies methods for reducing LLM usage costs while maintaining output quality. It reports significant cost savings with comparable quality in most cases, and identifies context-based model choice as a source of quality improvement in some cases.
6 CONCLUSION AND FUTURE WORK
- QC-opt combines its components with a theoretical study of optimizing LLM usage costs while maintaining output quality.The conclusion frames future work as extending quality and suitability estimation to a fully online, contextual setting.Source: §6 · p. 8
- The framework shows significant cost savings and comparable quality in most cases.In some cases, context-based model choice also improves quality.Source: §6 · p. 8
- The proposed next step is a fully online setting for contextual estimation of LLM quality and suitability.The conclusion does not report this online extension as completed.Source: §6 · p. 8
A.1 Proof of Theorem 4.1
The proof reduces Budget-Opt to 0-1 Knapsack in a relaxed setting with two models, no latency constraints, and one zero-cost model. Sections where the zero-cost model is at least as good can be assigned to it, leaving cost-bearing assignments to represent knapsack choices.
A.1 Proof of Theorem 4.1
- In the relaxed instance, each document section has quality scores for two models, while assigning it to the positive-cost model incurs cost proportional to its expected token length.The objective is to maximize total quality subject to budget B.Source: §A.1 · p. 9
- Sections where the zero-cost model scores at least as well can be assigned to it without lowering quality or increasing cost, so they leave the decision problem.For the remaining sections, the positive-cost model has a quality advantage Δ_i.Source: §A.1 · p. 9
- Choosing which remaining sections receive the positive-cost model is equivalent to 0-1 Knapsack: token-based costs are weights, and quality advantages Δ_i are values.The proof maps knapsack item values to Δ_i, weights to token lengths scaled by cost, and capacity to budget B.Source: §A.1 · p. 10
A.2 Proof of Theorem 4.2
The proof reduces a simplified Cost-Min feasibility problem to Partition by making each section’s total token size equal to an input element. Latency feasibility then corresponds to dividing the elements into equal-sum groups.
A.2 Proof of Theorem 4.2
- The reduction starts from Partition elements summing to 2B and asks whether they can be split into two groups of sum B.It constructs Cost-Min with two zero-cost models and latency constraints.Source: §A.2 · p. 10
- Each section’s input and output sizes are set so its total token size matches one Partition element, and each model receives the same latency coefficient.The latency threshold for either model is set to ℓB.Source: §A.2 · p. 10
- A Partition solution gives a latency-feasible assignment, and any latency-feasible assignment gives two groups whose element sums are B.Thus the feasibility instances have matching yes/no answers.Source: §A.2 · p. 10
A.3 Proof of Theorem 4.3
The proof selects, for each instance, the minimum-cost model meeting the quality threshold. Replacing any different optimal assignment with this choice cannot increase cost when there are no latency constraints.
A.3 Proof of Theorem 4.3
- For each instance, the greedy assignment chooses the minimum-cost model among those meeting the quality threshold.Feasible models satisfy S_i,j ≥ Q.Source: §A.3 · p. 10
- Any model chosen by an optimal solution must be feasible for that instance, or it would violate the quality constraint.The proof uses this feasibility to compare the optimal choice with the greedy choice.Source: §A.3 · p. 10
- With no latency constraints, replacing a different optimal choice by the greedy model cannot increase cost, so the greedy assignment is optimal.The proof applies this replacement to every instance where the optimal and greedy assignments differ.Source: §A.3 · p. 10
A.4 Proof of Theorem 4.4
The proof models section-to-model assignments as a minimum-cost maximum-flow problem, with feasible edges determined by estimated quality and model capacities limited by latency. Integrality yields minimum-cost assignments when feasible, and the algorithm runs in polynomial time.
A.4 Proof of Theorem 4.4
- Each model’s token capacity is the number of tokens it can accommodate while satisfying latency constraints.The proof normalizes model capacities and section sizes by d to obtain normalized capacity N̂_i.Source: §A.4 · p. 10
- The construction sends one unit of flow per document section through feasible section-to-model edges, with edge costs representing assignment costs.An edge exists only when the estimated quality satisfies S_i,j ≥ Q.Source: §A.4 · p. 10
- The token-capacity calculation assumes equal expected output-token size across sections under the earlier p-sentence summary assumption.The proof suggests estimating tokens per sentence from empirical data.Source: §A.4 · p. 10
- When all sections can be assigned feasibly, integral minimum-cost maximum flow yields the minimum-cost assignment; otherwise it maximizes feasible assignments at minimum cost.The stated complexity is O(|V|^2|E|), and Bellman–Ford can be used.Source: §A.4 · p. 11
B DATASETS
The token-optimization evaluation uses QA and NLI datasets, while Dataset I summarization is evaluated with BERTScore. The authors explain that long-form QA evaluation is limited by subjective metrics and LLMs’ ability to compensate for compressed or dropped text.
B DATASETS
- Token-optimization methods are evaluated using question-answering and NLI datasets, including multiple-choice QA over long-form QA datasets.The authors chose challenging MCQ and NLI tasks because suitable logical long-form QA datasets were unavailable to their knowledge.Source: B DATASETS · p. 11
- Dataset I summarization is evaluated with BERTScore, which the authors note has limitations for question-answering evaluation.Source: B DATASETS · p. 11
- Long-form QA results may not capture information loss from compression because LLMs can compensate for compressed text or dropped words, while performance variance across compression methods is minimal.The passage also notes that long-form QA metrics are unreliable given the task’s subjective nature.Source: B DATASETS · p. 11
C RESULTS ON TOKEN OPTIMIZATION
This section presents token-optimization results for CosmosQA, ConTRoL, and Dataset I, alongside qualitative simplification examples. It also illustrates the quality-loss and token-saving trade-off against a fixed-order brute-force heuristic on two Dataset I samples.
C RESULTS ON TOKEN OPTIMIZATION
- Results are presented for CosmosQA, ConTRoL, and Dataset I, with qualitative token-simplification examples also included.The results are identified in Tables 7–10.Source: C RESULTS ON TOKEN OPTIMIZATION · p. 11, Table 7 · p. 11, Table 8 · p. 11
- The token-saving and quality-loss trade-off is compared with brute-force application of all heuristics in a fixed order on two illustrative Dataset I samples.Quality loss is measured as 1−S-Bert cosine similarity, and the x% threshold sets tolerated loss to x% of the brute-force method’s total quality loss.Source: C RESULTS ON TOKEN OPTIMIZATION · p. 11, Table 11 · p. 11
- Table 6 overviews datasets used to evaluate the Token Optimization Module.Source: Table 6 · p. 12