Source-linked AI summary
FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance
Lingjiao Chen, Matei Zaharia, James Zou
TL;DR
Using LLMs at scale is costly because API prices vary widely and high-throughput applications can incur substantial expense. The paper proposes three cost-reduction strategies and instantiates adaptive LLM cascading in FrugalGPT. FrugalGPT matches the best individual LLM with up to 98% lower inference cost or improves performance by up to 4% at the same cost.
Problem
High-throughput LLM use is expensive, while heterogeneous API costs and quality make budget-constrained model selection difficult.
Method
The paper outlines prompt adaptation, LLM approximation, and LLM cascade, then implements FrugalGPT as an adaptive cascade that routes queries among LLM combinations.
Results
FrugalGPT saves up to 98% of the best individual LLM API’s inference cost while matching its performance, or improves performance by up to 4% at the same cost.
Takeaways & Limitations
FrugalGPT lays groundwork for optimizing LLM task performance under budget constraints while reducing inference costs.
Takeaways & Limitations
Effective cascade training requires labeled examples from the same or a similar distribution as test data, and cascade learning incurs an upfront resource cost.
Abstract
from arXiv · showhide
There is a rapidly growing number of large language models (LLMs) that users can query for a fee. We review the cost associated with querying popular LLM APIs, e.g. GPT-4, ChatGPT, J1-Jumbo, and find that these models have heterogeneous pricing structures, with fees that can differ by two orders of magnitude. In particular, using LLMs on large collections of queries and text can be expensive. Motivated by this, we outline and discuss three types of strategies that users can exploit to reduce the inference cost associated with using LLMs: 1) prompt adaptation, 2) LLM approximation, and 3) LLM cascade. As an example, we propose FrugalGPT, a simple yet flexible instantiation of LLM cascade which learns which combinations of LLMs to use for different queries in order to reduce cost and improve accuracy. Our experiments show that FrugalGPT can match the performance of the best individual LLM (e.g. GPT-4) with up to 98% cost reduction or improve the accuracy over GPT-4 by 4% with the same cost. The ideas and findings presented here lay a foundation for using LLMs sustainably and efficiently.
1 Introduction
Rapidly growing LLM API usage creates substantial financial and environmental costs, while heterogeneous prices and capabilities make affordable, accurate model selection difficult. FrugalGPT addresses this challenge through prompt adaptation, LLM approximation, and adaptive LLM cascades.
- Motivation: LLM APIs can be expensive for high-throughput applications and impose environmental and energy costs.The paper cites estimated operating costs exceeding $700,000 per day for ChatGPT and $21,000 per month for GPT-4 customer service.
- Motivation: API prices can differ by up to two orders of magnitude across commercial LLMs.The prompt cost for 10M tokens is $30 for GPT-4 versus $0.2 for GPT-J hosted by Textsynth.
- Problem: Heterogeneous cost and quality make it challenging to leverage all available LLM options affordably and accurately.Existing model cascades and FrugalML target predictive tasks with known labels and do not cover LLMs’ full capabilities.
- Contribution: FrugalGPT is a flexible framework for processing natural-language queries with LLM APIs under a budget constraint.The paper presents it as a vision for budget-aware LLM API usage.
- Results: 98% cost reduction was achieved while exceeding GPT-4’s performance on HEADLINES.The figure describes HEADLINES as a financial news dataset and compares FrugalGPT with the best individual LLM, GPT-4.
- Contribution: The paper discusses prompt adaptation, LLM approximation, and LLM cascade as three cost-reduction strategies.These strategies respectively target shorter effective prompts, cheaper approximations, and adaptive API selection.
2 Scope and Problem Statement
The paper studies natural-language query answering through a marketplace of K black-box LLM APIs with different costs. It formulates API usage as maximizing task performance subject to an average budget constraint.
- Scope: Natural-language query answering covers tasks such as news classification, reading comprehension, and commonsense reasoning.The objective is to answer a query sampled from a natural-language query distribution.
- LLM Marketplace: The LLM marketplace comprises K different APIs, each mapping prompts to generated answers.Queries must first be converted into prompts before API invocation.
- LLM Marketplace: API cost combines prompt length, generated-answer length, and sometimes a fixed per-query charge.The formal cost function weights these three components with API-specific constants.
- Illustrative Example: $21.2K per month is the estimated GPT-4 cost for a small business handling 360,000 queries.The estimate assumes 1,800-token prompts, 80-token answers, and GPT-4 input and response prices of $0.03 and $0.06 per thousand tokens.
- Problem Statement: The optimization maximizes expected task performance while keeping average processing cost at or below a user-defined budget b.The strategy search includes prompt choices, API selection, and response aggregation.
3 How to Use LLMs Affordably and Accurately
The paper presents three strategies for using LLM APIs within a budget: prompt adaptation, LLM approximation, and LLM cascade. These strategies reduce cost through shorter or shared prompts, cheaper substitutes, caching, fine-tuning, and adaptive model selection.
- Prompt adaptation: Prompt adaptation reduces inference cost by shortening prompts or sharing one prompt across multiple queries.Prompt selection retains a smaller subset of examples, while query concatenation sends one prompt for multiple queries.
- LLM approximation: LLM approximation replaces repeated calls to expensive APIs with cached responses or cheaper fine-tuned models.Completion caching reuses answers for similar queries, while fine-tuning transfers responses from an expensive model to a smaller model.
- LLM cascade: LLM cascade sequentially routes each query through selected APIs with heterogeneous strengths, costs, and performance.A generation scoring function evaluates answers, and an LLM router selects the API list and determines when to stop.
- LLM cascade: The cascade router is optimized under an average-cost budget while maximizing the quality of generated answers.The specialized optimizer prunes API lists with small answer disagreement and approximates the objective using a few samples.
- Compositions: Combining strategies can further reduce cost by jointly selecting prompts and models or searching across APIs and fine-tuned models.Joint prompt and LLM selection seeks the smallest prompt and most affordable model that achieves satisfactory task performance.
4 LLM Cascade Reduces Cost and Improves Accuracy
FrugalGPT learns adaptive cascades that route queries among LLM APIs, achieving substantial cost savings while matching or improving the best model’s accuracy. Across datasets, its benefits arise from heterogeneous API costs and complementary model errors.
- Experimental setup: FrugalGPT was evaluated on HEADLINES, OVERRULING, and COQA using 12 APIs from five providers.Each dataset was split into training and test sets for learning and evaluating the cascade.
- Cascade design: On HEADLINES, FrugalGPT sequentially queries GPT-J, J1-L, and GPT-4, accepting earlier answers when their reliability scores exceed learned thresholds.The thresholds are 0.96 for GPT-J and 0.37 for J1-L; otherwise GPT-4 is invoked.
- HEADLINES case study: 80% cost reduction and 1.5% higher accuracy were achieved on HEADLINES versus GPT-4 under a $6.5 budget.The budget was one-fifth of GPT-4’s cost.
- Model complementarity: Cheap LLMs can complement GPT-4: on HEADLINES, GPT-C, GPT-J, and J1-L each could improve GPT-4’s performance by up to 6%.MPI measures cases where one model is correct while the comparison model is wrong, so this is an improvement upper bound.
- Cost savings: FrugalGPT’s cost savings ranged from 50% to 98% while matching the best individual LLM’s performance.The cascade invokes smaller models for queries they can answer accurately and reserves expensive models for difficult queries.
- Performance-cost trade-offs: Across evaluated datasets, FrugalGPT enabled smooth accuracy-cost trade-offs and achieved a 1% accuracy gain with 73% lower cost than GPT-4 on OVERRULING.The paper attributes this example to integrating knowledge from multiple LLMs.
5 Discussions, Limitations and Future Prospects
FrugalGPT demonstrates that LLM cascade can substantially reduce inference costs while preserving the performance of cutting-edge LLMs. Its broader use is constrained by training-data requirements, upfront cascade-learning costs, and factors beyond cost and performance.
- Up to 98% cost reduction preserves the performance of cutting-edge LLMs.
- FrugalGPT’s cascade requires labeled examples from the same or a similar distribution as test queries.
- Cascade training introduces a one-time upfront resource cost that is advantageous when the final query dataset exceeds the training data.
- Future optimization should consider latency, fairness, privacy, environmental impact, and uncertainty alongside performance and cost.