PaperlayerView exact revision on arXiv

Source-linked AI summary

FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance

Lingjiao Chen, Matei Zaharia, James Zou

arXiv:2305.05176v1cs.LGcs.AIcs.CLcs.SE

TL;DR

High-throughput use of LLM APIs can be expensive. FrugalGPT learns query-adaptive cascades that use answer-reliability scores to decide when to consult another model. It matches the best individual LLM API’s performance with up to 98% lower inference cost, or improves performance by up to 4% at the same cost.

  • Problem

    High-throughput LLM use can be expensive, motivating the question of how to maximize task performance within a user-defined inference budget.

    Source: §1 · p. 1, §2 · p. 3

  • Method

    FrugalGPT learns a query-adaptive cascade of LLM APIs, using answer-reliability scores to decide whether to accept a response or query another model.

    Source: §1 · p. 2, §3 · p. 5

  • Results

    Up to 98% inference-cost savings with FrugalGPT preserve the best individual LLM API’s task performance; at the same cost, performance improves by up to 4%.

    Source: §1 · p. 2

  • Takeaways & Limitations

    More affordable LLM services can complement GPT-4 by answering queries it gets wrong, offering opportunities for better task performance.

    Source: §4 · pp. 8, 10

    FrugalGPT requires labeled training examples from the same or a similar distribution as test queries for its cascade to work well.

    Source: §5 · p. 10

Abstract

from arXiv · show

There is a rapidly growing number of large language models (LLMs) that users can query for a fee. We review the cost associated with querying popular LLM APIs, e.g. GPT-4, ChatGPT, J1-Jumbo, and find that these models have heterogeneous pricing structures, with fees that can differ by two orders of magnitude. In particular, using LLMs on large collections of queries and text can be expensive. Motivated by this, we outline and discuss three types of strategies that users can exploit to reduce the inference cost associated with using LLMs: 1) prompt adaptation, 2) LLM approximation, and 3) LLM cascade. As an example, we propose FrugalGPT, a simple yet flexible instantiation of LLM cascade which learns which combinations of LLMs to use for different queries in order to reduce cost and improve accuracy. Our experiments show that FrugalGPT can match the performance of the best individual LLM (e.g. GPT-4) with up to 98% cost reduction or improve the accuracy over GPT-4 by 4% with the same cost. The ideas and findings presented here lay a foundation for using LLMs sustainably and efficiently.

1 Introduction

FrugalGPT proposes a budget-constrained framework for using LLM APIs through prompt adaptation, LLM approximation, and LLM cascades. Its cascade implementation learns query-dependent model combinations to reduce inference costs while matching or improving performance.

Introduction

  • Up to two orders of magnitude separate commercial LLM API costs, making model selection consequential for expensive, high-throughput applications.API charges include prompt-length costs, generation-length costs, and sometimes a fixed per-query fee. Large LLMs also impose substantial environmental and energy costs [WRG+22].Source: §1 · p. 1
  • Cheaper models can perform much worse on difficult tasks [TLI+23], while dependence on one API provider risks service unavailability.For relatively simple tasks, aggregating GPT-J responses can instead offer performance similar to GPT-3 [ANC+22], illustrating the task-dependent cost–quality tradeoff.Source: §1 · p. 1

Our contributions

  • FrugalGPT combines three cost-reduction strategies within a budget-constrained framework: prompt adaptation, LLM approximation, and adaptive LLM cascades.Prompt adaptation seeks effective, often shorter prompts; approximation creates cheaper models that match expensive models on specific tasks. Cascades learn which combinations of LLM APIs to use for different queries.Source: §1 · pp. 1–2
  • Up to 98% inference-cost savings match the best individual LLM API’s downstream performance, while up to 4% performance improvement is achievable at equal cost.On HEADLINES, FrugalGPT reduces inference cost by 98% while exceeding the performance of GPT-4, the best individual LLM.Source: Figure 1 · p. 2, §1 · p. 2

Related Works: Prompt Engineering

  • Unlike prompt engineering that often lengthens prompts with explanations and examples, this paper explores concise prompts to reduce costs.The reviewed techniques include few-shot prompting and chain-of-thought prompting [WWS+22].Source: §1 · p. 2

Related Works: Model Ensemble

  • Conventional ensembles typically require white-box training access and query every model, conflicting with black-box LLM APIs and increasing costs.These requirements distinguish conventional model ensembles from selectively querying available LLM services.Source: §1 · p. 2

Related Works: System Optimization for LLMs

  • System optimization requires access to model internals that many commercial APIs withhold, while increasing LLM size makes retraining expensive.Reviewed approaches include post-training quantization [BHS+22] [YLW+23] [XLS+22], training pipeline parallelism [LZG+21], and hardware-aware pruning [KFA23].Source: §1 · pp. 2–3

Related Works: ML-as-a-Service

  • Existing classification API-selection methods [CZZ20] [CZZ22] require known labels, whereas natural-language outputs and coupled prompt/API choices give LLM applications a larger optimization space.Earlier model cascades likewise target predictive tasks with known label sets rather than LLMs’ full capabilities.Source: §1 · pp. 1, 3

2 Scope and Problem Statement

The paper studies natural-language query answering through LLM APIs, seeking to maximize task performance under a user-defined average-cost budget. Strategies can vary prompts, API choices, and response aggregation.

Natural language query answering

  • The task is to answer queries sampled from a natural-language query distribution, encompassing news classification, reading comprehension, and commonsense reasoning.These different natural-language tasks can all be formulated as query-answering problems.Source: §2 · p. 3

LLM marketplace

  • The marketplace contains K LLM APIs that generate answers from prompts, with costs based on prompt length, answer length, and sometimes a fixed per-query charge.Each query must first be converted into a corresponding prompt.Source: §2 · p. 3

An illustrative example

  • Approximately $21.2K per month is the illustrative GPT-4 customer-service cost for a small business serving 15,000 customers, adapted from.The example assumes 360,000 monthly queries, each averaging 1,800 prompt tokens and 80 answer tokens. Its GPT-4 rates are $0.03 per thousand input tokens and $0.06 per thousand response tokens.Source: §2 · p. 3

Problem statement: budget-aware LLM API usage

  • The objective is to maximize expected task performance while keeping average query-processing cost at or below a user-defined budget b.A reward measures agreement with the correct answer; strategies may choose prompts, select LLM APIs, and aggregate their responses.Source: §2 · p. 3

3 How to Use LLMs Affordably and Accurately

Prompt adaptation, LLM approximation, and LLM cascades offer three strategies for reducing API costs. Cascades use answer reliability to decide when to stop querying models, with model choices and stopping thresholds optimized under a budget.

Strategy 1: Prompt adaptation

  • Prompt adaptation lowers costs by retaining fewer in-context examples or combining multiple queries so they share one prompt.Query cost grows linearly with prompt size, making example selection a search for shorter prompts without compromising performance. Query concatenation avoids repeated prompt processing and requires explicit instructions to answer multiple queries.Source: §3 · p. 5

Strategy 2: LLM approximation

  • Completion caching approximates an expensive API by reusing stored responses to similar queries and invoking the API only on cache misses.Savings are substantial when similar queries recur frequently.Source: §3 · p. 5
  • Model fine-tuning uses responses from an expensive, powerful LLM to train a smaller, cheaper model for answering new queries.The fine-tuned model often needs shorter prompts, providing latency improvements alongside cost savings.Source: §3 · p. 5

Strategy 3: LLM cascade

  • LLM cascades query APIs sequentially and stop when an answer’s reliability score exceeds its threshold, reducing costs when inexpensive early models answer reliably.A router selects the API list, while a scoring function evaluates each query–answer pair. The scorer can be a regression model trained to predict answer correctness.Source: §3 · p. 5
  • Cascade training selects APIs and stopping thresholds to optimize answer quality under an average-cost budget, using search-space pruning and objective interpolation.The mixed-integer optimization is computationally expensive. The specialized optimizer discards API lists with little answer disagreement and approximates the objective by interpolating a few samples.Source: §3 · pp. 5–6

Compositions

  • Combining strategies can further reduce query costs and improve performance, but increases training computation.Examples include jointly selecting prompts and LLMs to achieve satisfactory performance, or selecting among existing APIs and fine-tuned models. These combinations introduce trade-offs among query cost, task performance, and computational cost.Source: §3 · p. 6

4 LLM Cascade Reduces Cost and Improves Accuracy

FrugalGPT matches the best individual LLM’s performance with 50%–98% cost savings and can improve accuracy while reducing costs. Its cascade exploits complementary model strengths, reserving expensive models for difficult queries.

Setups: LLM APIs, Tasks, Datasets, and FrugalGPT instances

  • 12 LLM APIs from five providers were evaluated using three-stage cascades on HEADLINES [SK21], OVERRULING, and adapted COQA [RCM19].The tasks cover gold-price trends from financial headlines, legal overruling detection, and conversational reading comprehension adapted to direct question answering. Each dataset was randomly split into training data for cascade learning and test data for evaluation.Source: §4 · p. 6

A Case Study

  • 80% lower cost and 1.5% higher accuracy than GPT-4 were achieved by the learned HEADLINES cascade.Under a $6.5 budget, regression-tailored DistilBERT [SDCW19] scores answers along a GPT-J, J1-L, GPT-4 cascade. GPT-J answers are accepted above 0.96 and J1-L answers above 0.37; otherwise, the cascade advances.Source: §4 · pp. 6, 8, Figure 3 · p. 7

LLM diversity

  • 13% of COQA examples receive correct GPT-3 answers despite GPT-4 errors, demonstrating complementary strengths that could improve performance.On HEADLINES, GPT-C, GPT-J, and J1-L each offer up to 6% potential improvement over GPT-4. Maximum performance improvement measures the probability that one model answers correctly and another incorrectly; these upper bounds need not be attainable.Source: §4 · p. 8

Cost Savings

  • 50%–98% cost savings let FrugalGPT match the best individual LLM’s performance by routing easier queries to cheaper models.Expensive models such as GPT-4 are invoked only for queries FrugalGPT identifies as challenging.Source: Table 3 · p. 8, §4 · p. 8

Performance and Cost Trade-offs

  • FrugalGPT enables performance-cost trade-offs across all evaluated datasets; on OVERRULING, it achieves a 1% accuracy gain while reducing costs by 73% compared with the best LLM API, GPT-4.At the same cost, FrugalGPT improves accuracy by up to 5%. In one example, uncertainty about early answers leads it to query every LLM despite identical answers; avoiding such cases remains an open problem.Source: Figure 5 · p. 9, §4 · p. 10
  • API cost rankings vary by dataset, and higher prices do not guarantee better performance: J1 costs more yet underperforms GPT-3 on HEADLINES.J1 is second-most expensive on HEADLINES, whereas GPT-3 holds that position on OVERRULING and COQA. J1 charges for generated tokens but not input tokens, while GPT-3 charges for both.Source: §4 · p. 10

5 Discussions, Limitations and Future Prospects

FrugalGPT reduces inference costs while preserving cutting-edge LLM performance, but training its cascade requires labeled examples, similar training and test distributions, and upfront resources. Future directions include broader optimization criteria and uncertainty quantification for risk-critical applications.

5 Discussions, Limitations and Future Prospects

  • Up to 98% cost reduction is achievable with FrugalGPT while preserving the performance of cutting-edge LLMs.FrugalGPT illustrates the LLM cascade strategy for reducing LLM API inference costs.Source: §5 · p. 10
  • FrugalGPT requires labeled training examples from the same or similar distribution as test examples for its cascade to work well.Learning the cascade incurs an upfront resource cost, which is beneficial when the final query dataset exceeds the cascade-training dataset.Source: §5 · p. 10
  • The paper establishes a foundation rather than a comprehensive or definitive solution, leaving other cost-saving strategies, such as faster attention computation, unexplored.Source: §5 · p. 10
  • Future optimization should incorporate latency, fairness, privacy, and environmental impact while maintaining performance and cost-effectiveness.Addressing the environmental effects of training and deploying LLMs requires joint efforts from users and API providers.Source: §5 · p. 10
  • Risk-critical applications require careful quantification of uncertainty in LLM-generated outputs.Source: §5 · p. 10

Related papers

Loading 2305.05176v1…