PaperlayerView exact revision on arXiv

Source-linked AI summary

Fly-Swat or Cannon? Cost-Effective Language Model Choice via Meta-Modeling

Marija Šakota, Maxime Peyrard, Robert West

arXiv:2308.06077v3cs.CL

TL;DR

Large LMs can improve performance but are costly, and inputs vary in which model can answer them satisfactorily. FORC predicts cost and performance to assign each input to a candidate LM; across 14 datasets, it matched the largest LM’s performance at a 62.79% cost reduction.

  • Problem

    Large language models can be expensive to run, and choosing the cheapest model that can solve an input requires predicting model performance without querying every LM on that input.

    Source: §1 · p. 1

  • Method

    FORC uses a trained meta-model to predict candidate-LM performance and combines those predictions with cost estimates to assign each query to at most one LM.

    Source: §1 · p. 2

  • Results

    62.79% lower costs across 14 datasets while maintaining the same performance as the biggest LM in the pool.

    Source: §6 · p. 8

  • Takeaways & Limitations

    The authors release a library for using and further developing FORC.

    Source: §6 · p. 8

    The meta-model may become stale as new LMs appear or existing LMs change behavior, so it should be retrained to account for updates.

    Source: §5.2 · p. 7

Abstract

from arXiv · show

Generative language models (LMs) have become omnipresent across data science. For a wide variety of tasks, inputs can be phrased as natural language prompts for an LM, from whose output the solution can then be extracted. LM performance has consistently been increasing with model size - but so has the monetary cost of querying the ever larger models. Importantly, however, not all inputs are equally hard: some require larger LMs for obtaining a satisfactory solution, whereas for others smaller LMs suffice. Based on this fact, we design a framework for cost-effective language model choice, called "Fly-swat or cannon" (FORC). Given a set of inputs and a set of candidate LMs, FORC judiciously assigns each input to an LM predicted to do well on the input according to a so-called meta-model, aiming to achieve high overall performance at low cost. The cost-performance tradeoff can be flexibly tuned by the user. Options include, among others, maximizing total expected performance (or the number of processed inputs) while staying within a given cost budget, or minimizing total cost while processing all inputs. We evaluate FORC on 14 datasets covering five natural language tasks, using four candidate LMs of vastly different size and cost. With FORC, we match the performance of the largest available LM while achieving a cost reduction of 63%. Via our publicly available library, researchers as well as practitioners can thus save large amounts of money without sacrificing performance.

1 INTRODUCTION

Larger language models can improve performance but cost more, while simpler inputs may not need their capabilities. FORC predicts model performance and cost before assigning queries, and the paper reports matching the largest model’s performance at 63% lower cost.

1 INTRODUCTION

  • The framework addresses the rising cost of large LMs and the need to predict which model can solve each input without querying every LM first.The paper notes that 33% of samples were solved by both the biggest and at least one smaller model, while 11% were solved only by smaller models.Source: §1 · pp. 1–2
  • FORC predicts each candidate LM’s cost and performance for each query, then assigns each query to at most one LM using a selectable optimization strategy.Cost estimates use provider API pricing; performance predictions come from a regression meta-model trained on existing query-performance pairs.Source: §1 · p. 2
  • FORC offers multiple cost-performance strategies and can handle inputs without knowing their task or dataset, including tasks absent from meta-model training.The authors also release a library for using the framework and training tailored meta-models.Source: §1 · p. 2
  • On 14 datasets, FORC reduced test-set costs by 63% while maintaining the biggest LM’s performance.The authors also report releasing FORC as open-source code.Source: §1 · p. 2

2 BACKGROUND AND RELATED WORK

Prior evaluations show that smaller models can match or outperform larger ones on some tasks, motivating a way to choose models by input. Related cost-reduction work includes inference techniques and LM cascades, whereas this paper describes a pre-query, dataset-general approach.

2.1 Language-model evaluation

  • Smaller LMs can match larger ones on some tasks, while larger models remain useful for more complex tasks; evaluation efforts such as HELM compare models across tasks and performance aspects.The cited Inverse Scaling Prize findings also include tasks where larger LMs perform worse than smaller counterparts.Source: §2.1 · p. 2

2.2 Inference-cost optimization

  • Prior inference-cost work includes quantization, distillation, and pruning, while Chen et al.’s LM cascade queries models sequentially and is dataset-specialized.In contrast, this work selects an LM before querying and develops a framework that does not require retraining its meta-model separately for each dataset.Source: §2.2 · pp. 2–3

3 METHOD

FORC assigns each query to at most one language model while balancing predicted performance and estimated cost against user requirements. Its framework combines a meta-model and cost estimates with assignment strategies ranging from per-query choices to batch-level optimization.

3.1 Problem setting

  • The setting assumes a set of queries, candidate LMs with user-defined costs, and a shared evaluation metric; the goal is to assign each query to at most one LM within cost–performance requirements.Queries may represent tasks such as question-answering, reasoning, or summarization.Source: §3.1 · p. 3

3.2 Framework setting

  • FORC predicts each LM’s performance on each query with a trained meta-model and estimates query costs before applying the user’s chosen assignment strategy.The meta-model takes a query and an LM token as input, and can be tailored to different LM pools, datasets, or metrics.Source: §3.2 · p. 3
  • Cost-insensitive strategies treat queries independently, using either one fixed LM, the model with the highest predicted performance, or thresholding to choose the cheapest model predicted to solve each query.For thresholding, if no model meets the threshold, the strategy can choose either the smallest or the largest LM.Source: §3.2 · p. 3
  • The cost-oriented ILP maximizes predicted performance across samples under a total-cost constraint, with assignment variables allowing samples to remain unassigned.Each assignment is associated with estimated cost and predicted performance.Source: §3.2 · pp. 3–4
  • The performance-oriented ILP minimizes cost while meeting the user’s minimum-performance requirement; with binarized performance, it minimizes cost to solve at least the required number of samples.Each sample is assigned to at most one LM.Source: §3.2 · p. 4
  • The greedy strategy processes samples sequentially, choosing the LM with the highest predicted performance until the cost limit is reached; remaining samples stay unassigned.In the experiments, unassigned samples count as incorrect under accuracy and incur zero cost.Source: §3.2 · p. 4

4 EXPERIMENTS

The experiments evaluate a meta-model that predicts language-model performance and cost-aware strategies for assigning queries to candidate LMs. The cost-oriented ILP strategy matches text-davinci-002’s accuracy at substantially lower cost, while meta-model performance varies with task and model complexity.

4.1 Meta-model evaluation

  • Using HELM raw runs across binary-metric tasks, the 66M-parameter DistilBERT meta-model estimates each LM’s query performance from query-plus-LM-token inputs.The evaluated task metrics include exact match, quasi-exact match, and mathematical equivalence.Source: §4.1 · p. 4
  • Meta-model performance was good across metrics but only modestly exceeded a dataset-aware dummy; complex tasks and larger LMs were harder to predict.The dummy classifier requires knowing the dataset, while the meta-model does not.Source: §4.1 · p. 5, Table 3 · p. 6
  • The meta-model was well calibrated: its calibration curve lay close to the diagonal for perfect calibration.The plots group output probabilities into equal-width bins and compare each bin’s mean probability with its fraction of positive samples.Source: §4.1 · p. 5, Figure 2 · p. 6
  • Leave-one-dataset-out evaluation found that performance was generally a little lower than with the full testing dataset, while the meta-model still generalized to the held-out dataset.Each Table 4 row trains on all datasets except the named one and tests on that dataset.Source: §4.1 · p. 5, Table 4 · p. 7

4.2 Framework evaluation

  • 62.1% absolute cost reduction with thresholding and 11.5% with performance maximization accompanied accuracy comparable to text-davinci-002.Both reductions are relative to text-davinci-002.Source: §4.2 · p. 5, Figure 3 · p. 7
  • 37.21% of text-davinci-002’s cost matched its accuracy with cost-oriented ILP; the greedy strategy needed 88.57% of its budget for the same result.The cost–accuracy plot compares assignment strategies and reports 95% confidence intervals.Source: §4.2 · p. 6, Figure 3 · p. 7
  • For the cost-oriented ILP strategy, matching text-davinci-002’s performance cost 71.76% as much on QA, 83.41% on reasoning, and 19.15% on text classification.These percentages are the price shares reported for each task.Source: §4.2 · p. 7
  • On toxicity detection, text-ada-001 gained 19.87% accuracy at 1.99% of text-davinci-002’s budget; on CivilComments, cost-oriented ILP gained 20.1% absolute accuracy at 1.99% of its cost.The CivilComments result essentially matched that dataset’s oracle.Source: §4.2 · p. 7

5 DISCUSSION

The discussion extends FORC to customizable objectives and prompt selection, then notes that its meta-model and cost estimates depend on current model behavior and prices.

5.1 Further use cases of the FORC framework

  • FORC supports user-defined combinations of cost and performance objectives and constraints beyond its cost- or performance-oriented ILP strategies.Users can tailor the framework to practical applications.Source: §5.1 · p. 7
  • With a fixed LM, FORC can compare prompt phrasings without querying the LM, but the meta-model should be retrained because small query changes were absent from training.Its behavior on such changes is unknown.Source: §5.1 · p. 7

5.2 Practical considerations

  • New LMs and behavioral shifts can make the meta-model stale, so retraining should account for model updates and additions.The authors say retraining is cheap and takes little time.Source: §5.2 · p. 7
  • Figure 4 and Figure 5 plot test-set accuracy against average US$ cost per query by dataset and task, respectively, with 95% confidence intervals.Source: Figure 4 · p. 8, Figure 5 · p. 8
  • OpenAI price changes require updating cost parameters because they affect cost–accuracy plots and sample-to-LM assignments.Source: §5.2 · p. 8

6 CONCLUSION

The paper addresses rising LM execution costs with FORC, an automated framework for selecting an LM per query. Across 14 datasets, it reports 62.79% lower costs while matching the biggest LM’s performance.

  • 62.79% lower costs while maintaining the same performance as the biggest LM in the pool across 14 datasets.Source: §6 · p. 8
  • FORC automatically decides which LM is the best option for a given input query.Source: §6 · p. 8
  • Costs were substantially cut on all task types and most evaluated datasets, and the authors released a library for using and developing the framework.Source: §6 · p. 8
Loading 2308.06077v3…