Source-linked AI summary
MetaLLM: A High-performant and Cost-efficient Dynamic Framework for Wrapping LLMs
Quang H. Nguyen, Thinh Dao, Duy C. Hoang, Juliette Decugis, Saurav Manchanda, Nitesh V. Chawla, Khoa D. Doan
TL;DR
LLM capabilities and costs vary, making it difficult to choose a model that is both accurate and cost-effective for each query. MetaLLM uses a multi-armed bandit framework to route queries among LLMs. It reports accuracy gains alongside API cost savings, with evaluation focused on zero-shot classification.
Problem
LLMs differ in capabilities and costs, and no single model consistently performs best across tasks, making cost-effective model selection challenging.
Source: §1 · pp. 1–2Method
MetaLLM frames per-query LLM selection as a multi-armed bandit problem, using a reward that balances answer performance and usage cost.
Source: §1 · p. 2, §3.3 · p. 5Results
MetaLLM improves the best API’s accuracy by around 1% while reducing costs for zero-shot classification and multiple-choice question answering by up to 60%.
Source: §5 · p. 10Takeaways & Limitations
A reward function tailored to application needs makes the framework versatile for practical use cases.
Source: §3.3 · p. 5, §5 · p. 10The evaluated framework is limited to zero-shot text classification; extending it to other tasks requires task-appropriate reward design, left for future work.
Source: §6.1 · p. 10
Abstract
from arXiv · showhide
The rapid progress in machine learning (ML) has brought forth many large language models (LLMs) that excel in various tasks and areas. These LLMs come with different abilities and costs in terms of computation or pricing. Since the demand for each query can vary, e.g., because of the queried domain or its complexity, defaulting to one LLM in an application is not usually the best choice, whether it is the biggest, priciest, or even the one with the best average test performance. Consequently, picking the right LLM that is both accurate and cost-effective for an application is necessary yet remains a challenge. In this paper, we introduce MetaLLM, a framework that dynamically and intelligently routes each query to the optimal LLM (among several available LLMs) for classification and multi-choice question-answering tasks, achieving significantly improved accuracy and cost-effectiveness. By framing the selection problem as a multi-armed bandit, MetaLLM balances prediction accuracy and cost efficiency under uncertainty. Our experiments, conducted on popular LLM platforms such as OpenAI and Together AI, as well as open-source LLM, showcase MetaLLM's efficacy in real-world scenarios, laying the groundwork for future extensions.
1 Introduction
Because LLM capabilities and costs vary across queries, choosing a model that balances accuracy and cost is challenging. MetaLLM frames query routing as a multi-armed bandit problem and reports improved accuracy with lower API costs, while the paper’s evaluation scope is limited.
1 Introduction
- LLMs vary in capabilities and costs, so even the largest or most expensive model may not perform best across all tasks.This makes identifying a suitable model under computational or API-price constraints challenging.Source: §1 · p. 1
- Ensembling can raise service costs, cascading still queries models sequentially, and defaulting to one model may miss cheaper models that answer correctly.Different LLMs can perform differently across tasks and data distributions.Source: §1 · p. 2
- MetaLLM wraps a set of LLMs and uses a multi-armed bandit to route each query toward a low-cost model likely to answer correctly.It treats model selection as decision-making under uncertainty, with a reward trading off overall performance and usage cost.Source: §1 · p. 2
- The proposed routing algorithm makes its decision without querying any LLMs.The framework is proposed for zero-shot text classification.Source: §1 · p. 2
- MetaLLM improves the best model’s accuracy by around 1% while saving up to 60% and 10% of total price on OpenAI and Together AI APIs, respectively.The experiments cover benchmark datasets, closed- and open-source LLMs, and popular API services.Source: §1 · p. 2
- The work focuses on zero-shot classification and multiple-choice question answering, while extension to other language tasks is left for future work.Such extensions would modify the reward function to incorporate suitable response-quality metrics.Source: §1 · p. 2
2 Related Works
Related work describes the expanding range of LLM capabilities and costs, then surveys approaches to improving or selecting models. These approaches include model-internal routing, ensembles and cascades, and routers that select a single LLM.
Large Language Models
- LLM services offer models with diverse capabilities and heterogeneous cost structures, creating a choice burden for users.The passage gives Together AI token-processing prices for Gemma, Llama, Mistral, and Qwen as examples.Source: §2 · pp. 2–3
Prompt Optimization and Mixture of Experts
- Mixture-of-Experts trains a router within a model, while prompting methods such as Chain-of-Thought and Tree of Thoughts seek to improve performance without additional training.The passage notes that these approaches do not exploit the broader set of available LLMs, some of which may be cheaper.Source: §2 · p. 3
Model Ensemble
- Ensembling combines model strengths, while cascades query models sequentially until a sufficiently confident response is obtained.The passage describes FrugalML and LLM cascades using prompt engineering, caching, and cascading techniques.Source: §2 · p. 3
Model Selection
- Single-LLM routers select models using predicted performance, optimization, reward-model preferences, or query difficulty, with some approaches carrying cost or model-count constraints.The cited examples include a router with high router cost and another applicable only when two LLMs are available.Source: §2 · p. 3
Commercialized Foundation Model Services
- Commercial services provide access to models for tasks including text generation, image generation, and multimodal applications.The passage names OpenAI models and Together AI offerings including Llama, Gemma, Mistral, Qwen, and Flux.Source: §2 · p. 3
3 MetaLLM Framework
The framework routes queries among APIs with heterogeneous capabilities and costs, formalizes a budgeted performance objective, and uses a bandit to make routing decisions under uncertainty.
3.1 Preliminaries
- For zero-shot classification, an API is capable on a query when its answer matches the ground-truth label, and routing balances response quality against API cost.The objective can select the least expensive API among those that give good responses.Source: §3.1 · pp. 3–4
3.2 The optimal solution of routing objective
- The routing objective seeks the highest performance under a usage budget; with an exact accuracy matrix, the relaxed optimization can identify an optimal API assignment.The formulation relaxes binary routing choices and uses a dual problem; an exact accuracy matrix permits optimizing the assignment directly.Source: §3.2 · pp. 4–5
3.3 The Proposed MetaLLM
- Earlier API-performance predictors cascade models until confidence is high, which becomes expensive when query volume is substantial.MetaLLM instead frames routing as a multi-armed bandit problem.Source: §3.3 · p. 5
- MetaLLM treats each LLM as a bandit arm, learns a reward model from training-query performance minus cost, then uses UCB to select and update chosen arms.The method assumes training queries with performance available for each LLM and uses a linear reward model.Source: §3.3 · pp. 5–6
4 Experiments
The experiments evaluate MetaLLM across OpenAI and Together AI models on classification and multiple-choice question answering. They examine its cost-accuracy performance, model heterogeneity, reward-cost choices, and cost-scaling parameter.
4.1 Experimental Setup
- The experiments use four OpenAI models and four Together AI models, evaluating SST-2 sentiment classification and MMLU multiple-choice questions.The Together AI models are Gemma, Llama, Mistral, and Qwen.Source: §4.1 · p. 6
4.2 Performance of MetaLLM on OpenAI Models
- MetaLLM achieves higher performance than defaulting to text-davinci-002 while spending 60% less.The comparison uses the same budget as each default model; Table 3 reports OpenAI model costs and accuracy.Source: Table 3 · p. 7, §4.2 · p. 7
4.1 Experimental Setup
- MetaLLM maps Sentence-BERT query embeddings to expected rewards and is evaluated on test accuracy and average cost per 10,000 queries.Training and inference comparisons include training-only optimization, online updates, and initialization from training data followed by online updates.Source: §4.1 · p. 7
4.3 The Heterogeneous Capabilities of Different LLMs
- Figure 2’s pairwise counts show that cheaper models can answer queries that more expensive models miss, enabling query-specific routing to exploit heterogeneous capabilities.The reported examples include text-ada-001 and text-babbage-001 answering samples missed by text-curie-001 and text-davinci-002.Source: §4.3 · p. 8, Figure 2 · p. 8
4.4 Reward Function With Dynamic Cost
- Using exact per-query costs does not significantly improve results over fixed costs and can lower accuracy at high budgets.The authors hypothesize that extra penalties on long, more complicated queries encourage routing to cheaper models that may not solve them effectively; they recommend fixed costs.Source: §4.4 · p. 8, Figure 3 · p. 8
4.5 Performance of MetaLLM on Together AI APIs
- On MMLU, MetaLLM with p = 0 achieves 80.90% accuracy; a positive p substantially lowers cost with minor accuracy degradation.MetaLLM can outperform Llama at lower cost.Source: §4.5 · p. 9, Table 4 · p. 9
4.6 Analysis of MetaLLM’s Cost Scaling Parameter
- At p = 0, optimizing only for performance on SST-2, MetaLLM selects text-babbage-001 more often than text-curie-001 or text-davinci-002; frequencies are shown separately for correct and incorrect predictions.At p = 0.001, MetaLLM rarely picks text-davinci-002 while achieving 90.98% accuracy.Source: §4.6 · p. 9
5 Conclusion
MetaLLM dynamically selects an LLM to balance performance and cost efficiency. The paper reports improved accuracy alongside substantial cost reductions for zero-shot classification and multiple-choice question answering.
5 Conclusion
- Around 1% higher accuracy than the best API, while reducing costs by up to 60% for zero-shot classification and multiple-choice question answering.The conclusion presents MetaLLM as a lightweight approach applicable to sets of off-the-shelf LLMs.Source: §5 · p. 10
6 Limitations and Societal Impacts
The paper’s evaluation focuses on zero-shot text classification, while broader task coverage and richer routing objectives remain open directions. It also frames cost reduction through cheaper-model routing as a potential practical benefit.
6.1 Limitations
- Zero-shot text classification is the studied scope, partly because correct outputs and corresponding rewards are straightforward to determine.Source: §6.1 · p. 10
- Extending the framework to tasks such as question answering or text generation requires suitable response-quality metrics, whose design the authors leave to future work.Source: §6.1 · p. 10
- MetaLLM trains a simple linear model on extracted query features, which may miss finer-grained features.Source: §6.1 · p. 10
- The reward optimizes performance and API cost, while inference time, robustness, emergent abilities, and training-distribution information are possible additional factors.Source: §6.1 · p. 10
6.2 Societal Impacts
- The authors say routing to cheaper models that may answer correctly can substantially reduce query costs and may improve performance by combining multiple LLMs.Source: §6.2 · p. 10