Source-linked AI summary
MixLLM: Dynamic Routing in Mixed Large Language Models
Xinyuan Wang, Yanchi Liu, Wei Cheng, Xujiang Zhao, Zhengzhang Chen, Wenchao Yu, Yanjie Fu, Haifeng Chen
TL;DR
Mixed LLM deployments require routing each query among models with different strengths while balancing quality, cost, latency, continual adaptation, and changing candidate sets. MixLLM addresses this with tag-enhanced embeddings, LLM-specific predictors, a meta-decision maker, and continual training. It achieves 97.25% of GPT-4’s quality at 24.18% of the cost under the reported time constraint.
Problem
LLM routing must balance response quality, cost, and latency while adapting continually to feedback and evolving sets of candidate models.
Method
MixLLM combines tag-enhanced query embeddings, lightweight LLM-specific quality and cost predictors, a trade-off-aware meta-decision maker, and continual training.
Results
97.25% of GPT-4’s quality is achieved at 24.18% of the cost under the reported latency constraint.
Takeaways & Limitations
MixLLM supports dynamic query routing with quality-cost-latency trade-offs, adaptation to evolving queries and feedback, and changing LLM candidate sets.
Takeaways & Limitations
MixLLM assumes access to refined feedback such as response quality and cost, which may not always be available in real-world settings.
Abstract
from arXiv · showhide
Large Language Models (LLMs) exhibit potential artificial generic intelligence recently, however, their usage is costly with high response latency. Given mixed LLMs with their own strengths and weaknesses, LLM routing aims to identify the most suitable model for each query in the stream to maximize response quality and minimize cost and latency. However, the challenges involve: (1) dynamic trade-offs among quality, cost, and latency; (2) enabling continual learning in deployed systems; and (3) navigating a varying (e.g., new LLM addition or old LLM removal) set of LLM candidates over time. To bridge these gaps, we develop MixLLM, a dynamic contextual-bandit-based routing system for query-LLM assignment. Specifically, we first leverage query tags to enhance query embeddings for the routing task. Next, we design lightweight prediction models to estimate the response qualities and costs of queries over LLMs. We then devise a meta-decision maker to choose the query-LLM assignments to best tradeoff response quality, cost, and latency. Finally, the system benefits from continual training, allowing it to adapt to evolving queries and user feedback over time. Our extensive experiments show that MixLLM achieves the best trade-offs in response quality, cost, and latency (97.25% of GPT-4's quality at 24.18% of the cost under the time constraint).
1 Introduction
MixLLM frames LLM routing as selecting among models with different strengths while balancing response quality, cost, and latency. It addresses latency bottlenecks, continual adaptation, changing candidate sets, and feedback-aware deployment through dynamic routing and lightweight model-specific prediction.
- Motivation: LLM routing selects among models with different strengths to balance response quality, cost, and latency.More powerful models can provide higher-quality responses but may impose costly computational requirements.
- Existing Challenges: Existing routing methods include non-predictive cascades and predictive systems that estimate candidate-model performance before selection.Cascades can increase cost and latency because multiple LLMs are involved.
- Existing Challenges: Current methods must address congestion from routing many queries to one LLM, continual learning from evolving queries and feedback, and changing candidate-model sets.The candidate set may require integrating new models and retiring outdated ones.
- MixLLM: MixLLM uses tag-enhanced query embeddings, lightweight LLM-specific quality and cost predictors, a meta-decision maker, and continual training.Its meta-decision maker balances response quality, cost, and latency for query-LLM assignments.
- Results: 97.25% of GPT-4’s quality is achieved at 24.18% of the cost while balancing response quality, cost, and latency.The reported experiments also incorporate latency penalties and extend RouterBench with Llama 3.1, prompt length, and response length.
- Contributions: MixLLM contributions include latency-aware routing for real-world query streams and adaptation to changing environments and user feedback.The system also targets optimal LLM selection and continual operation as candidate models change.
2 Related Work
Prior LLM routing work includes non-predictive cascades and predictive systems that select models using classifications, quality estimates, optimization, or bandit-based decisions. These approaches motivate routing methods that reduce cost while preserving response quality across diverse tasks.
- Overview: LLM routing research spans systems designed to select the most suitable model from mixed candidates.The related work distinguishes non-predictive and predictive routing systems.
- Non-predictive Routing: Non-predictive routing uses smaller LLMs first and switches to larger ones when needed, forming cascades that can reduce cost while maintaining response quality.FrugalGPT and AutoMix are examples of cascading strategies involving multiple LLMs.
- Predictive Routing: Predictive routing estimates response quality before routing each query to one LLM.Predictive systems include classification, quality prediction, optimization, and bandit-based solutions.
- Predictive Routing: Existing predictive methods use domain classifiers, reward models, tag-based label enhancement, similarity ranking, matrix factorization, supervised classification, and prompting classification.The cited systems include ME-Switch, Zooter, and RouteLLM.
3 Methodology
MixLLM routes streaming queries by combining tag-enhanced embeddings, LLM-specific quality and cost prediction, latency-aware meta-decision making, and continual learning. Its design supports quality-cost-latency trade-offs while adapting to feedback and changing candidate models.
- Problem formulation: MixLLM assigns streaming queries to candidate LLMs while trading off response quality, cost, and latency.The framework is designed for sequential queries and a changing candidate set.
- Tag-enhanced query embedding: Domain-aware tags refine query embeddings so routing representations capture distinctions associated with LLM response quality.The method generates fine-grained tags, clusters them into domains, and fine-tunes an encoder using intra-domain similarity and inter-domain separation.
- LLM-specific prediction: LLM-specific predictors estimate each candidate’s response quality and financial cost for a query before assignment.Cost combines known input cost with predicted output cost based on response length and unit prices.
- Meta decision maker: The meta-decision maker combines predicted quality-cost trade-offs, prediction uncertainty, and waiting-time penalties to select a candidate.The latency mechanism penalizes candidates with excessive waiting time, including startup and token-generation delays.
- Meta decision maker: A scaled latency threshold applies penalties before the maximum tolerable waiting time, discouraging selections that could create congestion.The waiting-time penalty uses γ, τ, and ξ·τ, where ξ is smaller than 1.
- Continual learning: Offline and online training update predictive models, uncertainty matrices, and decision mechanisms using refined feedback and selected-model feedback.Online training uses contextual-bandit learning to incorporate binary user feedback and adapt after deployment.
4 Experiments
The experiments evaluate MixLLM on an expanded RouterBench dataset using streaming queries, quality-cost trade-offs, latency constraints, and multiple baseline families. The setup also tests offline routing, budget variation, lightweight predictors, and the distinction between offline and online training.
- Dataset and setup: 36,497 queries from 8 NLP datasets and 11 LLMs form the RouterBench evaluation, extended with Llama 3.1 8B and 70B models.The dataset includes Chinese and English queries, response records, quality and cost metrics, and an 80%/20% train-test split.
- Baselines: MixLLM is compared with cascading, AutoMix, RouteLLM, Zooter, and RouterBench as non-predictive and predictive baselines.The baselines use sequential model escalation or classifier-based routing strategies.
- Evaluation protocol: Baseline comparisons use MixLLM’s offline component because the baseline algorithms lack online training after deployment.The online training component is evaluated separately.
- Metrics and constraints: Routing is evaluated on streaming test queries using total response quality and total cost under a maximum waiting-time constraint.Queries exceeding the maximum tolerable waiting time receive a quality score of 0.
- Metrics and constraints: Budget sensitivity is tested by varying λ from 10^-6 to 10^6, with larger λ prioritizing response quality.The parameter controls the quality-cost trade-off across budget levels.
- Implementation: Lightweight random-forest, MLP, and KNN predictors are used for quality and cost estimation, with an MLP model smaller than 2MB.The predictor choices vary by LLM and target response attribute.
4.2 Overall Results
MixLLM outperforms the baselines on the quality-cost trade-off under latency constraints, remaining stable where several baselines degrade as budgets increase.
- 97.25% of GPT-4’s quality at 24.18% of the cost is achieved by MixLLM when λ is 1.4.
- 96.39% of GPT-4’s quality at 32.94% of the cost is achieved by the best baseline, OptLLM, at the same comparison point.
- MixLLM remains stable beyond the comparison point, while OptLLM and other baselines experience quality drops as queries exceed waiting-time tolerance.
- The Oracle benchmark selects the lowest-cost LLM meeting each query’s quality threshold after testing all candidate LLMs.
- Single LLMs provide individual quality-cost points, with GPT-4 offering higher quality and GPT-3.5 a better cost-quality balance.
- AutoMix exhausts its budget and reaches the latency constraint because multiple LLMs handle each query.
4.3 Study on Continual Training
The continual-training study simulates offline and online query streams and finds that feedback-based online training improves routing performance, although the experiment includes only one online test.
- An 80:20 split assigns 80% of queries to offline training and 20% to online training.
- Online training uses refined feedback plus simulated user feedback based on response quality above 0.7 and waiting time below 15 seconds.
- Both refined and binary feedback improve overall response quality at comparable costs within each split ratio.
- The experiment implements one online test at the end of online training to demonstrate continued learning from and alignment with online feedback.
4.4 Study on Tag-Enhanced Embedding
The study evaluates tag-enhanced embeddings and latency-aware routing, finding consistent gains from tags and stable MixLLM performance without the latency constraint while baselines vary more.
- Tag-enhanced embedding: InsTag generates tags for training queries, including examples such as “data structure” and “legal ethics,” organized into 20 domains.
- Tag-enhanced embedding: Tag-enhanced embedding consistently improves routing performance across cost levels, although its gains diminish as budgets increase.
- Latency constraint: Latency is used as the primary constraint because hardware-resource limits converge on query waiting time.
- Latency constraint: Query latency is simulated from initial generation time and output length divided by each LLM’s generation speed.
- Latency constraint: Without the latency constraint, MixLLM still outperforms the baselines and maintains stable performance, while the baselines show greater variation.
- Latency constraint: At higher cost levels, predictive baselines route more queries to slower, more powerful LLMs, causing some queries to exceed the maximum tolerable waiting time.
4.6 Study on Adaptive Training
MixLLM supports changing LLM candidate sets through independently trained components and remains effective when new Llama 3.1 candidates are added, while online training improves out-of-domain robustness.
- Adaptive training: Each LLM-specific component operates independently, so adding or removing candidates affects the corresponding LLM without complete retraining.
- Adaptive training: 98.55% of GPT-4’s response quality at 16.79% of the cost is achieved after adding Llama 3.1 8B and 70B models when λ is 1.8.
- Out-of-domain generalization: The out-of-domain setup reserves 20% of testing data for domains absent from the 80% training data.
- Out-of-domain generalization: Offline-only training decreases performance by 5.44% at the same price-cost level, while offline-plus-online training limits the decrease to 3.35%.
4.8 Study on Different Choice Policy
The study compares policies that select different numbers of LLMs, revealing a cost-quality trade-off and a practical limitation in producing one definitive answer.
- Choice policies: Top 1, Top 2, and Top 3 select the LLMs with the highest 1, 2, or 3 scores, respectively.
- Choice policies: Selecting more LLMs shifts the quality-cost curve upward and rightward because costs sum while the chance of choosing the strongest model increases.
- Results: At the same budget, Top 3 achieves the highest response quality, surpassing GPT-4 at only 20% of its cost.
- Limitation: MixLLM does not incorporate multi-choice selection because users typically seek one definitive answer and reviewer-based selection adds time and resources.
5 Conclusion
MixLLM dynamically routes queries among LLMs by balancing response quality, cost, and latency. Its tag-enhanced representations, latency constraints, continual learning, and independent model predictions support changing queries and candidate sets while maintaining performance across budgets.
- MixLLM selects the most suitable LLM for each query by balancing response quality, cost, and latency.
- Tag-enhanced query embeddings and latency constraints address key challenges in real-world LLM deployment.
- Continual learning and independent prediction for each LLM allow adaptation as queries evolve and new models are introduced.
- MixLLM optimizes resource usage while maintaining strong performance across varying budget levels.
Limitations
The paper identifies limitations involving feedback availability, unseen domains, multi-output answer selection, system dispatch detail, routing complexity, and validation beyond idealized environments.
- MixLLM assumes refined feedback on response quality and cost, which may not always be available in real-world settings.
- MixLLM may face challenges when routing queries from brand-new domains, known as the out-of-domain problem.
- MixLLM faces challenges when practical scenarios require selecting one definitive answer from multiple LLM outputs.
- More detailed dispatch strategies using system information could improve practicality beyond the current latency constraint.
- Hierarchical routing and robustness in real-world applications remain unexplored or require further testing.