Source-linked AI summary
Hybrid LLM: Cost-Efficient and Quality-Aware Query Routing
Dujian Ding, Ankur Mallick, Chi Wang, Robert Sim, Subhabrata Mukherjee, Victor Ruhle, Laks V. S. Lakshmanan, Ahmed Hassan Awadallah
TL;DR
Large LLMs offer stronger quality but incur high deployment costs, while smaller models are cheaper yet generally weaker. The paper introduces a quality-aware router that sends queries to either model based on difficulty and desired quality, achieving up to 40% cost advantage with no significant quality drop.
Problem
Large LLMs require expensive computation, whereas cheaper smaller models generally provide lower response quality, creating a cost–quality trade-off.
Method
A router estimates query difficulty and model quality gaps, then routes each query to one model while tuning the quality–cost trade-off at test time.
Results
Up to 40% of queries can be routed for cost advantage with no significant drop in response quality.
Takeaways & Limitations
Hybrid quality-aware routing can make cost-efficient model-selection decisions while accommodating different quality requirements.
Takeaways & Limitations
The current routers make decisions using query inputs only; future work identifies task-aware routing and out-of-distribution generalization as extensions.
Abstract
from arXiv · showhide
Large language models (LLMs) excel in most NLP tasks but also require expensive cloud servers for deployment due to their size, while smaller models that can be deployed on lower cost (e.g., edge) devices, tend to lag behind in terms of response quality. Therefore in this work we propose a hybrid inference approach which combines their respective strengths to save cost and maintain quality. Our approach uses a router that assigns queries to the small or large model based on the predicted query difficulty and the desired quality level. The desired quality level can be tuned dynamically at test time to seamlessly trade quality for cost as per the scenario requirements. In experiments our approach allows us to make up to 40% fewer calls to the large model, with no drop in response quality.
1 Introduction
The paper proposes hybrid inference to balance LLM response quality and cost by routing queries between large and small models according to difficulty and quality requirements. Experiments show substantial cost reductions with minimal or no quality loss.
- Motivation: Large models provide stronger response quality but require expensive cloud deployment because of their size and compute demands.Smaller models can run on lower-cost devices, but generally lag in response quality.
- Motivation: Around 20% of queries have small-model responses comparable to or better than those of GPT-3.5-turbo.This quality-gap observation motivates routing easier queries to the small model.
- Approach: The router identifies easy queries using model quality-gap estimates while accounting for response randomness and disparity between models.Its test-time threshold can adjust routing to different desired response-quality levels.
- Results: 22% of queries were assigned to Llama-2 (13b) with less than 1% drop in BART-score response quality.The figure reports this as a reduction in calls to GPT-3.5-turbo.
- Implications: The approach targets cost reductions for both consumers and platform owners through edge-device routing or backend selection of lower-cost models.Consumers can reserve large-model API calls for more complex queries while maintaining specified quality levels.
2 Problem Formulation
The paper formulates hybrid LLM inference as single-model query routing between models with different costs and accuracies. It defines response quality and cost advantage for evaluating this trade-off, distinguishing the approach from multi-model cascades and ensembles.
- 2.1 Related Work: Static compression methods produce fixed smaller models, while hybrid routing dynamically selects between models across tasks and query difficulties.The formulation addresses the cost–accuracy trade-off without replacing the large model.
- 2.1 Related Work: Hybrid inference uses two models of different sizes, routing easy queries to the smaller, cheaper model to reduce inference cost.The smaller model generally has lower accuracy than the larger model.
- 2.1 Related Work: Prior multiple-LLM approaches may invoke several models per query, whereas this work makes one LLM call for each query.The distinction separates query routing from cascades, ensembles, and speculative decoding.
- 2.2 Problem Setting: The framework supports backend routing across platform models and local routing that sends hard queries to the platform from edge devices.Routing proportions can be tuned according to user quality requirements.
- 2.2 Problem Setting: A router r maps each query to either the small model S or the large model L, with exactly one model selected at inference time.The formal objective is to route x to S(x) when r(x)=0 and to L(x) otherwise.
- 2.3 Evaluation Metric: BART score is used to evaluate response quality because it is cheaper than GPT-based ranking and correlates with ground-truth judgments.Traditional metrics such as BLEU and ROUGE have limited applicability across diverse NLP tasks.
- 2.3 Evaluation Metric: Cost advantage is defined as the percentage of queries routed to the smaller model.This proxy avoids requiring platform-specific cost measurements such as latency, FLOPs, or energy.
3 Hybrid LLM Inference
Hybrid inference routes easy queries to a small model while preserving response quality, using router scores that account for quality gaps and response randomness. Data transformation addresses weak training signals when the large model substantially outperforms the small model.
- Hybrid LLM Inference: Easy queries are those where the small model’s response quality is close to the large model’s, making them candidates for small-model routing.The framework aims to reduce inference cost without much response-quality drop.
- Hybrid LLM Inference: The quality gap H(x) is the difference between small- and large-model response quality and is random because LLM responses are nondeterministic.Response quality is measured with a quality function such as BART score.
- Hybrid LLM Inference: The router predicts a score for each query, and a test-time threshold routes queries above that threshold to the small model.The score is designed to identify queries where the small model is likely to match or nearly match the large model.
- 3.1 Deterministic Router: The deterministic router uses single-response labels indicating whether the small model’s BART score is at least the large model’s, and trains with binary cross-entropy.Its score can be viewed as estimating the probability that the small model matches or exceeds the large model.
- 3.2 Probabilistic Router: The probabilistic router estimates the matching probability by sampling 10 responses from each model, reducing the variance of single-sample labels.The resulting router is designed to incorporate uncertainty from nondeterministic LLM comparisons.
- 3.3 Probabilistic Router with Data Transformation: When the large model is much stronger, transformed labels use Pr[H(x) ≥−t] to provide a stronger and more balanced training signal.The relaxation is selected by maximizing average pairwise differences between transformed labels and is implemented with grid search.
4 Evaluation
The evaluation tests three routers across model pairs with small, medium, and large performance gaps, measuring response quality and cost advantage. Results show that routing can reduce large-model usage while preserving quality, with data transformation especially useful for challenging gaps.
- Evaluation Setup: The evaluation uses MixInstruct across diverse tasks, BART score for quality, and the fraction routed to the small model as cost advantage.Routers use DeBERTa-v3-large and are compared with all-at-large, all-at-small, and random-routing baselines.
- Small Performance Gap: 20% and 40% cost advantages incur only 0.1% and 0.2% response-quality drops for Llama-2 (7b) versus Llama-2 (13b) with rdet.For this small performance gap, rprob consistently improves rdet, while rtrans matches or slightly improves rprob.
- Observed Behavior: At 20% cost advantage, rprob and rtrans achieve no quality drop relative to all-at-large for small performance gaps.The routers can route queries where the small model’s response quality exceeds the large model’s for some cases.
- Medium Performance Gap: 20% and 40% cost advantages produce ≤1% and ≤4% quality drops for medium-gap pairs; at 40%, rtrans improves rprob by 0.5% in quality drop.The routers show reasonable cost advantages with acceptable quality loss for Llama-2 (13b) and GPT-3.5-turbo.
- Large Performance Gap: 40% cost advantage yields a 10.3% quality drop for rtrans on large-gap pairs, 3.5% and 2.8% lower than rdet and rprob respectively.For large performance gaps, rdet and rprob perform only marginally better than random routing, whereas rtrans distinguishes relatively easy queries.
- Router Validation: Router-assigned small-model queries have higher quality gaps than large-model queries, unlike random routing, whose average difference is nearly zero.This indicates that the router preferentially assigns easier queries to the small model across cost advantages.
- Router Latency: The router is nearly 10× faster than the fastest evaluated LLM, adding minimal inference overhead through a single forward pass over each query.The router’s latency is lower because it scores queries once, while LLMs generate responses token by token.
- Threshold Selection: Validation-set thresholds generalize closely to test sets, although the Llama-2 (7b) and Llama-2 (13b) pair shows a slight increase in performance drop.Thresholds control the efficiency–performance trade-off and are selected using a 500-sample calibration set with grid search.
5 Discussion and Conclusion
The paper presents cost-effective, quality-aware query routing between LLMs, reporting up to 40% cost advantage without significant response-quality loss. It also identifies out-of-distribution generalization across model pairs and data distributions as future work.
- Up to 40% cost advantage is achieved with no significant drop in response quality across LLMs of varying sizes.
- The work is presented as the first exploration of cost-effective and quality-aware query routing between LLMs.
- Out-of-distribution generalization remains open because training and testing use a fixed model pair and data distribution.
A.1 More Router Performance Results
Additional evaluations compare routing across LLM pairs with small, medium, and large performance gaps. The results show stronger router advantages as routing becomes more challenging, while consistently identifying easy queries for the small model.
- Small, medium, and large performance-gap categories are represented by different FLAN-t5, Llama-2, and GPT-3.5-turbo pairs.
- As routing becomes challenging, rprob improves considerably over rdet, while rtrans starts to dominate the competition.
- The routing strategy consistently identifies easy queries and routes them to the small model.
- Table 4 summarizes the cost advantage versus performance drop trade-off.
B Dataset Statistics
The dataset contains 20k instruction examples, including a 10k-example training split of real-world queries uniformly sampled from MixInstruct.
- The dataset consists of 20k instruction examples.
- The training split contains 10k real-world queries.
- The training queries are uniformly sampled from the MixInstruct dataset.
C.1 Query Hardness: A Case Study
The case study examines query hardness for the FLAN-t5 (11b) versus GPT-3.5-turbo routing pair. The accompanying figures focus on error-cost tradeoffs and quality-gap differences across routing scenarios.
- The query-hardness case study uses the FLAN-t5 (11b) versus GPT-3.5-turbo routing pair.
- Figure 9 presents error-cost tradeoffs for rdet, rprob, and rtrans across different model performance gaps.
- Figure 10 reports average quality-gap differences between queries routed to the small and large models for different performance gaps.
C.1.1 Easy Query Example
The easy query receives the same response from the small FLAN-t5 (11b) model and the large GPT-3.5-turbo model. Because their outputs match in quality, the query is classified as easy.
- The present-tense rewrite is identical from FLAN-t5 (11b) and GPT-3.5-turbo.Both models answer: “She works at the company for the past 3 years.”
- The matching high-quality responses make this query easy by definition.
C.1.2 Hard Query Example
The hard-query example contrasts a terse small-model answer with a comprehensive large-model response about the benefits of having a dog. The example also notes that dog ownership requires substantial time, effort, financial responsibility, and commitment.
- The small model answers only that a dog is good for physical activity, whereas the large model introduces numerous benefits.
- The large-model response covers companionship, mental health, physical activity, responsibility, socialization, empathy, security, health, and family dynamics.
- Dogs can provide security, health benefits, and stronger family bonds, according to the extended response.
- Dog ownership requires time, effort, financial responsibility, and careful consideration of a family’s lifestyle, resources, and commitment.
- The small response is described as uninformative and unpolished, while the large response provides comprehensive, well-constructed arguments; the query is therefore hard.
C.2 BART score: A case study
The case study examines whether BART scores correlate with human judgments of response quality using five Llama-2 (13b) responses. Higher scores correspond to more complete and accurate answers in the example.
- The case study evaluates the correlation between BART scores and human judgments using five random Llama-2 (13b) responses.
- −0.927 is the highest BART score and accompanies a response containing both the Douglas Aircraft Company and El Segundo information.
- As BART scores decrease from −0.927 to −2.93, responses progressively omit city information, omit the company name, or become completely wrong.