PaperlayerView exact revision on arXiv

Source-linked AI summary

Cluster, Route, Escalate: Cascaded Framework for Cost-Aware LLM Serving

Yasmin Moslem, Magdalena Kacmajor, Vasudevan Nedumpozhimana, Ammar Abbas, Solmaz Panahi, David Lynch, Zhuangzhuang Nie, Alexandros Agapitos, Aleksandar Milenovic, Hongmeng Song, Yucheng Shi, Yue Pan, Patricia Buffini, John D. Kelleher

arXiv:2606.27457v1cs.PFcs.CL

TL;DR

LLM serving must balance accuracy against inference cost because a single strong model wastes resources on easy queries while a small model can fail on hard ones. The paper proposes cluster-level cost-aware routing followed by quality-estimation escalation, retaining near-strong-model accuracy while reducing TPOT. The framework uses task-correctness labels and offline preparation, but its routing does not adapt to inference-time distribution shift and TPOT omits several latency factors.

  • Problem

    LLM deployments face an accuracy-efficiency trade-off, while existing routing commonly requires annotations beyond standard task evaluation.

  • Method

    A two-stage cascade clusters queries for budgeted cost-aware routing, then uses quality estimation to escalate low-quality efficient-model outputs.

  • Results

    The system retains 97% of the strongest model’s accuracy on TeleQnA and stays within 0.7 pp on AIME 2024 at 18% lower TPOT.

  • Takeaways & Limitations

    Task-correctness labels support budget-feasible routing and selective escalation across changing model pools without manual reconfiguration.

  • Takeaways & Limitations

    Offline centroids and routing tables do not adapt to query-distribution shift, while TPOT omits queueing, prefill, TTFT, network, batching, and load-dependent latency.

Abstract

from arXiv · show

Efficient deployment of large language models (LLMs) in production forces a trade-off between accuracy and cost. Operators often default to a single model that is either expensive for easy queries or insufficient for hard ones. To address this challenge, we propose a two-stage cascaded solution. Stage 1 clusters incoming queries and assigns each cluster to its most cost-effective model. The cost budget for this routing process is set by an interpretable hyperparameter, tuned offline. Stage 2 adds a quality estimation (QE) cascade; when an output from Stage 1 is judged low-quality, the query is escalated to a stronger model. This ensures only hard or low-confidence cases reach the expensive models. On the test datasets, the cascaded system retains 97-99% of the strongest model's accuracy while reducing Time Per Output Token (TPOT). It requires only task-correctness labels and adapts to changes in the model pool without manual reconfiguration.

1 Introduction

The paper addresses the accuracy-efficiency tension in LLM serving with a two-stage cascade that uses clustering, cost-aware routing, and post-generation quality estimation.

  • Motivation: Single-model deployment over-invests on easy queries or underperforms on hard ones.Stronger models waste resources on easy queries, while smaller models sacrifice accuracy on difficult queries.
  • Research gap: Existing routing systems often require annotations beyond standard task-evaluation labels.The proposed framework targets routing and quality recovery using task-correctness labels from model evaluation.
  • Framework: Stage 1 clusters queries, scores models with a cost-aware hyperparameter, and tunes that parameter offline to meet a TPOT budget.The selected parameter is applied unchanged at test time.
  • Framework: Stage 2 uses a quality-estimation classifier to escalate low-quality efficient-model outputs to stronger models.The classifier is trained with the same task-correctness labels and requires no additional annotation.
  • Results: Stage 1+2 retains 97% of the strongest model’s accuracy on TeleQnA and nearly all of it on AIME 2024 at 18% lower TPOT.The evaluation covers telecommunications question answering and mathematical reasoning with model pools from multiple families.
  • Related work: The framework is positioned against routing and cascade methods that use human preferences, self-verification, uncertainty, or other supervision.The paper distinguishes its task-correctness supervision and two-stage design from prior approaches.

3 System Overview

The system combines offline cluster-level routing with selective post-generation escalation, using an interpretable cost penalty and budget-constrained parameter selection.

  • Offline preparation: Cluster centroids and routing tables are computed offline from task-correctness labels and reused at inference.Inference requires one embedding lookup and one model-pool arg min; pool updates reuse the existing training corpus.
  • Two-stage design: Stage 1 sends warranted hard-query clusters directly to strong models, reserving Stage 2 for failures within efficient-model clusters.This avoids running an efficient model before escalation on every query.
  • Stage 1 routing: Each query is assigned to a cluster and routed to the model minimizing cluster error plus a cost penalty.The cost term uses normalized TPOT, and ties favor the faster model.
  • Cost-aware scoring: λ directly controls the accuracy-latency trade-off, with larger values favoring efficient models over stronger ones.The fastest model has Costnorm=0 and the slowest has Costnorm=1.
  • Model-pool management: Pareto pruning removes models that are no faster or more accurate than another model across clusters.On TeleQnA, this reduces a four-model pool to two Pareto-efficient candidates.
  • Routing regions: For two models, each cluster has a crossover λ where routing changes from the stronger model to the efficient model.The N crossover points define N+1 interpretable routing regions.
  • λ selection: The optimal λ maximizes accuracy subject to a user-specified TPOT budget.The experiments adopt B=20 ms; η measures accuracy cost per millisecond saved relative to routing all queries to the strong model, with lower η preferred.

5 Stage 2: Quality Estimation Cascade

The QE cascade evaluates efficient-model completions after generation and escalates only outputs judged insufficient to stronger models.

  • Cascade operation: Stage 2 inspects efficient-model completions after generation and re-routes poor-quality responses to stronger models.Outputs already produced by strong models bypass the classifier.
  • QE classifier: A fine-tuned ModernBERT-base classifier predicts either accept or escalate.Its input combines the query, model output, and generation length.
  • Supervision: Training labels come from task-correctness of the efficient model’s output.The cascade therefore uses task-correctness supervision rather than additional annotation.
  • Inference: At inference, the classifier chooses the class with the highest probability to accept the output or escalate it.The classifier adds only a small per-token cost relative to generation.

6 Datasets and Model Pool

The evaluation spans telecommunications QA and mathematical reasoning across different dataset scales and model pools. It compares Stage 1 routing and the combined Stage 1+2 cascade against single-model baselines.

  • Datasets: The evaluation uses TeleQnA and AIME 2024 to test generalisation across telecommunications QA and mathematical reasoning.TeleQnA contains 9,000 training and 1,000 test queries, while AIME 2024 contains 921 training and 30 test queries.
  • Model Pool: TeleQnA uses four Gemma and Qwen models, whereas AIME 2024 uses VibeThinker-1.5B and Qwen3-30B-A3B as its primary pool.Silhouette analysis selects 2 clusters for TeleQnA and 3 clusters for AIME 2024.
  • Evaluation Setup: Stage 1 routing and the Stage 2 quality-estimation cascade are evaluated on both datasets and compared with single-model baselines.The combined Stage 1+2 system is assessed as the cascaded configuration.
  • Model Pool: VibeThinker is faster but less accurate than the larger, slower Qwen3-30B-A3B model on AIME 2024.This accuracy-efficiency contrast motivates routing decisions within the AIME model pool.
  • Stage 1 Routing: At TPOT budget B=20 ms, λ∗=0.06 assigns C1 to V and C0 and C2 to Qwen3-30B.At λ=0.06, V scores 0.083 versus Qwen3-30B’s 0.091 on C1; Qwen3-30B wins C0, 0.123 versus 0.130, and C2, 0.143 versus 0.182.

C0 C1 C2

The C0/C1/C2 results show cost-aware Stage 1 routing followed by selective quality-estimation escalation, improving the accuracy–TPOT trade-off on AIME 2024 and TeleQnA.

  • AIME 2024: 19% lower TPOT, from 11.8 to 9.5 ms, accompanies a 2.7% accuracy decrease from 89.1% to 86.4% on AIME 2024 Stage 1.The test-set performance mirrors training trends, supporting offline λ selection.
  • TeleQnA: On TeleQnA, Pareto analysis prunes dominated models, leaving Q3-4B and G-26B as the routing pool.Qwen3-4B dominates G-E2B, while G-26B dominates G-E4B on both clusters’ TPOT and error.
  • AIME 2024: Stage 2 recovers 2.0% accuracy for only 0.2 ms additional TPOT, bringing AIME 2024 within 0.7% of always using Q3-30B while 18% faster.The combined Stage 1+2 system lies on the Pareto frontier.
  • TeleQnA: At λ=0.07, TeleQnA Stage 1 routes 590 of 1,000 test queries in C0 to Q3-4B and sends C1 directly to G-26B.C1 bypasses the QE classifier because it is routed directly to the stronger model.
  • TeleQnA: TeleQnA C0 accuracy rises from 68.9% to 74.0% when the classifier escalates an average of 202 of 590 Q3-4B queries per run.The escalation targets outputs from the efficient model for rerouting to G-26B.
  • TeleQnA: Stage 2 recovers 3.1 percentage points at 4.7 ms additional TPOT, reaching 74.3% accuracy at 23.8 ms while remaining faster than always using G-26B.Q3-4B handles 59% of queries, and the combined system is within 2.1% of always using G-26B.

8 Conclusion and Future Work

The framework improves model-pool efficiency by combining budget-feasible cluster routing with selective quality-estimation escalation. It remains close to the strongest model while reducing TPOT and supports model-pool changes through Pareto analysis.

  • The cascaded system stays within 0.7 pp of the strongest model on AIME 2024 at 18% lower TPOT, and within 2.1 pp on TeleQnA.
  • Cluster-level routing selects budget-feasible operating points, while selective QE escalation recovers accuracy lost on efficient-model assignments.
  • The framework supports adding or removing models through Pareto analysis without committing to a fixed learned router.
  • Scaling to larger model pools, multi-class QE decisions, online λ adaptation, and additional efficiency metrics remain future work.

Limitations

The framework's conclusions are bounded by offline routing, hardware- and workload-specific efficiency measurements, and GPU-memory requirements. Its broader implications include both potential accessibility benefits and increased deployment at scale.

  • Offline cluster centroids and routing tables do not adapt to inference-time query distribution shift without new labels and pipeline reruns.
  • TPOT omits queueing, prefill, TTFT, network overhead, batching effects, and end-to-end latency under load.
  • TPOT measurements are specific to 2× A100 SXM 80 GB GPUs, and hardware differences can shift the Pareto frontier and optimal λ.
  • All candidate models must be simultaneously resident in GPU memory, restricting model-pool size in VRAM-constrained deployments.
  • Cost-reducing routing may improve accessibility and energy savings while also increasing deployment at scale.

A Inference Setup

Inference uses vLLM on fixed A100 hardware with controlled concurrency, sampling, prompts, and repeated runs. AIME uses explicit reasoning, whereas TeleQnA uses direct-answer settings.

  • All experiments use vLLM Server on 2× A100 SXM 80 GB GPUs, with maximum generation lengths of 40,960 tokens for AIME and 1,024 for TeleQnA.
  • Requests use max-concurrency 32 and seed 0 for reproducibility, with accuracy and TPOT averaged over 5 runs per dataset.
  • AIME thinking-mode models use temperature 0.6, top-p 0.95, top-k 20, and min-p 0, while TeleQnA uses temperature 0.7 and top-p 0.8.
  • AIME models generate explicit chain-of-thought reasoning, whereas TeleQnA models operate without explicit reasoning.
  • Each model receives the dataset question plus a dataset-specific instruction that standardises the output format.
  • AIME prompts request a brief approach followed by a numerical final answer.
  • TeleQnA prompts request a one-sentence explanation followed by an answer choice number.

B Framework Extensibility: AIME Pool Expansion

The extensibility experiment shows that Pareto analysis can reject dominated models and incorporate superior ones into the AIME routing pool. Adding Q3.5 improves the two-model configuration while preserving automated classifier-based escalation.

  • Framework Extensibility: AIME Pool Expansion: The framework reruns Pareto analysis when models change, automatically discarding dominated models or promoting new models into the routing pool.
  • Framework Extensibility: AIME Pool Expansion: Qwen3-4B-Thinking-2507-FP8 is Pareto-dominated by Q3-30B and is never selected for any λ.
  • Framework Extensibility: AIME Pool Expansion: Q3.5 achieves lower error rates than V and Q3-30B in all clusters and wins all three at λ=0.06.
  • Framework Extensibility: AIME Pool Expansion: 89.3% accuracy at 11.0 ms improves over 86.4% at 9.5 ms for the two-model configuration and 89.1% at 11.8 ms when always using Q3-30B.
  • Framework Extensibility: AIME Pool Expansion: The QE classifier labels efficient-model outputs Accept when correct and Route when escalation is needed.
  • Framework Extensibility: AIME Pool Expansion: For AIME, classifier inputs concatenate the query, model output, and output-token count, truncating model output to the last 1,000 words.
  • Framework Extensibility: AIME Pool Expansion: AIME classifier training uses 1,842 examples from AIME 1983–2023, while TeleQnA training uses 45,000 examples.

D QE Classifier Ablation (AIME)

The ablation tests the QE classifier independently by starting every AIME query with an efficient model and escalating selected outputs to Q3-30B. It shows that QE can recover accuracy on its own, while the combined cascade balances routing efficiency and accuracy.

  • The ablation isolates QE by processing all 30 queries with an efficient model before classifier-selected escalation to Q3-30B.
  • 0.93 system accuracy: VibeThinker’s cascade rises from 0.80 with 7 escalations and classifier accuracy 0.97.
  • 0.90 system accuracy: Qwen3-4B’s cascade escalates 14/30 queries, with classifier accuracy 0.87.
  • The combined two-stage system avoids some wasted efficient-model generations through Stage 1 routing, then recovers residual accuracy losses with Stage 2.
  • On AIME C1, runs 0–2 escalate one query with 10/10 accuracy, while runs 3–4 escalate none with 9/10 accuracy.

E.2 TeleQnA

TeleQnA results show stable QE behavior across runs, though cautious escalation incurs substantial false discoveries. Routing and classification add relatively small overhead compared with generation TPOT in the reported setup.

  • 73.4%–74.5% C0 accuracy across runs, a 1.1 pp spread, confirms classifier stability.
  • Approximately 51% of escalations are unnecessary, while the classifier detects roughly 98 true positives per run.
  • The classifier accepts an average of 85 incorrect answers per run as false negatives.
  • Clustering assigns queries through one embedding pass, and model selection is an in-memory lookup with negligible latency.
  • 20.7 ms per query: sequential float32 QE classification adds approximately 0.52 ms per output token, versus 15–25 ms/token generation TPOT.
Loading 2606.27457v1…