Source-linked AI summary
Towards Optimizing SQL Generation via LLM Routing
Mohammadhossein Malekpour, Nour Shaheen, Foutse Khomh, Amine Mhedhbi
TL;DR
Text-to-SQL systems often use highly capable LLMs for every query, creating unnecessary latency and cost on simpler queries. This paper introduces score- and classification-based routers that select the weakest capable model, achieving near-best accuracy while reducing costs by up to 1.4× on BIRD.
Problem
Highly capable LLMs handle complex Text-to-SQL queries well but add unnecessary latency and dollar cost for simpler queries.
Method
The paper introduces score-based and classification-based N-ary routers that select the weakest model capable of generating accurate SQL.
Results
Up to 1.4× cost reduction is achieved while accuracy remains close to that of the most capable LLM on BIRD.
Takeaways & Limitations
Query-dependent routing offers an explainable accuracy-cost trade-off for enterprise Text-to-SQL deployments.
Takeaways & Limitations
The empirical analysis is currently limited to BIRD and routing within the generation stage, with broader datasets and pipeline stages left for future work.
Abstract
from arXiv · showhide
Text-to-SQL enables users to interact with databases through natural language, simplifying access to structured data. Although highly capable large language models (LLMs) achieve strong accuracy for complex queries, they incur unnecessary latency and dollar cost for simpler ones. In this paper, we introduce the first LLM routing approach for Text-to-SQL, which dynamically selects the most cost-effective LLM capable of generating accurate SQL for each query. We present two routing strategies (score- and classification-based) that achieve accuracy comparable to the most capable LLM while reducing costs. We design the routers for ease of training and efficient inference. In our experiments, we highlight a practical and explainable accuracy-cost trade-off on the BIRD dataset.
1 Introduction
Text-to-SQL supports database access across queries of widely varying complexity, but using highly capable LLMs for every query creates unnecessary latency and cost. The paper proposes routing each query to the weakest capable model while preserving near-best accuracy.
- Motivation: Text-to-SQL enables non-experts and analysts to access structured data, including queries ranging from simple row retrievals to multi-way joins with aggregations.This variability motivates adapting model capability to query complexity.
- Motivation: Current pipelines retrieve contextual information before SQL generation, where highly capable models handle complex queries but impose latency and cost on simpler ones.These inefficiencies can affect user experience and average cost per query.
- Motivation: BIRD ranks submissions by accuracy and generally assumes a single model, reproducing the inefficiency of using the strongest model for every query.Six of the top ten BIRD solutions use GPT-4o or Gemini.
- Contribution: LLM routers select the weakest, cheaper, faster model capable of generating accurate SQL for each query.The proposed routers target query-dependent model selection rather than a single-model deployment.
- Contribution: 1.4× lower cost is achieved while accuracy remains close to that of the most capable LLM and exceeds the second-best model.The paper reports this accuracy-cost trade-off on the BIRD benchmark.
2 Preliminaries
The paper formulates routing as selecting the weakest model that can generate accurate SQL, using an ordered model set and execution accuracy as the primary objective. It frames this as an N-ary routing problem supported by historical model outputs and labels.
- 2.1 Problem Formulation: The routing objective is to select the smallest index l such that model M_l generates accurate SQL for query Q.Models are totally ordered by strength, with lower indices denoting weaker models and typically lower latency and cost.
- 2.1 Problem Formulation: Historical data H contains natural-language queries, model-predicted SQL, and ground-truth SQL for all models.The examples can come from execution logs, user feedback, and manual labeling.
- 2.1 Problem Formulation: The learned N-ary routing function maps Q to the minimal label l, while label N denotes that no available model is capable of generating accurate SQL.Labels 0 through N−1 identify the weakest capable model.
- 2.2 Related Work: This work presents the first reported N-ary LLM routing study and the first application of LLM routing to Text-to-SQL.Earlier routing work addressed binary decisions in question answering, summarization, and information extraction.
- 2.2 Related Work: Unlike ensembles that generate multiple answers, routing targets cost minimization; a prior cascade approach is described as unsuitable for Text-to-SQL.The cited cascade cannot determine whether SQL generated by the weakest model is accurate or meets requirements.
- 2.3 Metrics: Execution accuracy measures the proportion of queries whose predicted and ground-truth output relations contain identical tuples, independent of attribute ordering.The objective is to maximize execution accuracy while minimizing dollar cost through weaker-model selection.
3 Methods
The paper implements score-based and classification-based routers that identify the weakest model likely to generate accurate SQL. Both can fall back to the strongest model when no model is predicted capable, prioritizing accuracy.
- 3 Methods: The score-based router predicts each model’s SQL-generation capability and selects the weakest model whose score exceeds a threshold.It treats a model as capable when its predicted probability of execution accuracy reaches the input threshold.
- 3 Methods: The classification-based router directly predicts the label l identifying the weakest capable model, with N indicating that none is capable.This uses the label space defined in the problem formulation.
- 3 Methods: When no model is predicted capable, the system may abstain or route to a model for a different accuracy-cost trade-off.The reported implementation routes to the strongest model to maximize accuracy.
- 3.1 Score-based Routing: The score-based implementation retrieves the top-K similar historical queries and uses their execution accuracy for each model as the capability score.Models meeting threshold α are filtered before selecting the weakest one.
- 3.2 Classification-based Routing: The classification router uses a DistilBERT encoder trained on H with the natural-language query and relevant retrieved schema as input.It predicts the weakest model label, or N when no model is expected to generate accurate SQL.
4 Experiments
Experiments evaluate three base LLMs and two routing approaches on BIRD, measuring execution accuracy, routing distribution, and normalized dollar cost. Both routers trade some accuracy for lower cost, with savings up to 1.4× while remaining close in execution accuracy.
- Experimental Setup: The evaluation uses all 1534 BIRD development queries, with 9428 training queries serving as the historical routing set H.
- Models: The base models are ordered by execution accuracy as Llama with EX 0.34, gpt-4o-mini with EX 0.48, and gpt-4o with EX 0.55.
- Cost: The study omits latency and evaluates cost using normalized dollar units, setting gpt-4o-mini to 1 and gpt-4o to 16.6×.
- Router Implementations: The score-based router uses α as a candidate threshold over K similar queries, while the classification-based router fine-tunes DistilBERT on labeled routing data.The score-based implementation searches α and K; the classification approach splits labeled data by database for training and validation.
- Results: Routing is motivated by overlapping successful and failed query sets, allowing weaker models to handle some queries while preserving the strongest model as the fallback.
- Results: Both routers are up to 1.4× cheaper than routing every query to gpt-4o while remaining close in execution accuracy on BIRD’s dev set.The reported trade-off is lower accuracy for lower cost relative to the strongest model.
- Limitations and Future Work: Future work will test whether queries with relevant schema suffice for complexity classification and extend routing beyond generation to other datasets and pipeline stages.
5 Conclusion
The paper investigates two LLM routing approaches for Text-to-SQL on BIRD and frames cost-based optimization as important for enterprise systems where accuracy and cost both matter.
- The study investigates two LLM routing approaches for Text-to-SQL on the BIRD benchmark.
- The authors identify cost-based optimization as important for enterprise systems where both accuracy and cost are critical.
- Future work will broaden empirical analysis to improve execution accuracy and cost with similar techniques.
A Analysis of Failure Cases
Failure analysis compares correct and failed SQL generation across gpt-4o, gpt-4o-mini, and Llama. A large shared failure set limits routing’s ability to improve execution accuracy beyond the strongest model.
- The analysis compares distributions of failed and correct predictions for gpt-4o, gpt-4o-mini, and llama3.1:8b-instruct-q4_0.
- 37.56% of queries failed across all three models, while pairwise common failure rates ranged from 45.49% to 67.83%.
- Because many failures are shared across models, routing is unlikely to improve execution accuracy beyond the strongest model and is expected mainly to reduce cost.
B Effect of Varying K and α on Execution Accuracy and Model Distribution
The score-based router exposes an accuracy-cost trade-off through K, the number of similar queries, and α, the threshold score. Increasing either parameter favors stronger-model routing and higher accuracy at greater cost.
- Figures 3–5 examine execution accuracy and model distribution under different None-routing strategies and values of K and α.The empirical observations are based on 10% of BIRD’s training set.
- Higher K or α increases execution accuracy but routes more queries to gpt-4o, increasing overall cost.
- Lower K or α favors cheaper models such as gpt-4o-mini and Llama, reducing cost while potentially decreasing execution accuracy.
- Selecting K and α lets practitioners tailor the score-based router to their required balance between execution accuracy and cost efficiency.