PaperlayerView exact revision on arXiv

Source-linked AI summary

Reliable Text-to-SQL in Resource-Constrained Settings with Small LLMs via Iterative Retry Workflows

Bilel Abderrahmane Benziane, Xavier Schepler

arXiv:2610.17305v1cs.AI

TL;DR

Small and medium LLMs may be costly to deploy for Text-to-SQL, and their performance depends on model specialization and prompt representation. The paper evaluates these factors on Spider and tests retry workflows; SQL-specialized models perform strongly at small scales, while prompt-flow improves execution accuracy over single prompts with moderate added inference calls.

  • Problem

    LLMs can be resource-intensive, and the paper examines how model specialization and prompt representation affect Text-to-SQL performance with smaller models.

    Source: §I · p. 1, §IV · p. 4

  • Method

    The study evaluates small and medium models on Spider and tests three retry workflows that vary prompts, models, or both after failures, with syntax correction in the workflows.

    Source: D. Proposed Workflows · p. 3, C. Models · p. 3, §IV · p. 4

  • Results

    SQL-specialized models exceed 55% average execution accuracy, and prompt-flow improves execution accuracy over any single prompt style for both evaluated SQL-oriented models.

    Source: §IV · p. 4

  • Takeaways & Limitations

    Prompt-flow offers the best accuracy–cost trade-off in this setup, while model-and-prompt flow reaches the strongest performance when accuracy is prioritized.

    Source: §IV · p. 4, §V · p. 6

    The workflows require additional inference calls, and model-and-prompt flow requires more calls than prompt-flow.

    Source: §IV · p. 4

Abstract

from arXiv · show

Large Language Models (LLMs) have recently improved Text-to-SQL generation, but their computational cost can hinder deployment in resource-constrained settings. This paper evaluates the Text-to-SQL capabilities of small and medium open-source LLMs (1B-8B parameters) under a controlled inference budget, focusing on how model specialization and prompt formulation affect performance. Using the Spider benchmark, we compare multiple prompt representations and assess the accuracy of generated queries. Our results show that SQL-specialized models are consistently more reliable than general-purpose and chat-oriented models at small scales, and that performance can vary substantially depending on the prompt used. To reduce this variability without resorting to larger models, we introduce three lightweight inference workflows that automatically retry with alternative prompts and/or models when failures occur, including an optional syntax-correction step. These workflows significantly improve end-to-end success rates, with a trade-off in additional inference calls. These results demonstrate a practical path to stronger Text-to-SQL performance with small LLMs, by trading modest additional inference cost for higher reliability.

I. INTRODUCTION

The introduction frames Text-to-SQL specialization as a way to improve task-specific performance and broaden database access, while noting the resource costs of LLMs. Building on prior scale and prompt findings, the paper evaluates smaller models and proposes three workflows to improve accuracy.

I. INTRODUCTION

  • Text-to-SQL models are presented as specialized models that can help nontechnical users query complex databases.The introduction links this task specialization to democratizing data access.Source: §I · p. 1
  • LLMs can impose substantially higher resource costs than smaller models, motivating resource-conscious inference.Earlier compression work reduced memory use but had performance and tuning drawbacks, and was studied only for models below 350M parameters.Source: §I · p. 1
  • Prior work reported 39.6% average execution accuracy for CodeLLaMA-34B versus 19.4% for LLaMA-2-CHAT-13B, and found that question representation matters.The present study builds on those findings by examining medium and small LLMs and suggesting three workflows to improve accuracy.Source: §I · p. 1

II. RELATED WORK

The related-work discussion identifies fine-tuning and prompt engineering as approaches to improving LLM performance. It describes code- and SQL-oriented model families, including small models, and notes that this study evaluates already-pretrained models.

II. RELATED WORK

  • Prior literature discusses two broad techniques for improving LLM performance.The following discussion covers fine-tuning and prompt engineering.Source: §II · p. 1

A. Fine Tuning

  • Code-oriented models are often produced by fine-tuning generic models on code and then, in some cases, aligning them for instruction following.Examples include Qwen2.5-Coder-7B-Instruct, DeepSeek-Coder-Instruct, and CodeLlama-7B-Instruct [13]–[15].Source: A. Fine Tuning · pp. 1–2
  • The literature includes SQL-specialized models as small as 1B parameters, and this study evaluates already-pretrained models.Examples include CodeS-1B-Spider [16] and Prem-1B-SQL, based on DeepSeek-Coder-1.3B-Instruct [14].Source: A. Fine Tuning · p. 2

B. Prompt Styles

The section reviews prompt representations that add instructions or schema context to natural-language questions, then introduces this study’s evaluation of models and workflows across prompts and query types.

B. Prompt Styles

  • The Basic Prompt lists table names and columns, then presents the natural-language question and an answer prefix such as “A: SELECT.”It is a basic question representation for query generation [17].Source: B. Prompt Styles · p. 2
  • Text Representation adds an instruction, while OpenAI and Alpaca SFT prompts further specify the requested SQL output [18]–.The OpenAI prompt uses “#” for information fields and asks for SQL without explanation; Alpaca SFT uses a more detailed instruction.Source: B. Prompt Styles · p. 2
  • Core Representation Prompt addresses missing schema context by including each table’s full SQL schema and foreign keys [21].The text identifies foreign keys as important for generating multi-table queries.Source: B. Prompt Styles · p. 2
  • Evaluations of medium and small LLMs for SQL generation remain scarce, motivating evaluation across prompting styles and query types.The study also evaluates three SQL-generation workflows against a baseline.Source: B. Prompt Styles · p. 2

III. METHODOLOGY

  • The methodology covers the dataset, evaluation metrics, models, prompting configurations, and workflows intended to improve robustness in resource-constrained settings.The passage describes the scope of the methodology rather than individual workflow mechanics.Source: §III · p. 2

A. Dataset

The evaluation uses Spider’s development partition and reports both exact-match and execution accuracy. It compares model families under shared inference settings, including a structured prompt with explicit input sections.

A. Dataset

  • The evaluation uses the final 1,034 examples of Spider’s development partition, drawn from 10,181 questions across 200 databases.The set includes simple SELECT, GROUP BY, ORDER BY, JOIN, AGGREGATE, SUBQUERY, and HAVING query types; Fig. 1 shows their distribution.Source: A. Dataset · p. 2, Figure 1 · p. 2

B. Metrics

  • Exact-set-match accuracy compares predicted and reference SQL at the clause level, while execution accuracy compares their database results.The paper characterizes exact-set match as conservative and execution accuracy as more faithful when multiple semantically equivalent queries exist.Source: B. Metrics · p. 2

C. Models

  • The evaluated model families are fundamental, instruction-tuned, code, chat, and SQL-specialized LLMs.The paper groups models by specialization and tuning type.Source: C. Models · p. 2
  • The tested code and SQL-oriented models range from code checkpoints such as StarCoder2-3B and CodeS-1B to SQL-specialized models such as Prem-Text-to-SQL-1B and Defog-SQLCoder-7B.The code-model set also includes Qwen2.5-Coder, CodeLlama, and DeepSeek-Coder variants; the SQL-specialized set includes CodeS Spider and Bird variants.Source: C. Models · p. 3
  • All models receive a 150-token response budget and default temperature 0, and the study adds a structured prompt with instruction, schema, and question sections.The prompt uses a simplified schema description to support comparison across prompt styles; Fig. 2 depicts the template.Source: C. Models · p. 3, Figure 2 · p. 3

D. Proposed Workflows

Because performance may vary by question type and prompt representation, the paper proposes three workflows to improve robustness; its baseline fixes both the prompt style and model.

D. Proposed Workflows

  • Using one model and one prompt style can lead to more errors when performance varies with question type and prompt representation.Source: D. Proposed Workflows · p. 3
  • The paper proposes three workflows to improve robustness.Source: D. Proposed Workflows · p. 3
  • The single-model/prompt baseline fixes the model and prompt style, then returns the execution result if the generated SQL executes successfully.Source: D. Proposed Workflows · p. 3

1) Prompt Workflow:

The prompt workflow fixes one model and tries prompt styles in sequence, applying one syntax-correction attempt after a validation error.

1) Prompt Workflow:

  • The workflow keeps model M fixed and iterates through prompt styles P in sequence.Source: 1) Prompt Workflow · p. 3
  • After a SQL validation error, the same model makes one correction attempt conditioned on the question and candidate query.Source: 1) Prompt Workflow · p. 3
  • The workflow returns the first result from a valid query, or failure after exhausting all prompt styles.Source: 1) Prompt Workflow · p. 3

2) Model Workflow:

The model workflow fixes one prompt style and tries models in sequence, with one syntax-correction attempt by each model after a validation error.

2) Model Workflow:

  • The workflow keeps prompt style P fixed and iterates through models M in sequence.Source: 2) Model Workflow · p. 3
  • After a SQL validation error, the same model makes one correction attempt conditioned on the question and candidate query.Source: 2) Model Workflow · p. 3
  • The workflow returns the first result from a valid query, or failure after all models have been tried without a valid query.Source: 2) Model Workflow · p. 3

3) Prompt-and-Model Workflow:

The combined workflow tries model-and-prompt pairs, applying a same-model syntax correction after validation errors and returning the first valid result.

3) Prompt-and-Model Workflow:

IV. RESULTS AND DISCUSSION

SQL-specialized models outperform other model families at small scales, while prompt choice creates substantial performance variation. Retrying with alternative prompts improves execution accuracy, with additional inference calls as the cost of greater reliability.

IV. RESULTS AND DISCUSSION

  • SQL-specialized models all average above 55% EX, while general-purpose instruction-tuned models underperform their code-tuned counterparts and chat-oriented models average below 30%.Table I summarizes EM and EX across models and prompt styles.Source: §IV · p. 4, Table I · p. 5
  • Above 55% average EX for SQL-specialized 1B models, versus approximately 36% for code-oriented, 18% for chat-oriented, and 4% for fundamental models.The results identify SQL-specific fine-tuning and prompt choice as important factors at small scales.Source: §IV · p. 4
  • Performance varies by prompt and query type; the code representation prompt performs better on multi-table queries, likely because it provides richer schema information.No prompt style is uniformly optimal across models and query categories.Source: §IV · p. 4, Figure 3 · p. 5
  • Prompt-Flow improves EX over any single prompt; Model-and-Prompt Flow has the strongest Fig. 3 profile but matches the best Prompt-Flow EX with more calls.Prompt-Flow offers the best accuracy–cost trade-off in this setup; the best choice depends on deployment constraints.Source: §IV · p. 4, Figure 3 · p. 5, Table II · p. 6

V. CONCLUSIONS

Prompt-Flow provides the best robustness under resource constraints, while Model-and-Prompt Flow achieves the strongest performance when accuracy is prioritized, at higher computational cost. SQL-specific fine-tuning consistently outperforms other model families, with a more pronounced gap around 1B parameters.

V. CONCLUSIONS

  • SQL-specific fine-tuning consistently outperforms other model families, with the gap becoming more pronounced at around 1B parameters.Source: §V · p. 6
  • Prompt-Flow provides the best robustness under resource constraints, while Model-and-Prompt Flow achieves the strongest performance when accuracy is prioritized, at the cost of higher computation.Source: §V · p. 6
  • Future work should learn or predict efficient model-and-prompt selection from the input question to reduce inference cost while maintaining high accuracy.Source: §V · p. 6

Related papers

Loading 2610.17305v1…