PaperlayerView exact revision on arXiv

Source-linked AI summary

LLMSQL: Upgrading WikiSQL for the LLM Era of Text-to-SQL

Dzmitry Pihulski, Karol Charchut, Viktoria Novogrodskaia, Jan Kocoń

arXiv:2510.02350v2cs.CLcs.AI

TL;DR

WikiSQL has case, type, syntax, and answerability problems despite its role in text-to-SQL research. LLMSQL systematically revises and cleans the dataset for LLM-era use, and fine-tuned models under 10B parameters surpass 90% accuracy. The authors also identify unresolved aggregation-operator mismatches.

  • Problem

    WikiSQL contains case inconsistencies, type mismatches, syntax errors, and unanswerable questions that affect its reliability as a text-to-SQL benchmark.

    Source: §I · p. 1

  • Method

    The authors systematically analyze and clean WikiSQL through automated normalization and targeted manual corrections, producing full human-readable SQL queries.

    Source: §II · p. 2, §V · p. 8

  • Results

    Fine-tuned models under 10B parameters surpass 90% accuracy on LLMSQL.

    Source: §V · p. 9

  • Takeaways & Limitations

    LLMSQL provides a structurally sound, semantically consistent, executable resource for modern language models without additional preprocessing.

    Source: §V · p. 8

    Some aggregation operators still do not match the question intent or table contents, leaving syntactically valid but semantically incorrect queries.

    Source: D. Additional Unresolved Issues · p. 5

Abstract

from arXiv · show

Converting natural language questions into SQL queries enables non-expert users to interact with relational databases and has long been a central task for natural language interfaces to data. While the WikiSQL dataset played a key role in early text-to-SQL research, its usage has declined due to structural and annotation issues, including case sensitivity inconsistencies, data type mismatches, syntax errors, and unanswered questions. We present LLMSQL, a systematic revision and transformation of WikiSQL designed for the large language model era. We classify these errors and implement automated methods for cleaning and re-annotation. To assess the impact of these improvements, we evaluated multiple large language models, including Gemma 3, LLaMA 3.2, Mistral 7B, gpt-oss 20B, Phi-3.5 Mini, Qwen 2.5, OpenAI o4-mini, DeepSeek-R1, and others. Notably, DeepSeek-R1 achieves 88.40% accuracy in a zero-shot setting, and models under 10B parameters surpass 90% accuracy after fine-tuning. Rather than serving as an update, LLMSQL is introduced as an LLM-ready benchmark. Unlike the original WikiSQL, which was tailored for pointer-network models selecting tokens from input, LLMSQL provides clean natural language questions and full SQL queries as plain text, enabling straightforward generation and evaluation for modern natural-language-to-SQL models.

I. INTRODUCTION

The paper revises WikiSQL to address annotation and structural defects and make it a reliable benchmark for modern language models. It situates this work among text-to-SQL model advances and complementary benchmark efforts.

I. INTRODUCTION

  • WikiSQL’s type mismatches, case inconsistencies, and unanswerable questions undermine the reliability of a widely used text-to-SQL dataset.The paper argues these problems have often been ignored or only partly addressed in prior work.Source: §I · p. 1
  • LLMSQL analyzes WikiSQL errors, releases a standardized evaluation package, and refines annotations with manual and automated corrections.The package provides standardized scoring, execution-based evaluation, and reference utilities; corrections target stable query execution and correct results.Source: §I · pp. 1–2
  • LLMSQL preserves WikiSQL’s scale and diversity while providing validated, human-readable SQL suited to modern language-model evaluation.The authors position LLMSQL alongside Spider for multi-table and compositional reasoning and BIRD for robustness to noisy annotations.Source: §I · p. 2

II. RELATED WORK

  • Prior text-to-SQL work improved query structure, schema awareness, and execution-aware decoding, including SQLNet [4], TypeSQL, SeaD, and execution-guided decoding [7].SQLova [8] incorporates table headers and cell values; these model advances are primarily established on single-table WikiSQL.Source: §II · p. 2
  • Spider, UNITE [9], and BIRD broaden benchmark scope, while systematic cleaning of WikiSQL itself remained underexplored.The paper focuses on annotation analysis and correction alongside evaluation of modern LLMs.Source: §I · p. 1, §II · p. 2

III. DATA

WikiSQL is a large benchmark of natural-language questions paired with queries over Wikipedia tables. Its data files and query format constrain the benchmark to relatively simple SQL.

A. Introduction to NL2SQL and the WikiSQL Dataset

B. Identified Issues in the WikiSQL Dataset

The paper identifies answer failures, incomplete table information, type conflicts, duplicated examples, and a numeric SQL representation as WikiSQL quality problems. These defects affect query execution and dataset usability.

B. Identified Issues in the WikiSQL Dataset

C. Cleaning and Re-annotation Process

The re-annotation process repairs missing headers, normalizes data and SQL, and removes duplicates. It also targets empty query results, while some underlying causes remain unresolved.

Incomplete information

  • The 140 tables with one missing column name received manually annotated headers based on their stored values.For example, a missing header was replaced with “Prime Minister Ordinal Number(s).”Source: Incomplete information · p. 3

Datatype Conflicts

  • Datatype corrections converted numeric strings to real values, numeric values in text columns to strings, and integers in real columns to floats.Source: Datatype Conflicts · p. 4
  • Duplicate tables and questions were removed, questions were reassigned to nonduplicate tables, and tables without questions were discarded.Duplicate tables were identified by matching column names, types, and row values; duplicate questions by matching query text and table inputs.Source: Duplicates · p. 4
  • 49.25% of queries initially returned no rows, and matching string-literal case to table values eliminated the targeted 41.22% of queries.The remaining 8.03% were unaffected by case adjustments, and their causes remain to be investigated.Source: Empty Query Results · pp. 4–5
  • For remaining empty results, the process tested capitalization variants and updated SQL and question literals when a nonempty result was found.The process also adjusted equality literals to match corresponding table values and retested the query.Source: Empty Query Results · pp. 4–5

Non-intuitive SQL Format

The revised format replaces numeric SQL placeholders with named components and stores constructed SQL queries directly. The changes are intended to improve readability and database compatibility.

Non-intuitive SQL Format

  • Numeric placeholders for columns and operators were replaced with their actual names, and the original SQL storage was replaced by standard SQL queries.The mapping of numeric placeholders to aggregation and comparison operators is shown in Table II.Source: Non-intuitive SQL Format · p. 5, Table II · p. 5
  • The queries avoid functions beyond regular clauses, making them compatible with standard SQL databases.The passage describes the queries as compatible with any database following the SQL standard.Source: Non-intuitive SQL Format · p. 5
  • The refined JSON's sql field contains an already constructed query rather than the initial SQL description format.Listing 7 is identified as an example of the refined question format.Source: Non-intuitive SQL Format · p. 5

D. Additional Unresolved Issues

The unresolved issues include aggregation operators that do not reflect query intent and an example where COUNT is used instead of SUM for attendance. The example underscores the difference between counting rows and summing values.

D. Additional Unresolved Issues

  • An aggregation operator can be syntactically valid yet produce semantically incorrect results when it mismatches the question's intent or table contents.The paper identifies aggregation-operator alignment as an unresolved issue.Source: D. Additional Unresolved Issues · p. 5
  • The example replaces COUNT("Attendance") with SUM("Attendance") for the question about attendance when the opponent was the Twins.The original and corrected queries use the same table and opponent filter.Source: D. Additional Unresolved Issues · p. 5
  • The sample attendance values are 34348.0, 38237.0, and 23849.0; the example states that matching-row values should be summed.Source: D. Additional Unresolved Issues · p. 5

• COUNT: [[0]]

The COUNT-zero example links case matching and empty-result handling to a corrected sum. The evaluation section describes prompts and benchmark rules for assessing LLM-generated SQL.

• COUNT: [[0]]

  • For SUM, the query returned [[None]] because the empty-result criteria did not treat COUNT’s [[0]] as empty; matching the table’s case, Twins, would yield 23849.0.Source: • COUNT: [[0]] · p. 5
  • Automatically assigning the correct aggregation operator is difficult because it requires contextual understanding of the question's meaning.The passage says individual semantic assessment is the most reliable approach.Source: • COUNT: [[0]] · p. 5
  • Whether to correct every query with no matching rows remains under discussion, with some cases potentially left unchanged to retain dataset diversity.Source: • COUNT: [[0]] · p. 5

IV. LLM EVALUATION ON DIFFERENT SCENARIOS

  • LLMs can generate well-formatted, executable SQL from pretraining or instruction-tuning without additional fine-tuning.Source: §IV · p. 6
  • The evaluation uses standardized rules and few-shot prompts with tailored example rows and the real target table's first row, which the authors found helps models understand table structure and content.The full prompt and five-shot examples are provided in Appendix A.Source: §IV · p. 6

1) Temperature:

The evaluation protocol specifies decoding, output length, execution accuracy, and SQL extraction. It uses SQLite and extracts candidate queries from model responses before execution.

1) Temperature:

  • Most models used temperature zero for reproducibility, while reasoning-capable models used their recommended default sampling parameters.Source: 1) Temperature · p. 6

2) Maximum Number of Tokens:

  • Generation was capped at 256 new tokens; the authors chose this arbitrary limit because queries typically need about 50 tokens, leaving room for verbose reasoning.Source: 2) Maximum Number of Tokens · p. 6

3) Evaluation Metric:

  • Execution accuracy required matching rows, columns, and values regardless of result order; evaluation ran in SQLite, while the benchmark's basic SQL constructs limit dialect differences.Source: 3) Evaluation Metric · p. 6

4) SQL Extraction from Outputs:

  • Regex extraction recovers SQL from verbose outputs, caps candidates at 10, and counts a response correct if any executed candidate matches the reference results.The cap is intended to limit excessive generation and mitigate strategies that produce many variants.Source: 4) SQL Extraction from Outputs · pp. 6–7

A. Main Scenario (Benchmark)

The benchmark compares models across zero-, one-, and five-shot settings and examines fine-tuning on LLMSQL. Results vary by model and prompting regime: larger models lead the benchmark, while several smaller models exceed 90% execution accuracy after fine-tuning.

A. Main Scenario (Benchmark)

  • Model size is not decisive: Gemma 3 (4.3B) scores 60.9% in 0-shot versus Mistral 7B's 24.4%, while similarly sized Qwen 2.5 and Llama 3.2 differ sharply.Qwen 2.5 reaches 53.41% in 5-shot, compared with Llama 3.2's 22.44%.Source: A. Main Scenario (Benchmark) · p. 7
  • Accuracy rises consistently from 0-shot to 1-shot to 5-shot; Phi 3.5 (3.8B) moves from 24.7% to 62.07%.The authors suggest examples help models adapt to dataset specifics and avoid overshooting task complexity.Source: A. Main Scenario (Benchmark) · p. 7
  • Many models produce unnecessarily complex SQL or use unsupported functions such as SUBSTRING(), causing execution failures in SQLite.Source: A. Main Scenario (Benchmark) · p. 7

B. Additional Scenario: Fine-Tuning

  • Fine-tuning follows the original WikiSQL splits and tracks test execution accuracy over epochs under fixed hyperparameters; three epochs may not reach each model's peak.The fixed conditions were intended to put models on equal footing rather than extensively tune each one.Source: B. Additional Scenario: Fine-Tuning · pp. 7–8

A. Main Scenario (Benchmark)

  • 88.4%: DeepSeek R1 leads across settings, while OpenAI o4-mini reaches 86.45% in 5-shot; DeepSeek R1 falls from 88.4% zero-shot to 86.57% in 5-shot.The largest models show limited gains from additional in-context examples.Source: A. Main Scenario (Benchmark) · p. 7, Table IV · p. 8

B. Additional Scenario: Fine-Tuning

  • More than 90% execution accuracy is reached by fine-tuned Gemma3 4B IT, Llama3.2 1B Instruct, Phi3.5 Mini Instruct, and Qwen2.5 1.5B Instruct.The authors attribute the gains to improved understanding of dataset-specific structures and conventions.Source: B. Additional Scenario: Fine-Tuning · p. 8
  • No larger fine-tuned model exceeds 90% execution accuracy; gpt-oss-20B remains around 78–78.5% despite LoRA fine-tuning.The authors suggest larger models may need more specialized strategies or longer schedules.Source: B. Additional Scenario: Fine-Tuning · p. 8

V. CONCLUSIONS

The conclusion presents LLMSQL as a cleaned, executable WikiSQL resource with human-readable SQL suited to modern language models. It reports strong results for reasoning-oriented models and fine-tuned smaller models, and outlines plans to extend the dataset's coverage and scope.

V. CONCLUSIONS

  • LLMSQL combines automated normalization and targeted manual corrections to make examples structurally sound, semantically consistent, and practically executable, with complete human-readable SQL.The authors state that it is usable by modern language models without additional preprocessing.Source: §V · p. 8
  • Reasoning-oriented DeepSeek R1 and OpenAI o4-mini exceed 85% execution accuracy, while fine-tuned relatively small models surpass 90%.Source: §V · pp. 8–9
  • Over 62% of 8.1 million Uber production queries used JOIN, while fewer than 1% used UNION, INTERSECT, or EXCEPT; the authors use this to argue simple clauses matter in practice.Source: §V · p. 9

VI. FUTURE WORK

  • Planned extensions include more questions per table, JOIN queries, additional data types, multilingual support, and integration with other benchmarks.The stated aim is to expand coverage and complexity while retaining structural simplicity and practical applicability.Source: §VI · p. 9

APPENDIX

The appendix specifies a prompt for generating SQLite SQL from a question and table schema. It restricts the output format and SQL constructs, and supplies five examples.

  • The prompt asks the model to act as an SQLite SQL query generator and return only a valid SELECT query for the given question and schema.Source: APPENDIX · p. 10
  • The prompt fixes the table name as "Table" and restricts functions to MAX, MIN, COUNT, SUM, and AVG, with SELECT, WHERE, and AND as allowed keywords.Source: APPENDIX · p. 10
  • Five-shot examples show question, columns, types, sample row, and SQL; the prompt also requires quotes around every column and table name.Source: APPENDIX · p. 10

Related papers

Loading 2510.02350v2…