PET-SQL: A Prompt-Enhanced Two-Round Refinement of Text-to-SQL with Cross-consistency

1 Introduction

• The introduction situates PET-SQL within a three-stage Text2SQL paradigm of preprocessing, SQL generation, and post-calibration.Prior improvements highlighted include schema linking and self-consistency.Source: §1 · p. 2, Figure 1 · p. 2
• Formatting differences in database cell values can make semantically correct SQL execute incorrectly, especially across databases with varied formats.The paper notes that these variations are more commonplace and critical in industrial databases.Source: §1 · p. 2
• Cell-value references standardize database values for prompting and improve execution accuracy by a maximum of +4%.The prompt also uses schema information and magic instructions as prior knowledge.Source: §1 · pp. 2–3
• PreSQL-based schema linking parses mentioned tables and columns from a preliminary SQL instead of asking LLMs to name them directly, achieving up to +4% higher execution accuracy.Schema linking accounts for about 22% of GPT-4 mistakes, and the authors argue SQL generation is more natural for LLMs than this domain-specific linking task.Source: §1 · pp. 2–3
• Cross-consistency uses different LLMs at low temperatures to diversify SQL candidates, then applies majority or difficulty-aware voting.The authors contrast this with same-model self-consistency, which they say can increase hallucinations at higher temperatures.Source: §1 · p. 3
• 87.6% execution accuracy on the Spider benchmark is reported for the final predictions, with cross-consistency voting across LLMs.The introduction describes this as state-of-the-art on the Spider leaderboard.Source: §1 · p. 3

2 Related Work

• Earlier learning-based Text2SQL methods used pattern matching, relation-aware attention, grammar-based or sketch decoders, and later end-to-end pretrained sequence-to-sequence models.The cited works span [10] [11] [12], [13] [14], and [15]–[18].Source: §2 · p. 3
• LLM-based work uses in-context learning and reasoning, with examples including zero-shot C3 [5], decomposed DIN-SQL [3], and DAIL-SQL [4].Other approaches draw on chain-of-thought [23] and self-reflexion.Source: §2 · p. 3
• PET-SQL combines a tailored prompt and retrieved demonstrations with PreSQL-based schema linking, concise-prompt generation, and cross-model consistency.The overview reports 32% shorter prompts from schema linking and a 2% accuracy boost from cross-consistency.Source: Figure 2 · p. 4

3 Methodology

• The framework prompts an LLM with database information and demonstrations to generate PreSQL, prunes irrelevant schema, and generates FinSQL with one or more LLMs.It also checks consistency across the predicted results from multiple LLMs.Source: §3 · p. 4, §3.3 · p. 6, §3.4 · p. 7

3.1 Pre-processing: SQL-Tailored Prompting

• REp adds an optimization rule, randomly sampled cell-value references, and foreign-key declarations to the prompt to guide efficient SQL generation and convey database formats and table connections.The optimization rule asks the LLM to minimize execution time while preserving correctness.Source: §3.1 · pp. 4–5
• +4.0% performance gain is reported for cell-value references, which provide examples of database formats and values for query conditions.Three rows are randomly sampled from each table.Source: §3.1 · pp. 4–5

3.2 Pre-processing: To Generate A Preliminary SQL (PreSQL) for Schema Linking

• PreSQL is generated using demonstrations retrieved by question-skeleton similarity, then its table and column entities are parsed for schema linking; this stage performs table linking only.The paper notes that column linking has poor fault tolerance and may be inapplicable to weaker LLMs.Source: §3.2 · pp. 5–6
• Up to +12.6% execution accuracy is reported for the PreSQL pipeline compared with methods that rely on LLMs to directly link schema.The comparison is reported in Table 4.Source: §3.2 · p. 6

3.3 SQL Generation: To Generate the Final SQL (FinSQL)

• After linking, the prompt is simplified by removing schema information and database references irrelevant to the linked tables or columns.Prompt tokens can be reduced by as much as 31%.Source: §3.3 · p. 6

3.4 Post-calibration: Cross-Consistency Is Better than Self-Consistency

• Cross-consistency has multiple low-temperature LLMs generate SQL; their executed results are voted on, with difficulty-aware voting assigning candidate models by PreSQL complexity.PreSQL complexity is divided into easy, medium, hard, and extra grades.Source: §3.4 · p. 7

4.1 Experimental Setup

• Experiments use Spider’s 8,659 training, 1,034 development, and 2,147 test questions, and evaluate five LLMs using execution accuracy.The five models are CodeLlama-34B, SQLCoder-34B, InternLM-70B, SenseChat-70B, and GPT-4.Source: §4.1 · p. 7

4.2.1 Spider Leaderboard

• PET-SQL’s best reported Spider leaderboard result surpasses the second-ranked method by about 1%.The leaderboard table reports methods and execution accuracy (EX).Source: Table 1 · p. 7, §4.1 · p. 7

4.2.2 Comparison under other foundation LLMs

• The single-LLM, 9-shot comparison evaluates PET-SQL, DAIL-SQL, and DIN-DQL with schema linking and cross-consistency removed.The comparison also examines performance and consistency across different foundation LLMs.Source: Table 2 · p. 8, §4.2.2 · p. 8

4.3 A Deep Dive into Prompt Design

• Removing cell-value references lowers EX by 2.4% for InternLM and 6.4% for SQLCoder; dropping the optimization rule also reduces performance across all three models.Foreign-key declarations help CodeLlama by 7.5%, while InternLM performs better without them.Source: §4.3 · p. 8

4.4 A Deep Dive into Schema Linking

• Simplifying prompts with linked schemas improves test performance across all LLMs by 1%–6%, with larger gains for smaller models.The evaluation uses a 9-shot setting.Source: §4.4 · p. 9, Table 5 · p. 9

4.5 A Deep Dive into Cross-consistency

• 87.6% test execution accuracy is reached with difficulty-aware voting, improving on naive cross-consistency voting’s 86.7%.The experiments use 9-shot prompting.Source: Table 6 · p. 9, §4.5 · p. 9
• For easy questions, smaller LLMs achieve 93.2% EX, while difficult questions require a more powerful model such as GPT-4 to maintain high performance.The study recommends selecting LLMs according to question difficulty.Source: Table 7 · p. 9, §4.5 · p. 10
• Adding cross-consistency to DAIL-SQL improves Spider-test performance by 2%.CodeLlama is the single-LLM setting; SQLCoder and InternLM are added for cross-consistency.Source: Table 8 · p. 10, §4.5 · p. 10

5 Conclusion

• 87.6% execution accuracy on the Spider leaderboard is the reported PET-SQL result.The framework uses prompt enhancement and cross-consistency across LLMs.Source: §5 · p. 10
• PET-SQL uses PreSQL-based schema linking to simplify prompt information for final SQL generation.The conclusion also describes this as improving the efficiency and accuracy of SQL generation.Source: §5 · p. 10
• The framework enhances prompts and leverages cross-consistency across LLMs to address Text2SQL challenges.Source: §5 · p. 10

A.1 Implementation details

• Top-9 similar question-SQL pairs are retrieved by comparing sentence embeddings of the target and candidate question skeletons.Schema entities appearing in questions are masked when constructing the skeletons.Source: §A.1 · p. 12
• Executed results from five LLMs and PreSQL are voted with equal weights for cross-consistency.Source: §A.1 · p. 12
• Generation temperatures are 10^-7 for non-OpenAI LLMs and 0 for GPT-4, with input and output token limits of 4096 and 200.The temperatures are set low to decrease output randomness.Source: §A.1 · p. 12

A.2 The definition of table recall metrics

• The two recall measures assess schema-linking results using the test-set instances as their denominator.Both definitions average an indicator over N instances.Source: §A.2 · pp. 12–13
• Totally Equal recall counts instances whose linked tables exactly match the ground-truth SQL tables, with 1 as ideal.The definition treats exact table matching as both recalling the question-related tables and avoiding verbose information.Source: §A.2 · p. 13
• Subset recall counts instances where the ground-truth tables are contained within the linked tables.The paper describes this measure as the upper limit of accuracy achievable with schema linking, since useful tables remain in the prompt.Source: §A.2 · p. 13

A.3 The illustration of the simplified prompt with the schema linking results

• The illustrated schema-linking result keeps the singer table for “How many singers do we have?” and removes unrelated context.The example identifies singer_in_concert as irrelevant to the primary question.Source: §A.3 · p. 13
• The demonstration context includes similar count questions paired with SQL queries for farms, books, and actors.Source: §A.3 · p. 13
• The prompt asks for SQLite SQL only and includes foreign-key information linking singer_in_concert.Singer_ID to singer.Singer_ID.Source: §A.3 · p. 13
• The prompt example includes database schema and sampled cell-value references, including singer names, countries, song details, ages, and gender values.Source: §A.3 · p. 13

• Bird-dev is included as a dataset recognized for its heightened complexity.Source: §A.4 · p. 13
• DAIL-SQL leads across all evaluated Bird-dev metrics, while PET-SQL scores 0.391 with SQLCoder.Source: §A.4 · p. 14
• Table 9 compares SOTA methods across foundational LLMs on the Bird-SQL dataset in a 9-shot setting.Source: Table 9 · p. 14

A.5 The ablation of REp on Bird-dev

• Removing any REp component—OR, CV, or FK—decreases performance for CodeLlama-34B, InternLM-70B, and SQLCoder-34B on Bird-dev.Source: Table 10 · p. 14, §A.5 · p. 14

A.6 Further analysis of the proposed prompt REp

• About 7% to 10% relative performance boosts over CRp and ODp are reported across datasets and foundation LLMs in the zero-shot setting.Source: Table 11 · p. 14, §A.6 · p. 14

A.7 Further analysis of the PreSQL-based schema linking

• 94% table-linking recall contrasts with 85.2% PreSQL EX, and linked-table prompt simplification can correct wrong PreSQL cases.Source: Table 12 · p. 15, §A.7 · p. 15
• 4.89 average tables in the full-schema prompt falls to 1.60 with schema linking, close to the ground-truth average.Source: Table 13 · p. 15, §A.7 · p. 15

A.8 Further analysis of the cross consistency

• 87.60% Spider-test accuracy with all five CC models compares with 79.80% without GPT-4 and 79.08% replacing it with InternLM2-20B.Source: Table 14 · p. 15, §A.8 · p. 15