Prompting GPT-3.5 for Text-to-SQL with De-semanticization and Skeleton Retrieval

1 Introduction

• LLMs are not designed to understand SQL syntax and semantics, making accurate generation of complex SQL commands challenging.The introduction notes that distinguishing question intention is essential for accurately mapping SQL commands.Source: §1 · p. 2
• For the illustrated examples, skeleton similarity increases both the absolute similarity score and relative ranking compared with full-question similarity.The examples distinguish similarity in sorting intention from similarity in question vocabulary, with the goal of retrieving SQL demonstrations containing matching commands such as ORDER BY and DESC.Source: Figure 1 · p. 2
• Questions from different database schemes can contain different scheme-related information despite sharing intentions and similar SQL, complicating retrieval of useful demonstrations.The framework addresses this mismatch by extracting question skeletons and retrieving demonstrations with similar skeletons, while filtering scheme items related to question tokens and adapting schema range in prompts.Source: §1 · pp. 2–3
• The framework surpasses state-of-the-art models and exhibits strong generalization.The authors describe these as contributions of their approach.Source: §1 · p. 3

2 Related Work

• Encoder-based work moved from attention-based Bi-LSTM and intermediate-representation decoding to graph-based encoders and explicit modeling of question-schema relationships.Examples include IRNET, RATSQL, LGESQL, R2SQL, SDSQL [3], S2SQL, STAR, GRAPHIX-T5 [4], and RASAT.Source: §2 · p. 3
• Decoder-based methods include sequence-based, template-based, and stage-based approaches, as well as hierarchical methods.The passage names BRIDGE and PICARD as sequence-based examples, X-SQL [28] and HydraNet [29] as template-based examples, and GAZP and RYANSQL as stage-based examples; IRNet and RAT-SQL are hierarchical examples.Source: §2 · pp. 3–4
• The paper’s framework overview separates de-semanticization from prompt construction and depicts SQL revision along a separate path.The figure caption identifies the grey box as de-semanticization and the rest as prompt construction.Source: Figure 2 · p. 4
• LLM-based models can use a few unsupervised in-context exemplar annotations, and prior work has retrieved SQL examples based on question characteristics or intention.Yu et al. classify and cluster SQL queries using question characteristics; the authors describe retrieving examples with the same intention to support understanding of operators and their applications.Source: §2 · p. 4

3 Methodology

• The framework has two modules: question de-semanticization and LLM-based adjustable prompting.The first removes domain-related tokens while preserving question skeletons; the second uses same-intention SQL demonstrations and corresponding database schema to guide query generation.Source: §3 · p. 4
• De-semanticization removes question tokens semantically related to the database schema, leaving skeletons that represent question intentions.Removing schema-related information is used to better match demonstrations with similar intent during retrieval.Source: §3 · p. 4
• The task is to generate the corresponding SQL query P from a natural-language question Q and database schema S = ⟨T, C⟩.The passage denotes the schema components as tables T and columns C.Source: §3 · p. 4

3.1.1 Schema-Related Detection.

• Masking each question token in a concatenated question-and-schema sequence lets a pretrained language model reveal token–schema correlations through representation shifts.The schema is treated as domain knowledge; a shift beyond a predefined threshold indicates a token strongly correlated with its meaning.Source: §3.1.1 · pp. 5–6

3.1.2 Token Matching.

• Question–schema relevance scores combine Poincaré-distance-based correlations with explicit name- and value-based token matches.The parameter β controls the relative influence of the two strategies.Source: §3.1.1 · p. 5, §3.1.2 · p. 6

3.1.3 Part-of-Speech Tagging.

• Part-of-speech tags contribute lexical scores, with nouns and numbers receiving α and other tokens receiving 0.These scores are combined with relevance scores and a minimum-score threshold τ to generate the question skeleton.Source: §3.1.3 · pp. 6–7

3.1 Question De-semanticization

• The method removes domain-relevant question tokens to create skeletons for retrieving examples with more consistent and applicable intentions.The process first links question tokens to the schema and estimates their relevance, then masks tokens to form skeletons.Source: §3.1 · pp. 4–5, §3.1.3 · p. 7

3.2 LLM-Based Adjustable Prompting

• Prompts combine retrieved SQL demonstrations with relevant database schema, and a fallback mechanism can adjust the schema range.The prompt consists of three parts, as shown in Fig. 2.Source: §3.2 · p. 7
• De-semanticized question skeletons serve as keys for retrieving k-NN examples using cosine similarity between text vectors.The retrieved examples share question intent and may have common structures and comparable SQL queries.Source: §3.2 · p. 7

3.2.2 Schema-Relevance Filtering.

• Schema items are retained only when their question–schema relevance scores exceed threshold θ.The resulting scaled-down schema is used to prompt the LLM to generate SQL.Source: §3.2.2 · p. 7
• Narrowing the schema range prevents irrelevant schema from interfering with the model’s response to the current question.Source: §3.2.2 · p. 7

3.2.3 Fallback Revision.

• If a generated query fails execution or SQL generation, the framework retrieves the complete database schema and uses it to revise the query.The revised query is checked for validity and executability before being passed to the LLM for further processing.Source: §3.2.3 · p. 8
• Fallback attempts are capped; the process terminates if SQL generation still fails after the maximum number of attempts.Source: §3.2.3 · p. 8

4.1 Experimental Setup

• Experiments use Spider, Spider-Syn, and Spider-DK, evaluating valid SQL, execution accuracy, and test-suite accuracy rather than exact match.Test-suite accuracy is not applicable to Spider-DK.Source: §4.1 · p. 8, §4.3 · p. 9

4.2 Main Results

• The framework outperforms models on almost all reported metrics, except that its Spider VA score is 0.1 below RESDSQL-3B.The comparison covers three datasets; the reported exception is Spider VA on Spider.Source: Table 1 · p. 9, §4.2 · p. 9

4.3 Ablation Study

• Ablations report gains from Poincaré distance, de-semanticization, skeleton retrieval, schema filtering, and SQL revision.Schema filtering boosts performance even without revision; Poincaré distance improves efficacy over Euclidean distance for schema detection.Source: §4.3 · p. 10

4.4 Case Study

• In a Spider-DK case, PICARD and ChatGPT add an unnecessary Age column, while three models miss the requested sorting order.The database has a birthday column, whose ascending order is equivalent to sorting by age; the proposed method orders results in the opposite direction from the question.Source: §4.4 · p. 10, Figure 3 · p. 10

• The proposed Text-to-SQL framework prompts LLMs with a few retrieved demonstrations.The paper also states that the approach can generalize to knowledge-base question answering and code generation.Source: §5 · p. 11
• The approach depends on LLMs having sufficient SQL code-generation capability.Source: §5 · p. 11
• Future work will address external knowledge reasoning and efficiency with large databases.Source: §5 · p. 11