PaperlayerView exact revision on arXiv

Source-linked AI summary

A Preview of XiYan-SQL: A Multi-Generator Ensemble Framework for Text-to-SQL

Yingqi Gao, Yifu Liu, Xiaoxia Li, Xiaorong Shi, Yin Zhu, Yiming Wang, Shiqi Li, Wei Li, Yuntao Hong, Zhiling Luo, Jinyang Gao, Liyu Mou, Yu Li

arXiv:2411.08599v3cs.AIcs.CLcs.DBcs.LG

TL;DR

NL2SQL methods based on prompting can incur inference overhead, while SFT methods can struggle with complex reasoning and transfer to new domains. XiYan-SQL combines ICL and SFT candidate generation with M-Schema and a learned selection model; it reports state-of-the-art execution accuracy across relational and non-relational benchmarks.

  • Problem

    Prompt-based NL2SQL methods often use multi-path generation with inference overhead, while SFT methods can struggle with complex reasoning and transfer to new domains.

    Source: §1 · p. 2

  • Method

    XiYan-SQL combines ICL and SFT generators with M-Schema, candidate refinement, and a fine-tuned model that selects among candidates.

    Source: §1 · p. 2, §3 · p. 3

  • Results

    XiYan-SQL reports state-of-the-art execution accuracy across relational and non-relational benchmarks, including 75.63% on Bird.

    Source: §1 · p. 3, §8 · p. 9

  • Takeaways & Limitations

    The reported results span relational benchmarks and the non-relational NL2GQL dataset.

    Source: §1 · p. 3, §7.5 · p. 8

Abstract

from arXiv · show

To tackle the challenges of large language model performance in natural language to SQL tasks, we introduce XiYan-SQL, an innovative framework that employs a multi-generator ensemble strategy to improve candidate generation. We introduce M-Schema, a semi-structured schema representation method designed to enhance the understanding of database structures. To enhance the quality and diversity of generated candidate SQL queries, XiYan-SQL integrates the significant potential of in-context learning (ICL) with the precise control of supervised fine-tuning. On one hand, we propose a series of training strategies to fine-tune models to generate high-quality candidates with diverse preferences. On the other hand, we implement the ICL approach with an example selection method based on named entity recognition to prevent overemphasis on entities. The refiner optimizes each candidate by correcting logical or syntactical errors. To address the challenge of identifying the best candidate, we fine-tune a selection model to distinguish nuances of candidate SQL queries. The experimental results on multiple dialect datasets demonstrate the robustness of XiYan-SQL in addressing challenges across different scenarios. Overall, our proposed XiYan-SQL achieves the state-of-the-art execution accuracy of 75.63% on Bird benchmark, 89.65% on the Spider test set, 69.86% on SQL-Eval, 41.20% on NL2GQL. The proposed framework not only enhances the quality and diversity of SQL queries but also outperforms previous methods.

1 Introduction

XiYan-SQL combines prompt engineering and supervised fine-tuning to generate diverse SQL candidates, addressing limitations of each approach alone. It introduces M-Schema and reports leading execution accuracy across several benchmarks.

1 Introduction

  • XiYan-SQL combines prompt engineering with fine-tuned models to generate high-quality, diverse SQL candidates.SFT models are trained with different preferences, while ICL examples are selected using question skeleton similarity after masking named entities.Source: §1 · p. 2
  • The framework refines candidate queries using execution results or error information, then fine-tunes a model to select among candidates.The selection model is designed to distinguish subtle differences rather than rely only on self-consistency.Source: §1 · p. 2
  • M-Schema represents database structure hierarchically in a semi-structured format, and LLMs using it outperform those using DDL or MAC-SQL Schema.It adds data types and a more compact format.Source: §1 · p. 2
  • The workflow links the schema, generates candidates with ICL and SFT generators, and selects a final response, using M-Schema as the LLM schema representation.Figure 1 depicts these three agents.Source: Figure 1 · p. 3
  • 75.63% Bird, 89.65% Spider, 69.86% SQL-Eval, and 41.20% NL2GQL execution accuracy are reported as state-of-the-art results.The evaluations cover relational and non-relational databases.Source: §1 · p. 3

2 Overall Framework

XiYan-SQL organizes text-to-SQL generation into schema linking, candidate generation, and candidate selection. The schema-linking output is represented as M-Schema and passed to candidate generation.

2 Overall Framework

  • Schema Linking selects relevant columns and retrieves values to reduce irrelevant schema information before generation.The selected context is organized as M-Schema and fed to the candidate-generation module.Source: §2 · p. 3
  • Candidate Generation uses the linked schema context to produce potential SQL queries, which are then refined through self-refinement.Source: §2 · p. 3
  • Candidate Selection compares the candidates and determines the final SQL query.Source: §2 · p. 3

3 M-Schema

M-Schema presents database structure and column information in a semi-structured format for LLM prompts. It builds on MAC-SQL Schema with more compact representations and additional schema details.

3 M-Schema

  • M-Schema organizes database, table, and column relationships hierarchically, using identifiers for databases, tables, and foreign keys.Column entries include names, data types, descriptions, primary-key identifiers, and example values.Source: §3 · p. 3
  • DDL Schema lacks table and column descriptions and example values, which can make similar columns difficult for LLMs to distinguish.M-Schema is derived from MAC-SQL Schema and is described as more compact.Source: §3 · p. 3
  • M-Schema adds data types and primary-key markings to its column representation.The figure caption also identifies changed rules for displaying sample values.Source: §3 · p. 3, Figure 2 · p. 4
  • M-Schema obtains more detailed column descriptions from the database and simplifies value-example labels and display rules.The rules include limits on string length and the number of examples.Source: §3 · p. 4
  • Leading spaces are removed from column representations, and released tools support building M-Schema for MySQL and PostgreSQL.Source: §3 · p. 4

4 Schema Linking

Schema Linking maps question references to schema elements through retrieval and column selection. It combines keyword-based column retrieval, value retrieval, and relevance-based filtering.

4 Schema Linking

  • Schema Linking connects natural-language references to database tables, columns, and values through retrieval and column selection.Source: §4 · p. 4
  • The retrieval module extracts question keywords and entities, retrieves top-k columns by semantic similarity, and finds values using LSH followed by semantic similarity.The selected schema is the union of the column- and value-retriever results.Source: §4 · p. 4
  • The column selector uses few-shot prompting to retain only columns relevant to the query from the retrieved M-Schema.Its stated aim is to reduce the schema to a minimally sufficient set for SQL generation.Source: §4 · p. 4

5 Candidate Generation

Candidate Generation combines fine-tuned generators, ICL, and SQL refinement to produce high-quality, diverse candidate queries. Fine-tuning uses staged training, while ICL selects examples by question similarity and SQL complexity.

5 Candidate Generation

  • Candidate generation combines fine-tuned models with ICL generators to produce high-precision SQL candidates with diverse syntactic styles.The framework also describes refining generated queries before selection.Source: §5 · p. 4

5.1 Fine-tuned SQL Generator

  • Fine-tuned generators use two-stage, multi-task training to produce SQL candidates with distinct strengths and preferences.Basic-syntax training establishes SQL generation; generation-enhance training adds semantic and stylistic preferences through multi-task and syntactic-preference data.Source: §5.1 · p. 5, Figure 3 · p. 5
  • Fine-tuning combines task data with syntactic preferences, including SQL-to-question, SQL-to-evidence, and SQL discrimination and regeneration tasks.The report says these tasks strengthen links between SQL and context, while rephrased queries expand the available syntactic styles.Source: §5.1 · p. 5

5.2 ICL SQL Generator

  • ICL selects examples by question-skeleton similarity after replacing named entities with type tokens, reducing overfocus on entities while preserving their semantics.For questions linked to two or more tables, selected SQL examples must involve multiple tables; at most five examples are used.Source: §5.2 · pp. 5–6

5.3 SQL Refiner

  • The SQL Refiner regenerates queries using schema context, the original SQL, and execution results, including potential error information.The original and regenerated queries can then be passed to the selection model, and refinement may be repeated iteratively.Source: §5.3 · p. 6

6 Candidate Selection

Candidate selection addresses the difficulty of choosing a correct query from a pool when self-consistency is inconclusive. XiYan-SQL groups candidates by execution-result consistency and fine-tunes a model to select among them.

  • Self-consistency can fail when no queries agree, and the most consistent query is not always correct.The report identifies these as limitations of methods that select the most frequently recurring query.Source: §6 · p. 6
  • XiYan-SQL groups candidates by execution-result consistency, then fine-tunes a selection model to choose the most reasonable candidate using context and the candidate set.Training data is paraphrased to align with the candidates’ varying syntactic preferences.Source: §6 · p. 6

7 Experiments

XiYan-SQL is evaluated on relational and graph databases using execution accuracy; results cover Bird, Spider, SQL-Eval, NL2GQL, and component ablations. Its reported benchmark results include leading scores across all four datasets.

7.1 Experimental Setup

  • Experiments evaluate execution accuracy on SQLite-based Spider and Bird, PostgreSQL SQL-Eval, and graph-database NL2GQL.Because the Bird test set is unavailable, evaluation uses its development set.Source: §7.1 · p. 6

7.2 Bird Results; 7.3 Spider Results

  • 75.63% on Bird (0.84% ahead of second place) and 89.65% on Spider (0.05% above previous leading models) are XiYan-SQL’s reported results.On Bird, XiYan-SQL votes among five candidates, compared with CHASE-SQL’s 21.Source: §7.2 · p. 7, Table 2 · p. 7, §7.3 · p. 7

7.4 SQL-Eval Results

  • 69.86% on SQL-Eval is the highest score, 8.59% above SQL-Coder-8B fine-tuned on LLaMA-3.SQL-Eval supplies multiple reference SQL queries; the experiments use the first as ground truth.Source: Table 4 · p. 7, §7.4 · p. 8

7.5 NL2GQL Results

  • 41.20% execution accuracy on NL2GQL is the best reported result, exceeding GPT-4o, DeepSeek, Gemini 1.5 Pro, and Claude 3.5 Sonnet.The evaluation samples 288 examples from NL2GQL.Source: Table 5 · p. 7, §7.5 · p. 8

7.6.1 M-Schema

  • M-Schema improves average Bird accuracy by 2.03% over DDL Schema across four tested LLMs, but two models slightly decline versus MAC-SQL Schema.GPT-4o and Claude 3.5 Sonnet improve by 0.65% and 0.78%; DeepSeek and Gemini 1.5 decline by 0.13% and 0.26%.Source: §7.6.1 · p. 8, Table 6 · p. 8

7.6.2 Schema Linking

  • Schema linking reaches 74.74% precision and improves execution accuracy by 2.15%; without it, precision is 10.14% and execution accuracy is 57.95%.The ablation uses GPT-4o and reports only a slight decrease in recall with schema linking.Source: §7.6.2 · p. 8, Table 7 · p. 9

7.6.3 Candidate Generation and Selection

  • The three-candidate XiYan-SQL setup reaches 71.51% accuracy, and increasing the candidate count to five raises it to 73.34%.Removing the selection model and relying on self-consistency lowers performance by approximately three percentage points; removing fine-tuned generators, the ICL generator, or Refiner also lowers performance.Source: §7.6.3 · pp. 8–9, Table 8 · p. 9

8 Conclusion

XiYan-SQL combines supervised fine-tuning and in-context learning in a multi-generator ensemble, with candidate selection for NL2SQL. It reports state-of-the-art performance on several relational and non-relational benchmarks.

8 Conclusion

  • XiYan-SQL reports state-of-the-art performance on Spider, SQL-Eval, and NL2GQL.The conclusion identifies these as publicly available relational and non-relational databases.Source: §8 · p. 9
  • The framework combines supervised fine-tuning and in-context learning to generate high-quality, diverse SQL candidates.It uses two-stage, multi-task training for models with different preferences and a strategy to select the most reasonable candidate.Source: §8 · p. 9
  • XiYan-SQL’s results highlight its potential for high-quality NL2SQL generation on unseen samples from different distributions.Source: §8 · p. 9

A SQLite Example

This section presents an example of natural language to SQLite.

A SQLite Example

B PostgreSQL Example

This section presents an example of natural language to PostgreSQL.

B PostgreSQL Example

C NL2GQL Example

The appendix provides an NL2GQL example and examples of prompts for the ICL generator, candidate selection, and the refiner.

C NL2GQL Example

D ICL Generator Prompt

E Candidate Selection Prompt

F Refiner Prompt

Related papers

Loading 2411.08599v3…