Source-linked AI summary
RASAT: Integrating Relational Structures into Pretrained Seq2Seq Model for Text-to-SQL
Jiexing Qi, Jingyao Tang, Ziwei He, Xiangpeng Wan, Yu Cheng, Chenghu Zhou, Xinbing Wang, Quanshi Zhang, Zhouhan Lin
TL;DR
Text-to-SQL benefits from relational structures, but specialized architectures make it difficult to use large pretrained models. RASAT augments T5 self-attention with relation representations while retaining pretrained weights, and achieves state-of-the-art execution accuracy across three benchmarks. Its scope is limited by substantial computational requirements, 512-token truncation, and English-only experiments.
Problem
Existing relational structures improve text-to-SQL, but their specialized architectures make leveraging large pretrained models difficult.
Method
RASAT augments T5 encoder self-attention with unified relation representations, including coreference relations for multi-turn text-to-SQL, while retaining pretrained weights.
Results
RASAT achieves state-of-the-art execution accuracy across Spider, SParC, and CoSQL, including 75.5% EX on Spider, 52.6% IEX on SParC, and 37.4% IEX on CoSQL.
Takeaways & Limitations
RASAT provides a unified way to introduce diverse structural relations into a pretrained sequential T5 model for single-turn and multi-turn text-to-SQL.
Takeaways & Limitations
The method requires substantial computation, truncates source sequences to 512 tokens, affects about 3% of CoSQL training data, and is evaluated only in English.
Abstract
from arXiv · showhide
Relational structures such as schema linking and schema encoding have been validated as a key component to qualitatively translating natural language into SQL queries. However, introducing these structural relations comes with prices: they often result in a specialized model structure, which largely prohibits using large pretrained models in text-to-SQL. To address this problem, we propose RASAT: a Transformer seq2seq architecture augmented with relation-aware self-attention that could leverage a variety of relational structures while inheriting the pretrained parameters from the T5 model effectively. Our model can incorporate almost all types of existing relations in the literature, and in addition, we propose introducing co-reference relations for the multi-turn scenario. Experimental results on three widely used text-to-SQL datasets, covering both single-turn and multi-turn scenarios, have shown that RASAT could achieve state-of-the-art results across all three benchmarks (75.5% EX on Spider, 52.6% IEX on SParC, and 37.4% IEX on CoSQL).
1 Introduction
Text-to-SQL translates natural-language questions into SQL, but incorporating relational structures has often required specialized architectures that make leveraging large pretrained models difficult. RASAT augments T5 self-attention with unified relation representations and achieves state-of-the-art execution accuracy across single-turn and multi-turn benchmarks.
- Text-to-SQL translates natural-language questions into SQL queries and can lower barriers for nonexpert database users.
- Schema encoding, database-content relations, and other structural relations have improved generalization and performance in text-to-SQL.
- Specialized relational and tree-decoder architectures make it difficult to leverage large pretrained models for text-to-SQL.
- RASAT augments encoder self-attention while retaining pretrained T5 weights and unifies schema, syntactic, and other relations, including coreference for multi-turn tasks.
- 52.6% IEX on SParC and 37.4% IEX on CoSQL establish state-of-the-art multi-turn results for RASAT.
2 Related Work
Related work advances text-to-SQL through sketch-based methods, graph-structured relational modeling, conversational-context modeling, and pretrained seq2seq models. These approaches address query complexity, schema relationships, interaction history, or grammatical validity through different architectural choices.
- Sketch-based slot-filling methods decompose SQL generation into independent components but often fail on complex Spider queries.
- Graph structures encode complex relationships among database schema items, questions, and syntactic dependencies for multi-table SQL.
- Conversational text-to-SQL methods model historical interaction context using SQL editing, cross-turn schema graphs, tree reuse, state tracking, or auxiliary tasks.
- Fine-tuned T5-3B achieved competitive results, while PICARD improved generated SQL by filtering grammatically invalid beam-search sequences during inference.
3 Preliminaries
The task maps a natural-language question and database schema to an SQL query, with multi-turn inputs extending this formulation to question and query sequences. Relation-aware self-attention injects pairwise relation embeddings into key and value representations.
- 3.1 Task Formulation: Given a natural-language question Q and database schema S, the task predicts an SQL query Y.
- 3.1 Task Formulation: The schema contains tables, columns associated with each table, and database content V.
- 3.1 Task Formulation: In multi-turn text-to-SQL, Q and Y become sequences of questions and corresponding SQL queries, with prediction using the available interaction history.
- 3.2 Relation-aware Self-Attention: Relation-aware self-attention augments vanilla self-attention by introducing relation embeddings into key and value entries.
- 3.2 Relation-aware Self-Attention: The relation embeddings represent the relation between the i-th and j-th tokens and are used with learnable attention weights across heads.
4 RASAT
RASAT retains T5’s seq2seq architecture while replacing encoder self-attention with relation-aware self-attention over an interaction graph. It integrates diverse schema, question, coreference, and database-content relations at subword level while inheriting T5 parameters.
- Model architecture: RASAT uses T5 as its base model, replacing encoder self-attention with relation-aware self-attention.The added attention uses separate relation embeddings for key and value computations.
- Input representation: The encoder input serializes questions, database schema, database-content mentions, and delimiters, with previous questions prepended in multi-turn settings.History is truncated when the sequence reaches length 512.
- Parameter initialization: RASAT inherits all T5 parameters and randomly initializes only the additional relation embeddings, increasing parameters by less than 0.01%.The relation-aware encoder can therefore incorporate structured relations without replacing the pretrained seq2seq backbone.
- Interaction graph: RASAT represents relations as triplets over input tokens, forming an interaction graph with up to |X|^2 directed edges.Generic relations are reserved for token pairs without a specific edge, and the model uses 51 relation types overall.
- Relation types: The model incorporates schema encoding, schema linking, question dependency, cross-question coreference, and database-content mention relations.Coreference links connect expressions such as “their” and “they” to the earlier referent “students” across dialogue turns.
- Relation propagation: Relation propagation converts word- or phrase-level relations into dense subword-level connections compatible with pretrained tokenization.For example, a relation between two multi-token column names is replicated across their constituent subword pairs.
5 Experiments
Experiments evaluate RASAT across Spider, SParC, CoSQL, and Spider-Realistic, using official metrics, ablations, and comparisons with pretrained and relation-aware baselines. RASAT achieves state-of-the-art or competitive performance across these settings, with especially large gains in executable accuracy on multi-turn benchmarks.
- SParC: RASAT + PICARD achieves state-of-the-art results on all four SParC evaluation metrics.
- SParC: 52.6% IEX on the official SParC leaderboard improves over 21.6% by 31% absolute.
- CoSQL: 37.4% IEX on the official CoSQL leaderboard improves over 8.4% by 29% absolute.
- Spider and Spider-Realistic: 75.5% EX on the official Spider leaderboard establishes new state-of-the-art performance.
- Spider and Spider-Realistic: RASAT also achieves new state-of-the-art performance on Spider-Realistic, a variant that removes or paraphrases explicit column-name mentions.
- Ablations: Ablations examine SQL difficulty, T5 model size, and relation types, with larger performance gaps reported for RASAT on smaller pretrained models.
6 Conclusion
RASAT augments T5 with relation-aware self-attention to incorporate structural relations while retaining pretrained weights. It achieves state-of-the-art performance, especially on execution accuracy, across three text-to-SQL benchmarks.
- RASAT is a Relation-Aware Self-Attention-augmented T5 model for text-to-SQL generation.
- RASAT introduces various structural relations into the sequential T5 model while leveraging pretrained weights.The model augments encoder self-attention and adds new parameters rather than using only conventional fine-tuning or prompt tuning.
- RASAT achieves state-of-the-art performance across the three most common text-to-SQL benchmarks, especially on execution accuracy.
Limitation
The method has substantial computational and input-length constraints, and its evaluation is limited to English. Long inputs affect about 3% of CoSQL training data.
- The method consumes substantial computational resources because it uses the large T5-3B model.Training uses 8 A100 GPUs with 80G memory for around 2 days.
- The model truncates source sequences to 512 tokens, which may cause information loss for long inputs.About 3% of CoSQL training data is affected.
- The evaluation covers only English because English has richer analytical tools and resources than other languages.
A Model Size
RASAT adds only a very small number of parameters relative to an equivalently sized T5 model. Its added embedding matrices are shared across encoder layers and attention heads.
- The overall parameter increase is less than 0.01% compared with the same-size T5 model.RASAT adds two encoder embedding matrices containing 2×µ×d_kv parameters.
- The two added embedding matrices are shared in each encoder layer and each attention head.The relation count is µ = 51, while d_kv is 64 for T5-small/base/large and 128 for T5-3B.
B Output Comparation between T5 and Tree-based Decoder Model
Tree-based decoder models may output SQL placeholders instead of executable database values, whereas this comparison highlights a limitation relevant to execution-based evaluation.
- AST-tree-based decoder models often use the placeholder “value” instead of the real database value, such as “France.”Such outputs cannot be executed in a real database and fail execution-accuracy evaluation.
C Case Study
The case study compares RASAT-SQL with vanilla T5-3B on two three-turn SParC interactions. RASAT-SQL correctly handles omitted conditions and foreign-key-based schema access that T5-3B misses.
- RASAT-SQL recovers the condition “employees who are under age 30” that vanilla T5-3B neglects in Question #3.
- Both examples contain three turns, and RASAT-SQL predicts correctly in the cases shown while vanilla T5-3B does not.
- RASAT correctly accesses column course through a foreign key when the course_arrange table lacks a course column, whereas vanilla T5-3B fails.
D Relations Used in Experiment
The experiment uses a broad inventory of relational structures, totaling 51 relation types. The relation inventory is largely consistent with prior systems, and one listed relation identifies matched question items extracted from the question.
- The experiment uses 51 kinds of relations in total.
- Most relations used are consistent with those in RAT-SQL and LGESQL.
- The relation inventory includes V, the matched question item extracted from Q.