Source-linked AI summary
Exploring Underexplored Limitations of Cross-Domain Text-to-SQL Generalization
Yujian Gan, Xinyun Chen, Matthew Purver
TL;DR
Cross-domain text-to-SQL models may fail on questions requiring rarely observed domain knowledge despite strong benchmark performance and related training examples. The paper defines five knowledge types, introduces the human-curated Spider-DK benchmark, and evaluates existing models on it. Performance drops dramatically on Spider-DK, and the results indicate that models do not reliably understand domain knowledge for prediction.
Problem
Existing text-to-SQL models generalize poorly when unseen-domain questions require domain knowledge that is infrequent in training data.
Method
The paper defines five domain-knowledge types and constructs Spider-DK by selecting Spider examples and manually modifying some to incorporate domain knowledge.
Results
Performance drops by about 20% to 30% from SpiderD to Spider-DK across all evaluated models.
Takeaways & Limitations
Improving model understanding of domain knowledge is an important direction for cross-domain text-to-SQL generalization.
Abstract
from arXiv · showhide
Recently, there has been significant progress in studying neural networks for translating text descriptions into SQL queries under the zero-shot cross-domain setting. Despite achieving good performance on some public benchmarks, we observe that existing text-to-SQL models do not generalize when facing domain knowledge that does not frequently appear in the training data, which may render the worse prediction performance for unseen domains. In this work, we investigate the robustness of text-to-SQL models when the questions require rarely observed domain knowledge. In particular, we define five types of domain knowledge and introduce Spider-DK (DK is the abbreviation of domain knowledge), a human-curated dataset based on the Spider benchmark for text-to-SQL translation. NL questions in Spider-DK are selected from Spider, and we modify some samples by adding domain knowledge that reflects real-world question paraphrases. We demonstrate that the prediction accuracy dramatically drops on samples that require such domain knowledge, even if the domain knowledge appears in the training set, and the model provides the correct predictions for related training samples.
1 Introduction
Cross-domain text-to-SQL models achieve strong benchmark accuracy but generalize poorly when questions require domain knowledge, even when related knowledge appears in training data.
- Recent systems exceed 70% accuracy on Spider and 90% on WikiSQL, suggesting strong benchmark performance.
- Follow-up studies report worse generalization when schema mentions are removed, schema words are replaced by synonyms, or databases change.
- Spider-DK evaluates whether models understand domain knowledge while keeping questions and SQL queries similar in distribution to training data.
- Models consistently fail when specific domain knowledge is required, even if that knowledge is moderately represented in training examples.
- University questions may require job-title and gender knowledge, whereas geography questions emphasize knowledge of places.
2 Spider-DK Dataset
Spider-DK is a curated Spider-derived benchmark designed to isolate domain-knowledge understanding through selected and manually modified questions, while preserving comparable SQL difficulty.
- Dataset construction: Spider-DK selects Spider development examples requiring domain knowledge and manually modifies some questions to simulate domain-specific user utterances.
- Dataset construction: The benchmark contains 535 NL-SQL pairs: 270 original pairs and 265 modified pairs incorporating domain knowledge.
- Dataset composition: Spider-DK preserves a SQL-hardness distribution close to Spider, with easy, medium, hard, and extra-hard examples comprising 20.6%, 41.8%, 14.8%, and 19.1%.
- Domain-knowledge motivation: Some domain knowledge is difficult to infer without in-domain examples, as shown by the GeoQuery interpretation of “major” as population > 150000.
- Domain-knowledge types: The dataset defines five domain-knowledge types, including omitted expressions, query-order inference, synonym substitution, boolean-like values, and max-related generation errors.
3 Experiments
Experiments show that existing text-to-SQL models generalize poorly to Spider-DK’s domain-knowledge challenges, despite strong performance on related Spider examples. Breakdown and error analyses identify model-specific knowledge handling differences and several causes of failure.
- 3.2 Main Results: All models’ exact match accuracy drops by about 20% to 30% from SpiderD to Spider-DK.Models are trained on the original Spider training set, while Spider-DK is designed from SpiderT examples with high exact-match performance.
- 3.2 Main Results: RAT-SQL + GAP outperforms RAT-SQL + BERT on Spider-DK, despite the reverse ordering on SpiderD.The result indicates that GAP facilitates better domain-knowledge understanding.
- 3.3 Performance on Knowledge Type Splits: RAT-SQL + GAP performs best across domain-knowledge examples and exceeds other models from T2 through T5.IRNet benefits from ConceptNet on T3, while GNN and RAT-SQL lack comparable extra knowledge components; ConceptNet does not generalize to other knowledge types.
- 3.4 Error Analysis: Error analysis attributes failures to rare domain knowledge, insufficient generalization across similar problems, and words serving both schema-item and SQL-structure roles.For example, imbalanced training data can associate “from old to young” with descending age order even when the relevant column is date of birth.
4 Conclusion
The paper introduces Spider-DK, a human-curated Spider-based dataset for evaluating text-to-SQL generalization involving domain knowledge. Its results show dramatic performance degradation even when the relevant knowledge appears in training, highlighting domain-knowledge understanding as an important cross-domain generalization gap.
- Spider-DK is a human-curated dataset based on Spider for evaluating text-to-SQL models’ understanding of domain knowledge.
- Existing text-to-SQL models’ performance drops dramatically on Spider-DK, even when the domain knowledge appears in the training set.
- Improving model understanding of domain knowledge is identified as an important direction for cross-domain text-to-SQL generalization.