Source-linked AI summary
Towards Robustness of Text-to-SQL Models against Synonym Substitution
Yujian Gan, Xinyun Chen, Qiuping Huang, Matthew Purver, John R. Woodward, Jinxia Xie, Pengsheng Huang
TL;DR
The paper asks whether text-to-SQL models remain robust when users refer to schema elements with synonyms rather than exact lexical matches. It introduces Spider-Syn, a human-curated benchmark, and evaluates schema-annotation and adversarial-training defenses. Models degrade substantially under synonym substitution, while both defense categories improve robustness and schema-annotation methods are more effective.
Problem
Existing text-to-SQL models commonly rely on exact lexical matching between natural-language questions and table schemas, although real-world users may not know precise schema wording.
Method
The paper creates Spider-Syn by manually replacing schema-related words in Spider questions with synonyms and studies schema-annotation and adversarial-training approaches for improving robustness.
Results
Models trained on original Spider data show a significant performance drop on Spider-Syn, while both defense categories significantly improve robustness and schema-annotation methods are more effective.
Takeaways & Limitations
Robust text-to-SQL evaluation should include human-curated synonym paraphrases, and multiple schema annotations provide an effective defense without additional training.
Takeaways & Limitations
Spider-Syn is designed to study realistic paraphrasing rather than worst-case adversarial attacks, and its synonym substitutions do not guarantee exactly the same natural-language meaning as the original questions.
Abstract
from arXiv · showhide
Recently, there has been significant progress in studying neural networks to translate text descriptions into SQL queries. Despite achieving good performance on some public benchmarks, existing text-to-SQL models typically rely on the lexical matching between words in natural language (NL) questions and tokens in table schemas, which may render the models vulnerable to attacks that break the schema linking mechanism. In this work, we investigate the robustness of text-to-SQL models to synonym substitution. In particular, we introduce Spider-Syn, a human-curated dataset based on the Spider benchmark for text-to-SQL translation. NL questions in Spider-Syn are modified from Spider, by replacing their schema-related words with manually selected synonyms that reflect real-world question paraphrases. We observe that the accuracy dramatically drops by eliminating such explicit correspondence between NL questions and table schemas, even if the synonyms are not adversarially selected to conduct worst-case adversarial attacks. Finally, we present two categories of approaches to improve the model robustness. The first category of approaches utilizes additional synonym annotations for table schemas by modifying the model input, while the second category is based on adversarial training. We demonstrate that both categories of approaches significantly outperform their counterparts without the defense, and the first category of approaches are more effective.
1 Introduction
Text-to-SQL models often depend on exact lexical matches between questions and schemas, an assumption that may not hold in real-world use. The paper introduces Spider-Syn to evaluate synonym substitution and proposes defenses that improve robustness.
- Motivation: State-of-the-art models achieve around 70% accuracy on Spider, but their cross-domain generalization may rely on exact lexical matching.Spider and WikiSQL questions commonly state table and column names explicitly, creating strong correspondence with schemas.
- Motivation: Exact lexical matching is unrealistic because users may lack precise schema knowledge, making complex SQL queries tedious to formulate.The paper frames synonym substitution as a test of robustness when users refer to tables or columns differently.
- Dataset and study: Spider-Syn is a human-curated benchmark that modifies Spider questions by replacing schema-related words with manually selected synonyms while preserving the corresponding SQL.The dataset targets real-world paraphrases rather than only worst-case adversarial attacks.
- Findings: Models trained only on original Spider data suffer a significant performance drop on Spider-Syn when synonym substitution removes explicit question-schema correspondence.The benchmark is designed to evaluate robustness without changing the intended SQL query.
- Defenses: The paper evaluates schema-annotation and adversarial-training defenses, finding that both improve robustness while schema-annotation approaches are effective without additional training.The contributions emphasize using multiple schema annotations as a lower-resource alternative to adversarial training.
2 Spider-Syn Dataset
Spider-Syn is a human-curated Spider-based benchmark that replaces schema-related words and phrases with domain-appropriate synonyms to test robustness under realistic paraphrasing. Its construction uses controlled substitutions, manual review, and consistency-preserving dataset design.
- Dataset construction: Spider-Syn manually modifies Spider questions with synonyms for database-related schema items and cell values, targeting users who may not know table schemas.The benchmark contains 7000 training and 1034 development examples and no public test set.
- Construction principles: The benchmark avoids worst-case attacks by selecting relatively common, domain-appropriate substitutions that preserve SQL consistency.Substitutions may be valid only in a specific domain, such as replacing “flight number” with “flight code” in aviation.
- Construction principles: Spider-Syn covers both single-word and multiword substitutions, including changes guided by database contents and cell-value semantics.For example, “location” may be replaced by “city” when the column stores city names, while “dog” may become “puppy” although the SQL retains the database value “dog.”
- Annotation and quality control: Four computer-science graduate students performed manual annotation after domain separation and training, followed by cross-review and native-English-speaker review.Disagreements were discussed to consensus, and extracted substitution reports supported efficient review.
- Dataset statistics: 5672 questions were modified, including 5634 schema-item changes and 27 cell-value changes, using 273 synonymous words and 189 synonymous phrases.Across all examples, the average was 0.997 changes per question and 7.7 modified words or phrases per domain.
3 Defense Approaches
The paper addresses synonym substitution with two defense families: multiple schema annotations selected at inference time and adversarial training using generated substitutions. Multiple-annotation selection preserves compatibility with existing models without additional training, while adversarial training augments training data with targeted examples.
- 3.1 Multi-Annotation Selection (MAS): Multiple-annotation selection (MAS) adds alternative schema words, such as “nation” and “State” for “country,” so schema linking can match paraphrased questions.The method selects annotations appearing in the NL question and otherwise uses the default schema annotation.
- 3.1 Multi-Annotation Selection (MAS): MAS requires no additional training and can be applied to existing models trained without synonym-substitution questions.Multiple schema annotations may be created automatically or manually.
- 3.2 Adversarial training: Adversarial training iteratively generates synonym-substitution examples and trains RAT-SQL on the augmented dataset.The implementation uses BERT-Attack and the TextAttack framework while aiming to follow Spider-Syn’s principles rather than arbitrary perturbations.
- 3.2 Adversarial training: BERT-Attack uses the entire NL question to select context-sensitive substitutions, distinguishing meanings such as “head” in different phrases.Domain information is added to reduce ambiguity, and selected same-domain questions are used instead of schema annotations as that information.
- 3.2 Adversarial training: Adversarial example generation is restricted to schema-item and cell-value words and excludes reserved words.These constraints are intended to keep perturbations focused on database-related synonym substitution.
4 Experiments
The experiments show that synonym substitution substantially harms text-to-SQL models, primarily through schema-item matching failures. Multi-annotation selection and adversarial training improve robustness, with ManualMAS strongest on Spider-Syn and annotation-based defenses requiring less training.
- Models Trained on Spider: Models trained on Spider lose about 20% to 30% performance on Spider-Syn, despite its synonyms not targeting worst-case model attacks.Training on Spider alone is insufficient because it contains few synonym-substitution questions.
- Models Trained on Spider: Schema-item matching accounts for most degradation on Spider-Syn, while KEY-WORDS and AND/OR components show only marginal declines.This pattern matches Spider-Syn’s focus on substituting schema-item words.
- Comparison of Different Approaches: SPRSYN improves Spider-Syn accuracy by 11.7% with a 1.9% Spider drop, whereas ADVBERT improves Spider-Syn accuracy by 10.3%.ManualMAS performs best on Spider-Syn because its annotations contain the dataset’s synonym substitutions.
- Worst-Case Attacks: Under worst-case attacks, AutoMAS best defends ADVGLOVE, while ADVBERT training performs best against ADVBERT attacks.AutoMAS covers the substitutions generated by ADVGLOVE; the attacks reduce baseline accuracy by 31.7% and 20.9%, respectively.
- Further Discussion on MAS: MAS consistently improves synonym-substitution robustness across base models, and AutoMAS also improves performance against adversarial attacks.The authors attribute poor ADVGLOVE defense to context-sensitive BERT embeddings that can redirect schema linking toward an incorrect item.
- Further Discussion on MAS: MAS requires no additional training and can be combined with existing defenses, allowing one pretrained model to use different schema annotations for different robustness requirements.The same pattern appears when MAS is combined with GNN and IRNet.
5 Related Work
Related work covers text-to-SQL benchmarks, data augmentation and adversarial training for text-to-SQL, and synonym substitution in broader NLP research. The paper positions its study within these strands by examining synonym-based robustness for text-to-SQL models.
- Text-to-SQL Translation: Text-to-SQL research uses benchmarks ranging from earlier single-domain datasets to large-scale cross-domain datasets such as WikiSQL and Spider.Spider targets synthesis of more complex SQL queries than single-table settings.
- Data Augmentation for Text-to-SQL: Prior text-to-SQL work has explored data augmentation and adversarial training, including methods aimed at improving cross-domain generalization.The cited AugmentGAN approach supports single-table SQL queries such as those in WikiSQL.
- Synonym Substitution for Other NLP Problems: Synonym substitution has been used for data augmentation and adversarial attacks across NLP, with several attacks successfully compromising existing models.The TextAttack framework integrates many such methods for easier use.
6 Conclusion
The paper introduces Spider-Syn to evaluate text-to-SQL robustness to synonym substitution and proposes multi-annotation selection and adversarial training as defenses. Both approaches are effective, with the paper demonstrating improved robustness against synonym-based changes and attacks.
- Conclusion: Spider-Syn is a human-curated dataset based on Spider for evaluating text-to-SQL robustness to synonym substitution.The dataset modifies schema-related language to test paraphrasing beyond exact lexical correspondence.
- Conclusion: The study finds dramatic performance drops on Spider-Syn and other synonym-substitution attacks, then evaluates multi-annotation selection and adversarial training as remedies.The conclusion presents both categories as effective approaches for improving robustness.