PaperlayerView exact revision on arXiv

Source-linked AI summary

Natural SQL: Making SQL Easier to Infer from Natural Language Specifications

Yujian Gan, Xinyun Chen, Jinxia Xie, Matthew Purver, John R. Woodward, John Drake, Qiaofu Zhang

arXiv:2109.05153v1cs.CL

TL;DR

Text-to-SQL translation must bridge a mismatch between natural-language descriptions and SQL, particularly for complex Spider queries. The paper introduces NatSQL, an SQL intermediate representation that simplifies query structure and schema linking while preserving SQL functionality. NatSQL improves several existing models, outperforms other IRs, and achieves state-of-the-art execution accuracy on Spider, with exact-match effects from equivalent representations remaining a limitation.

  • Problem

    Complex Spider queries expose a mismatch between natural-language intent and SQL details, while previous intermediate representations can be complicated or have limited SQL coverage.

  • Method

    NatSQL simplifies SQL by removing difficult clauses, nested subqueries, and set operators while reducing schema items needed for linking and retaining core SQL functionality.

  • Results

    NatSQL consistently improves several neural architectures on Spider, outperforms SQL and other IRs, and enables state-of-the-art execution accuracy.

  • Takeaways & Limitations

    Designing intermediate representations that better align query structure with natural-language descriptions is a promising direction for improving text-to-SQL reasoning.

  • Takeaways & Limitations

    Exact-match accuracy can drop when NatSQL represents equivalent SQL differently, and Spider’s evaluation process may negatively affect predictions for extra-hard queries.

Abstract

from arXiv · show

Addressing the mismatch between natural language descriptions and the corresponding SQL queries is a key challenge for text-to-SQL translation. To bridge this gap, we propose an SQL intermediate representation (IR) called Natural SQL (NatSQL). Specifically, NatSQL preserves the core functionalities of SQL, while it simplifies the queries as follows: (1) dispensing with operators and keywords such as GROUP BY, HAVING, FROM, JOIN ON, which are usually hard to find counterparts for in the text descriptions; (2) removing the need for nested subqueries and set operators; and (3) making schema linking easier by reducing the required number of schema items. On Spider, a challenging text-to-SQL benchmark that contains complex and nested SQL queries, we demonstrate that NatSQL outperforms other IRs, and significantly improves the performance of several previous SOTA models. Furthermore, for existing models that do not support executable SQL generation, NatSQL easily enables them to generate executable SQL queries, and achieves the new state-of-the-art execution accuracy.

1 Introduction

Text-to-SQL models face a mismatch between natural-language intent and SQL’s implementation details, especially on Spider’s complex queries. NatSQL addresses this by simplifying SQL structure while preserving coverage and improving model performance.

  • 1 Introduction: Spider exposes limitations of WikiSQL-oriented models because it contains complex and nested SQL queries.Previous models designed for WikiSQL experience a significant performance drop on Spider.
  • 1 Introduction: Previous intermediate representations remain either complicated, incomplete, or mismatched with natural-language descriptions.The paper motivates improving the representation rather than relying only on direct SQL generation.
  • 1 Introduction: In the motivating example, NatSQL aligns more closely with the question by avoiding an unnecessary INTERSECT distinction for the word “and.”The resulting translation is shorter and simpler than the alternatives.
  • 1 Introduction: NatSQL removes difficult SQL operators and clauses, including GROUP BY, HAVING, FROM, and JOIN ON, while preserving core SQL functionality.Its design also removes the need for nested subqueries and set operators.
  • 1 Introduction: NatSQL improves existing text-to-SQL models on Spider and enables RAT-SQL+GAP to achieve state-of-the-art execution accuracy.The comparison includes SQL and other intermediate representations.

2 Review: Text-to-SQL Paradigm

Text-to-SQL systems separately generate SQL keywords and schema items, using linking signals between question tokens, grammar elements, and database schemas. The review motivates intermediate representations that reduce this mismatch and schema-linking burden.

  • 2 Review: Text-to-SQL Paradigm: Text-to-SQL models generate SQL keywords using similarity linking between question hidden states and production-rule embeddings.The example links “less” with the SQL “<” keyword.
  • 2 Review: Text-to-SQL Paradigm: SQL’s GROUP BY and JOIN ON clauses can be difficult to predict because they often have no explicit counterparts in natural-language questions.Intermediate representations can remove such implementation details from prediction.
  • 2 Review: Text-to-SQL Paradigm: NatSQL retains SELECT, WHERE, and ORDER BY while dispensing with GROUP BY, HAVING, FROM, JOIN ON, set operators, and subqueries.Its grammar uses table and column forms, operators, ordering, and conjunction rules to retain functionality.
  • 2 Review: Text-to-SQL Paradigm: Schema linking is essential because models must identify tables and columns mentioned in the question and sometimes infer unmentioned schema items.Removing schema linking causes a dramatic performance decrease in cited ablation studies.

3 NatSQL

NatSQL simplifies SQL while preserving broad SQL coverage by removing difficult-to-align clauses, simplifying set operators and subqueries, and retaining SQL-like syntax. Its examples show improved compatibility over some prior intermediate representations and a closer alignment with natural-language structure.

  • 3 NatSQL: NatSQL retains SELECT, WHERE, and ORDER BY while dispensing with GROUP BY, HAVING, FROM, JOIN ON, set operators, and subqueries.Deleted-clause functionality is implemented through new keywords and conjunct placement before WHERE conditions.
  • 3 NatSQL: NatSQL’s main design principle is to simplify SQL structure and bring its grammar closer to natural language.The approach removes implementation details such as INTERSECT that are rarely mentioned in questions.
  • 3 NatSQL: NatSQL improves over SemQL by supporting a wider range of SQL queries, simplifying set operators, reducing schema items, and retaining SQL-compatible syntax.The paper also reports that NatSQL can improve text-to-SQL models by simplifying set operators and nested subqueries.
  • 3.3 Scalability of NatSQL: SemQL cannot convert the Figure 2 query because it misses the has_pet table, whereas RAT-SQL and NatSQL can represent the needed multi-table structure.NatSQL uses a WHERE condition operator join for otherwise unremovable JOIN ON clauses.
  • 3.4.1 Simplifying Queries with Set Operators: NatSQL unifies ambiguous natural-language OR cases into one operator that converts to UNION when following conditions cannot be concatenated.Conversion is triggered by higher precedence or conditions that are disconnected or disjoint, including conditions from different tables.
  • 3.4 NatSQL for SQL Keyword Generation: NatSQL removes subquery brackets by representing a subquery’s required SELECT column as the right column of a WHERE condition.This representation supports SQL conversion without retaining a complete nested SELECT clause.

3.5 How NatSQL Help Schema Item Generation

NatSQL simplifies schema item generation by replacing difficult-to-infer columns with table-only references or a placeholder, keeping generated columns tied to the question. It also makes executable SQL generation easier by restricting value slots to WHERE conditions and copying values in appearance order.

  • 3.5 How NatSQL Help Schema Item Generation: NatSQL replaces difficult-to-infer columns with a table-only reference or @, so its generated columns are mentioned in the natural-language question.In the example, visitor.id becomes @ and visit.visitor_id becomes visit.*.
  • 3.5 How NatSQL Help Schema Item Generation: @ marks a column inferred in Cond_L, while table.* represents a table whose target column is inferred using the surrounding schema context.Algorithm 1 uses the tables before @ and the table next to @ to infer replacement columns.
  • 3.6 Executable SQL Generation: NatSQL makes condition-value generation easier because its value slots occur only in the WHERE clause.Previous approaches may require searching slots across WHERE, subqueries, set operators, and HAVING clauses.
  • 3.6 Executable SQL Generation: NatSQL copies possible numeric or database-cell values from the question in order of appearance without training, enabling executable SQL generation for models built for exact-match metrics.This avoids confusion among multiple value slots illustrated by the Figure 5 example.

4 Experiments

Experiments on Spider compare NatSQL with SQL and other intermediate representations using gold representations and existing neural models. NatSQL consistently improves prediction and executable SQL generation, achieving a 2.2% absolute execution-accuracy improvement over the strongest counterparts.

  • 4.2.1 Gold IRs: NatSQL’s gold representation converts to more gold SQL than SemQL because it handles otherwise unremovable JOIN ON clauses and is more compatible with subqueries.SemQL is designed for exact-match evaluation and cannot directly generate executable SQL.
  • 4.2.1 Gold IRs: NatSQLG simplifies queries further with only 0.6% execution-accuracy degradation while enabling better model prediction performance than the SQL-like RAT-SQL representation.NatSQLG outperforms NatSQL on gold queries, but the gap becomes small when the representations are used by models.
  • 4.2.2 IRs for Prediction: NatSQL consistently outperforms SemQL across four model architectures, including IRNet, under the controlled evaluation setup.The reported IRNet+SemQL exact-match accuracy is 51.8% after excluding 1,659 additional training samples for all models.
  • 4.2.2 IRs for Prediction: NatSQL enables executable SQL generation for models that do not directly support it, with execution-match accuracy generally improving alongside exact-match accuracy.IRNet’s execution-match accuracy is slightly lower than its exact-match accuracy because it does not predict DISTINCT, which exact match does not check.
  • 4.2.2 IRs for Prediction: NatSQL improves extra-hard SQL accuracy by an average of 4.74% absolute across models, while also increasing accuracy on most samples at other difficulty levels.The largest gains align with extra-hard queries containing set operators or subqueries, which NatSQL simplifies.
  • 4.3 Overall Performance Analysis: 2.2% absolute execution-accuracy improvement over the best counterparts establishes a new Spider execution benchmark state of the art for RAT-SQL+GAP with NatSQL.This result is reported on the Spider test set against models at the top of the leaderboard.

5 Related Work

Related work spans natural-language database interfaces, cross-domain text-to-SQL benchmarks, and intermediate representations designed to simplify SQL generation.

  • Spider introduced complex and nested cross-domain SQL evaluation, exposing substantial performance drops for models designed for WikiSQL.
  • Cross-domain text-to-SQL remains challenging across benchmarks including WikiSQL and Spider, motivating continued model development.
  • Previous SQL intermediate representations either remain complicated or cover only limited SQL structures, leaving mismatch between natural-language intent and SQL implementation.
  • Earlier intermediate representations removed selected clauses or combined conditions, but NatSQL further simplifies SQL while moving closer to natural-language descriptions.

6 Conclusion

The paper concludes that NatSQL reduces schema-linking and structural difficulty while improving text-to-SQL prediction and executable SQL generation on Spider.

  • NatSQL consistently improves several neural architectures and achieves state-of-the-art performance on the challenging Spider benchmark.
  • NatSQL simplifies SQL structure and reduces the difficulty of schema linking, making it easier to infer from natural-language specifications.
  • NatSQL enables existing models to generate executable SQL queries without modifying their architectures.

A Further Discussion on Set Operators

NatSQL represents set-operator cases with simpler structures and can reduce schema-item predictions, but some WHERE and HAVING conditions cannot be concatenated by OR.

  • NatSQL can simplify INTERSECT and EXCEPT cases, including set operators that function as conditions after WHERE.
  • NatSQL requires predicting the cartoon table rather than the cartoon.channel column, avoiding a harder and potentially ambiguous column prediction.
  • Some conditions cannot be concatenated because a WHERE condition cannot be combined with a HAVING condition using OR.

B Further Discussion on Executable SQL Generation

NatSQL’s executable generation benefits from training data whose condition ordering follows the order in which conditions appear in natural-language questions.

  • Training data is central because NatSQL expands examples with multiple WHERE conditions ordered consistently with their natural-language mentions.
  • NatSQL queries in several figures contain more WHERE conditions than SQL and other intermediate representations.

C Gold NatSQL Error Analysis

NatSQL’s remaining errors concentrate in GROUP BY and IUE matching, while some apparent errors arise from exact-match evaluation rather than semantic differences.

  • C Gold NatSQL Error Analysis: NatSQL’s main errors come from GROUP BY and IUE matching, although many GROUP BY mismatches do not affect execution results.The analysis reports partial matching F1 across SQL components and distinguishes conversion errors from execution-impacting errors.
  • C Gold NatSQL Error Analysis: NatSQL supports only one IUE operator per query, which causes its IUE errors.
  • C Gold NatSQL Error Analysis: Spider exact-match evaluation can mark semantically equivalent subqueries as wrong when their FROM and JOIN ordering differs from the gold SQL.The cited example shows equivalent subqueries with T1 JOIN T2 versus T2 JOIN T1 being treated differently.

D SQL, SemQL and NatSQL Examples

Table 11 presents examples comparing SQL, SemQL, and NatSQL from Spider, including a question about countries without a single car maker.

  • D SQL, SemQL and NatSQL Examples: One example asks which countries have no single car maker.
  • D SQL, SemQL and NatSQL Examples: Table 11 presents comparative SQL, SemQL, and NatSQL examples from the Spider benchmark.
  • D SQL, SemQL and NatSQL Examples: The examples are presented to illustrate how the three representations handle Spider questions.
Loading 2109.05153v1…