PaperlayerView exact revision on arXiv

Source-linked AI summary

Robust Text-to-SQL Generation with Execution-Guided Decoding

Chenglong Wang, Kedar Tatwawadi, Marc Brockschmidt, Po-Sen Huang, Yi Mao, Oleksandr Polozov, Rishabh Singh

arXiv:1807.03100v3cs.CLcs.AIcs.DBcs.LGcs.PL

TL;DR

Natural-language users may be unable to query databases directly because doing so requires specialized SQL knowledge. The paper uses execution results from partially generated queries to guide decoding and filter faulty candidates. Across the evaluated models and datasets, the effect depends on model and integration; Coarse2Fine gains 5.4% WikiSQL execution accuracy, from 78.4% to 83.8%.

  • Problem

    Querying databases usually requires specialized SQL knowledge, limiting users’ access to the information they contain.

    Source: Introduction · p. 1

  • Method

    Execution guidance conditions decoding on partial-query execution results to filter candidates that cannot be completed to a correct query.

    Source: Introduction · p. 1, Execution-Guided Decoding · p. 2

  • Results

    Across four models and three datasets, impact depends on the model and integration; Coarse2Fine gains 5.4% WikiSQL execution accuracy, from 78.4% to 83.8%.

    Source: WikiSQL Experiments · p. 5, Conclusion · p. 7

  • Takeaways & Limitations

    Execution guidance increases semantically meaningful programs, but not all such programs are semantically correct.

    Source: Discussion & Analysis · p. 5

    Execution guidance is used only to filter predictions during inference; integrating it into training is left for future work.

    Source: Conclusion · p. 7

Abstract

from arXiv · show

We consider the problem of neural semantic parsing, which translates natural language questions into executable SQL queries. We introduce a new mechanism, execution guidance, to leverage the semantics of SQL. It detects and excludes faulty programs during the decoding procedure by conditioning on the execution of partially generated program. The mechanism can be used with any autoregressive generative model, which we demonstrate on four state-of-the-art recurrent or template-based semantic parsing models. We demonstrate that execution guidance universally improves model performance on various text-to-SQL datasets with different scales and query complexity: WikiSQL, ATIS, and GeoQuery. As a result, we achieve new state-of-the-art execution accuracy of 83.8% on WikiSQL.

Introduction

Natural-language questions can make database knowledge accessible without requiring users to know SQL. The paper introduces execution guidance, which uses partial-query execution during decoding, and reports gains across text-to-SQL models and datasets.

Introduction

  • SQL expertise remains a barrier to querying organizational databases, motivating natural-language interfaces to executable queries.Source: Introduction · p. 1
  • The method targets runtime errors and queries that produce no results, extending grammar-based sequence-to-tree approaches to semantic errors.Source: Introduction · p. 1
  • Execution guidance conditions decoding on partial-query execution to exclude candidates that cannot be completed to a correct SQL query.The approach can condition autoregressive decoders on non-differentiable partial execution results at appropriate timesteps.Source: Introduction · p. 1
  • Execution guidance improves results by 1%–6% over base models, and Coarse2Fine reaches 83.8% WikiSQL execution accuracy.The paper evaluates four models on WikiSQL, ATIS, and GeoQuery.Source: Introduction · p. 2

Execution-Guided Decoding

Execution-guided decoding runs executable partial SQL queries during generation and uses their results to remove candidates with execution errors or empty outputs. Checks can occur at intermediate decoding stages, with timing depending on the decoder.

Execution-Guided Decoding

Execution Errors

  • Runtime errors arise from operator–operand type mismatches, while empty outputs can result from overly restrictive predicates; empty output is treated as an error when queries are expected to return results.Source: Execution Errors · p. 2

Using Execution Guidance in Decoding

  • The decoder retains the top-k beam states without execution errors or empty outputs whenever a partial program is executable.Checks can be applied to partial programs rather than only after beam decoding; their frequency and stages depend on the decoder.Source: Using Execution Guidance in Decoding · p. 2

Base Models

The paper extends four prior text-to-SQL models spanning recurrent generative and template/slot-filling decoders. Their execution-guidance integrations reflect whether and when their generated programs can be executed.

Base Models

  • Four prior state-of-the-art models cover recurrent sequence-to-sequence decoders and template/slot-filling decoders.Source: Base Models · p. 3

Seq2Seq with Attention (Iyer et al. 2017)

Integrating Execution Guidance

Template-Based Model (Finegan-Dollak et al. 2018)

  • The template-based model predicts a template and slot-filling tokens separately; because their agreement is not enforced, guidance selects among joint-probability candidates without execution errors.Partial execution is unavailable because the model has no autoregressive recurrent decoder.Source: Template-Based Model (Finegan-Dollak… · pp. 3–4

Coarse2Fine (Dong and Lapata 2018)

  • Coarse2Fine first predicts a query sketch and then fills its slots; execution guidance checks aggregation choices and condition candidates during slot filling, backtracking to another sketch if needed.The sketch itself is not an executable partial program, so guidance is not applied at that stage.Source: Coarse2Fine (Dong and Lapata 2018) · p. 4

Experimental Results

Execution-guided decoding improves the evaluated models on WikiSQL, ATIS, and GeoQuery, with gains varying by model and dataset. Ablations indicate that guidance contributes most during condition generation.

Experimental Results

  • The evaluation uses the same pretrained models for baselines and execution-guided variants because guidance changes only the test-time generation procedure.The four models span generative and slot-filling families and integrate guidance to different degrees.Source: Experimental Results · p. 4

WikiSQL Experiments

  • 83.8% execution accuracy for Coarse2Fine on WikiSQL test is a new state of the art; Pointer-SQL improves from 71.9% to 78.3%.With beam size 5, Coarse2Fine improves from 78.4% to 83.8%.Source: Table 1 · p. 5, WikiSQL Experiments · p. 5

ATIS and GeoQuery Experiments

  • ATIS and GeoQuery include more complex SQL than WikiSQL, including multi-table queries, nesting, inner joins, and group-by operations.ATIS averages six tables per query and GeoQuery averages three.Source: ATIS and GeoQuery Experiments · p. 5
  • Execution-guided decoding improves test execution accuracy by 0.4% to 5% across the ATIS and GeoQuery model evaluations.The template-based model gains 5% on ATIS and 0.4% on GeoQuery; the sequence-to-sequence model gains 0.9% on ATIS and 2.5% on GeoQuery.Source: ATIS and GeoQuery Experiments · p. 5, Table 2 · p. 6

Discussion & Analysis

  • Execution errors drop 10% for Coarse2Fine on WikiSQL, alongside a 5.4% execution-accuracy gain; Seq2Seq on ATIS sees a 4.2% error drop but only a 0.9% accuracy gain.The authors attribute the large number of remaining ATIS errors to its more complex SQL queries and speculate that finer-grained partial-program checks may be more effective than posthoc filtering.Source: Table 3 · p. 6, Discussion & Analysis · p. 6, Table 4 · p. 6
  • Ablations show that execution guidance contributes little to aggregation-function choices but significantly to condition generation.The authors conclude that fine-grained guidance eliminating incorrect partial candidates at intermediate decoding steps is important.Source: Table 5 · p. 6, Discussion & Analysis · p. 6

Related Work

Prior semantic-parsing work includes neural models constrained to produce valid structures, training objectives that optimize sequence-level behavior, and neural-symbolic program synthesis. This paper instead applies symbolic partial-program execution to guide neural generation at inference time.

Semantic Parsing

  • Semantic parsing maps natural language to a logical form used for tasks including question answering and robot navigation.Liang (2016) surveys statistical semantic parsers under a unified framework.Source: Related Work · p. 6
  • Grammar- and tree-based decoding constrain output structure, while Dong and Lapata (2018) use a rough sketch to constrain a second decoding pass.Seq2Tree produces syntactically valid trees; grammar-based approaches use production rules as decoding constraints.Source: Related Work · pp. 6–7
  • WikiSQL broadened the use of SQL for semantic parsing, while standardized datasets such as ATIS and GeoQuery support evaluation on joins and nested queries.The paper uses the Finegan-Dollak et al. (2018) standardization of ATIS and GeoQuery.Source: Related Work · p. 7

Sequence-Level Objectives

  • Sequence-level objective methods incorporate additional objectives into training, whereas this work changes inference-time decoding and applies across different models.The paper contrasts its approach with reinforcement-learning objectives and sequence-level costs in beam search.Source: Related Work · p. 7

Program Synthesis

  • Neural-symbolic program synthesis uses probabilistic models to guide symbolic search; this work uses symbolic partial-program execution to guide neural program generation.The paper describes its execution-guidance idea as similar to neuro-symbolic synthesis but with the guidance direction reversed.Source: Related Work · p. 7

Conclusion

Execution-guided decoding improved four state-of-the-art models across three datasets, with Coarse2Fine reaching 83.8% accuracy on WikiSQL. This work used guidance only to filter predictions at inference time, while training-time integration and other tasks remain future directions.

Conclusion

  • 5.4% accuracy gain (78.4% to 83.8%) on the WikiSQL test dataset made execution-guided Coarse2Fine the new state of the art on this task.The paper reports that improvement for the Coarse2Fine model extended with an execution-guided decoder.Source: Conclusion · p. 7
  • Execution guidance improved four state-of-the-art models evaluated on three datasets, demonstrating practical utility across those experiments.The conclusion describes the approach as applicable to arbitrary autoregressive decoders at inference time.Source: Conclusion · p. 7
  • The improvement depends on the extended model and how closely execution guidance is integrated with its decoding procedure.Source: Conclusion · p. 7
  • This work used execution guidance only for inference-time filtering; applying it to other tasks or integrating it into training is proposed future work.The authors suggest that training-time decisions conditioned on partial execution results may further improve performance.Source: Conclusion · p. 7

Related papers

Loading 1807.03100v3…