PaperlayerView exact revision on arXiv

Source-linked AI summary

On the Potential of Lexico-logical Alignments for Semantic Parsing to SQL Queries

Tianze Shi, Chen Zhao, Jordan Boyd-Graber, Hal Daumé III, Lillian Lee

arXiv:2010.11246v1cs.CLcs.AI

TL;DR

Logical-form datasets support supervised semantic parsing, but they do not specify fine-grained links between question phrases and logical-form tokens. SQUALL adds manual SQL forms and lexical alignments to 11,276 WikiTableQuestions examples and uses them in two training strategies; in 5-fold cross validation, the additional supervision improves execution accuracy by 4.4%.

  • Problem

    Logical forms alone do not identify fine-grained relationships between individual words or phrases and logical-form tokens.

    Source: §1 · p. 1

  • Method

    SQUALL enriches 11,276 WikiTableQuestions examples with manual SQL logical forms and lexical alignments, used for supervised attention and auxiliary column prediction.

    Source: §1 · pp. 1–2, §8 · p. 9

  • Results

    In 5-fold cross validation, additional fine-grained supervision improves execution accuracy by 4.4% over models supervised with only logical forms.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors conclude that lexical alignments are more cost-effective for improving model accuracy than having only logical forms.

    Source: §8 · p. 9

    Machine-induced attention distributions are not intended for direct human interpretation.

    Source: §6 · p. 8

Abstract

from arXiv · show

Large-scale semantic parsing datasets annotated with logical forms have enabled major advances in supervised approaches. But can richer supervision help even more? To explore the utility of fine-grained, lexical-level supervision, we introduce Squall, a dataset that enriches 11,276 WikiTableQuestions English-language questions with manually created SQL equivalents plus alignments between SQL and question fragments. Our annotation enables new training possibilities for encoder-decoder models, including approaches from machine translation previously precluded by the absence of alignments. We propose and test two methods: (1) supervised attention; (2) adopting an auxiliary objective of disambiguating references in the input queries to table columns. In 5-fold cross validation, these strategies improve over strong baselines by 4.4% execution accuracy. Oracle experiments suggest that annotated alignments can support further accuracy gains of up to 23.9%.

1 Introduction

SQUALL adds expert-produced SQL logical forms and lexical alignments to 11,276 WikiTableQuestions examples, enabling experiments with finer-grained supervision. The paper tests two alignment-based training strategies and reports improved accuracy, alongside evidence that alignments are cost-effective to annotate.

1 Introduction

  • SQUALL enriches an 11,276-instance, SQL-translatable subset of WikiTableQuestions training data with manually created logical forms and lexical alignments.The annotations pair question tokens such as “how many” with corresponding SQL fragments such as COUNT(. . .).Source: §1 · pp. 1–2
  • The paper proposes supervised attention to match human alignment judgments and column prediction to identify which table column a question fragment refers to.The two strategies use lexical alignments to supervise encoder-decoder models.Source: §1 · p. 2
  • 4.4% execution-accuracy improvement over models supervised with only logical forms in 5-fold cross validation; oracle models improve absolute logical form accuracy by up to 23.9%.Using BERT features, models reach 54.1% execution accuracy on WTQ test, surpassing the previous weakly supervised state of the art at 48.8%.Source: §1 · p. 2
  • Lexical alignments take less than half the time to annotate as a logical form, and aligning 5% of training logical forms improves execution accuracy by 2.5 percentage points.The authors conclude that lexical alignments are cost-effective for training parsers.Source: §1 · p. 2
  • The work contributes a semantic-parsing dataset with manual logical forms and alignments, plus two training strategies that use those alignments.The paper describes this as the first large-scale semantic-parsing dataset with manual lexical-to-logical alignments.Source: §1 · pp. 1–2

2 Task: Table-based Semantic Parsing

Table-based semantic parsing maps a question and table to a logical form that can reproduce the target form or, when executed, yield the correct answer. The section contrasts weak and supervised training and identifies the missing input-to-output correspondences that make supervised parsing challenging.

2 Task: Table-based Semantic Parsing

  • A semantic parser receives a question and its table, and aims to reproduce the target logical form or generate a query that returns the correct answer when executed.The former is measured by logical-form accuracy; the latter by execution accuracy.Source: §2 · p. 2
  • Weakly supervised examples pair inputs with answers, whereas supervised examples pair inputs with logical forms.The section notes that supervised models are more accurate than weakly supervised ones.Source: §2 · p. 2
  • Even with logical-form supervision, training is challenging because both input and output are structured, and examples do not specify which input parts generate which output parts or how they combine.The paper introduces SQUALL to explore whether finer-grained supervision can help.Source: §2 · p. 2

3 SQUALL: Our New Dataset

SQUALL extends WikiTableQuestions with manually written SQL queries and lexical alignments between question fragments and SQL fragments. The section describes how examples were annotated and filtered, then outlines the encoder-decoder base model.

3 SQUALL: Our New Dataset

  • SQUALL starts from compositional questions over Wikipedia tables and adds SQL equivalents and alignments between question tokens and SQL fragments.The dataset is based on WikiTableQuestions; the paper leaves alignments for other text-to-SQL datasets and cross-dataset generalization to future work.Source: §3 · pp. 2–3

3.1 Data Annotation

  • SQUALL provides SQL for 11,468 questions, while 18.9% were excluded for non-deterministic answers, extra preprocessing needs, or insufficient SQL expressiveness.After post-processing, the dataset contains 11,276 table-question-answer triplets with logical forms and lexical alignments.Source: §3.1 · p. 3, §3.2 · p. 4
  • Annotators aligned question and logical-form fragments as units that could be non-contiguous, while leaving some tokens unaligned.The annotation covers 49.8% of question tokens and 90.2% of SQL keywords outside the basic SELECT…FROM…WHERE structure; implicit equalities and conjunctions often remain unaligned.Source: §3.1 · p. 3
  • SQL annotation took an average of 33.9 seconds per query, with alignment adding 15.0 seconds.After discussion, the annotators agreed on 94.5% of data instances and 93.3% of alignments.Source: §3.1 · p. 4

3.2 Post-processing

  • The base model is an attention-and-copying sequence-to-sequence system that generates SQL tokens conditioned on the input and prior output tokens.It represents questions and table columns with LSTMs or BERT features and copies column names or literal strings from the input.Source: §3.2 · pp. 4–5

5 Using Alignments in Model Training

The section uses manual alignments to supervise model attention and train an auxiliary question-token-to-column prediction task. Oracle experiments show that replacing induced attention with manual alignments can substantially raise logical-form accuracy.

5.1 Supervised Attention

  • The full model combines sequence-to-sequence, supervised-attention, and column-prediction losses.The alignments supervise attention weights to approximate human judgments, using Euclidean distance; multi-element alignments use a uniform target distribution.Source: §5 · p. 5, §5.1 · p. 5

5.2 Oracle Experiments with Manual Alignments

  • Up to 23.9% absolute higher logical-form accuracy over the base model is achieved with oracle access to manual alignments.The experiment replaces attention weights with true alignment vectors during training and inference.Source: §5.2 · p. 5, Table 2 · p. 5

5.3 Column Prediction

  • Manual alignments also support an auxiliary task that predicts the table column corresponding to each question token.The column-prediction loss is cross-entropy over tokens annotated as matching columns.Source: §5.3 · p. 6
  • Exact-match features provide an automatically generated, lower-coverage alternative to manual alignment supervision.The baseline features mark question-token matches to column headers or table cells and column-name matches in the question.Source: §5.3 · p. 6

6 Experiments

Across five shuffled table splits, the experiments compare ALIGN with SEQ2SEQ+ using logical-form and execution accuracy; they examine alignment gains, model variants, and generalization.

WTQ Test Results

  • BERT-feature models trained with ALIGN exceed prior state-of-the-art on WTQ test execution accuracy by 5.3%.The paper identifies the benefit of additional lexical supervision, rather than these test-set numbers, as its main scientific interest.Source: §6 · p. 6

Effect of Alignment Annotations

  • ALIGN improves over SEQ2SEQ by 2.3% with BERT and 3.1% without, with Table 4 reporting logical-form and execution accuracies on dev and test sets.Both models have logical-form supervision, while ALIGN additionally uses lexical alignments.Source: §6 · p. 6, Table 4 · p. 6

Effect of Individual Strategies

  • Supervised encoder attention is the most effective individual strategy; each ALIGN component improves accuracy over SEQ2SEQ+, and removing exact-match column features lowers ACCEXE by 8.1%.Table 5 compares model variations without BERT; the oracle attention model illustrates unrealized alignment potential.Source: §6 · p. 7, Table 5 · p. 7

Effect of Annotation Availability

  • Aligning only 5% of the logical forms yields more than half of ALIGN’s accuracy improvement, while ALIGN reaches SEQ2SEQ+ accuracy levels with about half as many training examples.Alignment annotation takes less than half the time required to annotate a logical form.Source: §6 · p. 7, Figure 2 · p. 8

Where Do Our Models Improve the Most?

  • ALIGN gains more over SEQ2SEQ+ on column selection (+4.9%) than on correct SQL-template generation (+2.0%), with larger gains on complex templates.Table 6 reports logical-form, template, and column accuracies, with parenthetical deltas relative to baseline.Source: §6 · p. 7, Table 6 · p. 7

Do Our Models Generalize Better to Unseen Query Templates?

  • On unseen SQL templates, ALIGN is more accurate than SEQ2SEQ+ by a large margin; Table 7 averages results over the 10 most frequent templates.The evaluation excludes a template from training before testing on it.Source: §6 · p. 7, Table 7 · p. 7

Are the Induced Attention Weights Similar to Manual Alignments?

  • Without explicit supervision, attention concentrates on a few items that usually differ from manual alignments; supervised decoder attention also yields q2c attention resembling human alignments.The authors caution that machine-induced attention distributions are not intended for direct human interpretation.Source: §6 · pp. 7–8, Table 8 · p. 8

Qualitative Analysis

  • With little textual overlap between a question and its referred column, ALIGN can select the correct “Serial Name” column for “show,” where SEQ2SEQ+ selects “Co-Star.”The example illustrates a case where additional supervision helps column selection.Source: §6 · p. 8

7 Related Work

Related work situates the paper’s manual alignments among attention supervision, lexicalized semantic parsing, annotator rationales, and text-to-SQL datasets.

Attention and Alignments

  • Attention supervision has helped machine translation, image captioning, and visual question answering, while semantic parsing work used exact-match heuristics or induced latent alignments; this paper trains with manual alignments.The cited examples include machine translation (Liu et al., 2016; Mi et al., 2016), image captioning (Liu et al., 2017), visual question answering (Gan et al., 2017), and semantic parsing (Rabinovich et al., 2017; Wang et al., 2019).Source: §7 · p. 8

Lexical Focus and Semantic Parsing

  • The annotations resemble semantic lexicons used in lexicalized-grammar semantic parsing, but their type-flexible alignments leave compositional modeling as future work.The passage names syntax-based alignment as one possible direction.Source: §7 · p. 8

Annotator Rationales

  • Annotator rationales provide evidence alongside labels in some reading-comprehension datasets; the paper says its alignments could, at a stretch, count as rationales for SQL.Dua et al. (2020) report improved accuracy for a given annotation budget on machine reading comprehension.Source: §7 · p. 9

Text-to-SQL Datasets

  • Unlike WikiSQL’s limited templates, WTQ contains more diverse semantics and logical operations; extending lexical alignments to Spider’s more complex queries is left for future work.Spider includes a higher percentage of nested queries and multiple-table joins.Source: §7 · p. 9

8 Conclusion

SQUALL provides a large-scale semantic parsing dataset with SQL queries and lexical alignments, which the paper uses for finer-grained model supervision. The authors report accuracy benefits and identify more effective use of the annotations as a future challenge.

8 Conclusion

  • SQUALL adds manually produced SQL queries and lexical alignments to WTQ questions, enabling supervised attention and column prediction in encoder-decoder models.The authors describe it as the first large-scale semantic-parsing dataset with both hand-produced logical forms and manually derived lexical alignments.Source: §8 · p. 9
  • The oracle studies show substantial unrealized potential for the annotations, leaving more effective use of lexical alignments as a future research challenge.The conclusion points to further work using the annotations more effectively.Source: §8 · p. 9
  • The annotation-cost analysis finds that collecting additional lexical alignments is more cost-effective for accuracy improvement than using only logical forms.The authors hope the findings inform future dataset design and extensions of existing datasets, and identify cross-dataset/domain evaluation as a future direction.Source: §8 · p. 9

A Model Implementation Details

The appendix describes the model’s feature and training setup, compares its baseline with a state-of-the-art parser, and specifies SQL and alignment annotation rules.

A Model Implementation Details

  • The BERT setup fine-tunes BERTbase with 768-dimensional final-layer representations, while the encoder and decoder LSTMs have 128 hidden units and two layers.Training uses batch size 8, dropout 0.3, Adam at learning rate 0.001, and early stopping; hyperparameters are not tuned.Source: A Model Implementation Details · p. 12

B Comparison of Our Baseline Model with a State-of-the-Art Text-to-SQL Parser

C Annotation Guidelines

  • Annotation guidelines favor semantically faithful but simpler SQL, broad semantic alignments, and exact lexical matches when available.For alignments, they prioritize bare nouns over full noun phrases in many cases, but include “the” in superlatives; they generally avoid relying on table order unless the question explicitly asks about presentation order.Source: General Rules · pp. 12–13
  • For counts, annotators use DISTINCT only with clear linguistic cues and prefer COUNT(col) when a question offers a column match.For max/min rows, the guidelines generally use ORDER BY with LIMIT 1, and align performance words to result/time columns, then rank, then id if needed.Source: Some Frequent Specific Cases · p. 13

D Database Construction

The database construction assigns table columns to a defined set of data types and transforms composite values and aggregate rows for SQL processing.

D Database Construction

  • WTQ tables are assigned nine basic data types, including numbers, dates and times, scores, spans, addresses, and raw text.The types also include numbers with units, time spans, and fractions.Source: D Database Construction · p. 13
  • Binary tuples are split into subcolumns, while lists are moved to separate tables joined to the original through primary-foreign key relations.Source: D Database Construction · p. 13
  • Column types are identified with regular expressions and manually verified; out-of-set types become raw text, and aggregation rows are filtered out.Source: D Database Construction · p. 13

E Additional Alignment Data Statistics

The appendix reports which token types align most often with SQL, compares attention-loss choices, and tests automatic versus manual alignments.

E Additional Alignment Data Statistics

  • Comparative and superlative adjectives and adverbs align most often with SQL, while punctuation, wh-determiners, auxiliary verbs, existential “there,” and pronouns align least often.The table notes that highly aligned comparative and superlative forms correspond to operations such as MAX.Source: E Additional Alignment Data Statistics · p. 13, Table E2 · p. 14

F Different Loss Functions for Supervised Attention

G ALIGN Trained with Heuristically-Generated Alignments

  • Manual alignments provide an additional +1.7% ACCEXE over automatic alignments, which already improve on SEQ2SEQ+.The authors attribute the greater informativeness of manual annotations to column mentions without lexical overlap, such as “who” and “athlete.”Source: Table G4 · p. 14, G ALIGN Trained with… · p. 14

H Template-based Evaluation

On the ten most frequent templates, ALIGN’s gains are more pronounced for column accuracy than template accuracy, especially on complex templates.

H Template-based Evaluation

I Evaluation Results on Unseen SQL Templates

When evaluated on SQL templates excluded from training, ALIGN outperforms SEQ2SEQ+ in execution accuracy on most of the ten most frequent templates.

I Evaluation Results on Unseen SQL Templates

Related papers

Loading 2010.11246v1…