PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning a Neural Semantic Parser from User Feedback

Srinivasan Iyer, Ioannis Konstas, Alvin Cheung, Jayant Krishnamurthy, Luke Zettlemoyer

arXiv:1704.08760v1cs.CL

TL;DR

Natural-language database interfaces are difficult to build across new domains because existing methods may sacrifice SQL expressivity or require extensive engineering. The paper directly generates SQL with neural sequence models, deploys the parser for user feedback, and sends selected errors for crowd annotation. Experiments report successful three-day learning from scratch in an academic domain, with performance improving through use, while feedback quality remains a limitation.

  • Problem

    Existing natural-language database interfaces either use meaning representations lacking full database-query expressivity or require extensive feature engineering for new domains.

  • Method

    An attention-based neural sequence-to-sequence model maps utterances directly to SQL, while online feedback selects predictions for crowd annotation and model improvement.

  • Results

    The approach successfully learns an academic-domain semantic parser from scratch after three days online and is comparable to previous systems on GEO880 and ATIS.

  • Takeaways & Limitations

    The approach can be rapidly deployed in new domains, uses SQL’s full expressivity, and can be ported to query languages such as SPARQL or ElasticSearch.

  • Takeaways & Limitations

    Feedback quality can be insufficient because incorrect results may reflect entity-identification or database incompleteness, and users may struggle to judge correctness.

Abstract

from arXiv · show

We present an approach to rapidly and easily build natural language interfaces to databases for new domains, whose performance improves over time based on user feedback, and requires minimal intervention. To achieve this, we adapt neural sequence models to map utterances directly to SQL with its full expressivity, bypassing any intermediate meaning representations. These models are immediately deployed online to solicit feedback from real users to flag incorrect queries. Finally, the popularity of SQL facilitates gathering annotations for incorrect predictions using the crowd, which is directly used to improve our models. This complete feedback loop, without intermediate representations or database specific engineering, opens up new ways of building high quality semantic parsers. Experiments suggest that this approach can be deployed quickly for any new target domain, as we show by learning a semantic parser for an online academic database from scratch.

1 Introduction

The paper proposes rapidly deployable semantic parsers that map utterances directly to SQL and improve through an online feedback-and-annotation loop. Experiments cover benchmark domains and a three-day academic-domain deployment.

  • Existing NLIDB approaches limit query expressivity or require extensive feature engineering, hindering deployment in new domains.
  • The approach directly maps utterances to SQL, deploys predictions online for feedback, and uses crowd annotations to improve the model.SQL annotations are used directly for model improvement and reduce annotation effort.
  • Non-linguists can write SQL for complex, compositional computations, supporting the use of skilled crowd workers for annotation.
  • 55% of the data suffices for high-quality parsers in simulated interactive learning with perfect user feedback.
  • A three-day online deployment learned an academic-domain parser from scratch, and actual users supplied feedback that supported continued improvement.

2 Related Work

Prior database semantic parsers commonly use specialized meaning representations or constrained SQL-generation systems. This work instead exploits SQL’s standardization and direct expressivity while reducing annotation effort through crowd-sourced labeling.

  • Logic programs, lambda calculus, and λ-DCS are uncommon outside semantic parsing and lack broad database implementation support.
  • Direct SQL parsing provides a popular, widely supported query language and enables programmers on skilled-labor crowd markets to label examples.
  • Earlier SQL-generation systems rely on manually engineered rules, lexical matches, or fixed syntax, limiting the queries they can produce.
  • The proposed approach uses neural sequence-to-sequence models to generate SQL directly from natural-language questions.
  • The method combines fully labeled SQL queries for accuracy with binary user feedback to reduce annotation effort.

3 Feedback-based Learning

Feedback-based learning rapidly deploys a preliminary semantic parser, gathers user feedback, and selectively obtains SQL annotations to improve later training stages. The cycle can be repeated as parser accuracy improves and annotation needs decline.

  • The algorithm deploys a preliminary parser for a new domain, then iteratively improves it using user feedback and selective query annotation.
  • Direct SQL generation makes selected user utterances cheaply and efficiently annotatable by crowd workers.
  • Training alternates with prediction and feedback-gathering stages, beginning with synthetic data generated from domain-independent schema templates.
  • The procedure can be repeated indefinitely, ideally increasing parser accuracy while requesting fewer annotations at each stage.

1 Procedure LEARN(schema)

The supplied procedure passage contains only the algorithm’s closing line and does not provide enough detail to summarize its procedure.

  • The excerpt marks the end of Algorithm 1 without stating its inputs, operations, or outputs.

4 Semantic Parsing to SQL

The parser maps anonymized natural-language questions directly to SQL with a neural encoder-decoder and global attention. Schema templates, paraphrases, entity handling, and interactive feedback support deployment across database domains.

  • Model: A neural sequence-to-sequence model maps natural-language questions directly to SQL queries, avoiding intermediate meaning representations.The encoder uses a bidirectional LSTM, while the decoder predicts SQL tokens with global attention.
  • Model: Global attention computes decoder context from encoder representations of source words, conditioning the next SQL-token prediction.Attention weights use the decoder hidden state and each source-token representation; the context vector is their weighted sum.
  • Model: Dropout regularizes non-recurrent connections, and beam search decodes SQL queries after learning.
  • Entity Anonymization: Entities are replaced by typed, incrementally numbered placeholders during training and restored in generated SQL using database search at test time.TF-IDF matching retrieves database entities for utterance spans, allowing variants such as “Donald Knuth” to match “Donald E. Knuth.”
  • Data Augmentation: Twenty-two schema-agnostic language/SQL templates bootstrap initial training data by filling slots with database entities, columns, values, and join conditions.The templates are designed to apply across databases and provide simple initial questions before more labeled examples become available.
  • Data Augmentation: PPDB supplies automatically generated paraphrases to improve generalization from training utterances.The method uses one-one and one-many paraphrases from PPDB’s large version, which contains over 220 million paraphrase pairs.

5 Benchmark Experiments

The benchmark experiments evaluate direct SQL generation on GEO880 and ATIS, using standard dataset splits and denotation-based test accuracy. The model reaches comparable performance to prior work without database-specific feature engineering, while paraphrase augmentation helps in ablations.

  • Datasets: The evaluation uses GEO880 and ATIS, two benchmark semantic-parsing datasets converted or represented with SQL queries.GEO880 uses 600/280 training/test examples; ATIS uses 4,473 training, 497 development, and 448 test utterances.
  • Datasets: GEO880 contains shorter but more compositional queries, whereas ATIS has longer utterances and queries spanning approximately 6 tables on average.Almost 40% of GEO880 SQL queries contain at least one nested subquery; ATIS averages 11-word utterances and 67-token SQL queries.
  • Experimental setup: The experiments augment training data with schema templates and three paraphrases per example, anonymize utterances, and tune models on development data or cross-validation.Development data supports hyperparameter tuning and early stopping; GEO880 uses cross-validation on its training set.
  • Results: The model is reported to achieve comparable performance to prior work without database-specific feature engineering, a first result of this kind for direct SQL parsing.The comparison is based on test accuracies for GEO880 and ATIS.
  • Results: The SQL model achieves reasonably high accuracies approaching the best non-SQL results on both datasets despite directly producing SQL.The reported evaluation uses denotations, while some comparison systems measure logical-form accuracy or use different data subsets.
  • Ablations: Paraphrasing consistently improves development performance, whereas schema-template augmentation does not significantly help in the fully supervised setting.The ablation contrasts the effects of PPDB paraphrases and schema templates under full supervision.

6 Interactive Learning Experiments

The interactive-learning experiment trains an academic-domain parser through repeated user feedback and crowd annotation, while simulated studies examine accuracy, augmentation, and annotation effort. The system improved across stages, but feedback and SQL execution errors remained important constraints.

  • Three-Stage Online Experiment: After three train-deploy cycles, the academic-domain system correctly answered 63.51% of users’ questions.The experiment recruited new users in each stage, used their feedback to identify incorrect utterances, and retrained with crowd annotations.
  • User Interface: The interface helped users assess predictions through entity type highlighting and five feedback options beyond a binary correct/incorrect judgment.Options included Correct, Wrong Types, Incomplete Result, Wrong Result, and Can’t Tell.
  • Three-Stage Online Experiment: 25% of questions were answered correctly in stage one, and accuracy increased while annotation effort decreased in each successive stage.The first-stage model used only synthetic schema-template data and paraphrases; later stages incorporated additional user utterances and labels.
  • Three-Stage Online Experiment: 22.3% of generated queries failed to execute, while users mislabeled 6.1% of correct queries and 6.3% of incorrect queries.Erroneous feedback could create redundant annotations or incorrectly labeled training examples that may deteriorate accuracy over time.
  • SCHOLAR Dataset: The released SCHOLAR dataset contains 816 SQL-labeled utterances split 600/216, with fully supervised accuracy of 67% versus 52.75% for TF-IDF nearest neighbors.The dataset includes an executable academic-paper database and covers authors, citations, journals, keywords, and datasets.
  • SCHOLAR Dataset: 15% of predicted queries did not execute, mainly because of missing joins or incorrect types that could not be deanonymized from the utterance.Other well-formed errors included ignoring parts of utterances and failing to transfer some utterance types into SQL.
  • Simulated Interactive Learning: In simulated GEO880 and ATIS learning, accuracy improved with successive batches, while paraphrasing helped at all stages and template benefits varied by domain.Templates helped early GEO880 performance but not ATIS, possibly because the templates generated only single-entity questions whereas ATIS often used two entities.
  • Simulated Interactive Learning: A GEO880 batch size of 50 required annotation for only 54.3% of examples, illustrating that smaller batches can reduce annotation effort.The authors connect more frequent deployment of improved models with fewer mistakes requiring annotation.

7 Conclusion

The paper presents a neural, feedback-driven approach that directly maps utterances to SQL and iteratively improves through selective crowd annotation. It performs comparably to prior systems on GEO880 and ATIS and learns an academic-domain parser from scratch, with portability to other query languages.

  • Conclusion: The approach combines an attention-based neural sequence-to-sequence model, database and paraphrase augmentation, online feedback, and selective crowd annotation.The model generates SQL directly, and user feedback selects utterances for annotation and subsequent improvement.
  • Conclusion: The parser is comparable to previous logical-form and SQL-generation systems on GEO880 and ATIS.The paper also demonstrates online learning of an academic-domain parser from scratch.
  • Conclusion: The approach is not language-specific and can be ported to query languages such as SPARQL or ElasticSearch.This portability follows the paper’s direct query-language generation framing.
Loading 1704.08760v1…