PaperlayerView exact revision on arXiv

Source-linked AI summary

Model-based Interactive Semantic Parsing: A Unified Framework and A Text-to-SQL Case Study

Ziyu Yao, Yu Su, Huan Sun, Wen-tau Yih

arXiv:1910.05389v1cs.CLcs.AI

TL;DR

Interactive semantic parsing must handle ambiguity, imperfect parser accuracy, and users’ difficulty validating neural outputs. The paper introduces MISP, a model-based agent that maintains parse state, detects errors, asks clarification questions, and updates its state from feedback. Across WikiSQL and Spider, its text-to-SQL instantiation improves accuracy with targeted feedback and asks half as many questions as DialSQL, while future work remains for learning in non-stationary feedback environments.

  • Problem

    Interactive semantic parsing needs a general solution for ambiguous utterances, limited parser accuracy, and difficult user validation.

    Source: §1 · p. 1

  • Method

    MISP models interactive parsing as an agent with a parse state, world model, error detector, and natural-language actuator, instantiated for text-to-SQL.

    Source: §1 · p. 2, §4 · p. 3, §4.1 · p. 4

  • Results

    10% to 15% absolute accuracy gains are achieved with targeted feedback, while the approach asks half as many questions as DialSQL and performs better.

    Source: §1 · p. 2

  • Takeaways & Limitations

    MISP offers a simpler, more general framework that can integrate different base parsers and support interactive text-to-SQL on WikiSQL and Spider.

    Source: §1 · p. 2, §4 · p. 3

    Learning from user feedback can create a non-stationary environment that makes agent training, such as error-detector learning, unstable.

    Source: §6 · p. 9

Abstract

from arXiv · show

As a promising paradigm, interactive semantic parsing has shown to improve both semantic parsing accuracy and user confidence in the results. In this paper, we propose a new, unified formulation of the interactive semantic parsing problem, where the goal is to design a model-based intelligent agent. The agent maintains its own state as the current predicted semantic parse, decides whether and where human intervention is needed, and generates a clarification question in natural language. A key part of the agent is a world model: it takes a percept (either an initial question or subsequent feedback from the user) and transitions to a new state. We then propose a simple yet remarkably effective instantiation of our framework, demonstrated on two text-to-SQL datasets (WikiSQL and Spider) with different state-of-the-art base semantic parsers. Compared to an existing interactive semantic parsing approach that treats the base parser as a black box, our approach solicits less user feedback but yields higher run-time accuracy.

1 Introduction

Interactive semantic parsing addresses ambiguity, imperfect parser accuracy, and users’ difficulty validating neural semantic parses. MISP provides a general model-based framework and a text-to-SQL instantiation that improves accuracy with less feedback than DialSQL.

Motivation

  • Natural-language interfaces remain challenging because utterances may be ambiguous, parsers are not sufficiently accurate, and neural outputs are difficult for users to validate.These challenges motivate involving users in the parsing loop.Source: §1 · p. 1
  • Existing interactive semantic parsers improve collaboration but are often tied to particular formal languages or datasets, limiting generalization beyond settings such as WikiSQL.DialSQL, for example, is difficult to extend beyond its dataset-specific SQL templates.Source: §1 · pp. 1–2

MISP framework

  • MISP models interactive parsing as an agent that maintains a parse state, detects likely errors, generates questions, and updates the state from user feedback.The framework repeats these interactions until a terminal state is reached.Source: §1 · p. 2
  • MISP separates the world model, error detector, and actuator, allowing task-specific components to be improved independently while preserving the general framework.The formulation also facilitates applying techniques such as reinforcement learning.Source: §1 · p. 2

Empirical results

  • 10% to 15% absolute accuracy gains result from small amounts of targeted, test-time feedback across WikiSQL and Spider.Compared with DialSQL, the approach achieves better parsing accuracy while asking only half as many questions.Source: §1 · p. 2

2 Background & Related Work

Semantic parsing maps language to formal representations, while prior interactive systems improve accuracy but remain specialized to particular languages or datasets. MISP instead provides a general design that integrates existing base parsers through modular components.

Semantic Parsing

  • Spider targets complex multi-keyword, multi-table SQL, and SyntaxSQLNet achieves 20% accuracy in this challenging setting.WikiSQL uses simpler grammar sketches, whereas Spider requires recursive syntax-aware generation.Source: §2 · p. 2
  • For SQLNet, MISP represents the current state as a partially generated SQL query and transitions it as the parser fills the next slot.This illustrates how the framework naturally incorporates a base parser’s generation process.Source: §2 · pp. 2–3

Interactive Semantic Parsing

  • MISP integrates existing semantic parsers as base parsers while decoupling the error detector, question generator, and world model.This design substantially reduces the engineering cost of building interactive parsers.Source: §2 · p. 3

3 Model-based Interactive Semantic Parsing

MISP represents interactive parsing as a model-based agent loop: users provide utterances and feedback, the agent maintains a parse state, detects errors, and presents clarification questions. Its world model predicts state transitions from user feedback.

Environment

  • The environment supplies an initial utterance and subsequent feedback reflecting the user’s intended semantic parse.These percepts drive the agent’s interaction session.Source: §3 · p. 3

Agent State

  • The agent state is the current semantic parse, optionally augmented with prediction probabilities or uncertainty for error detection.This state summarizes the agent’s internal interpretation of the environment.Source: §3 · p. 3

World Model

  • The world model compresses interaction history and models p(s_t+1|s_t, u_t), allowing deterministic or stochastic transitions after user feedback.Here u_t is feedback at step t and s_t+1 is the new agent state.Source: §3 · p. 3

Error Detector

  • The error detector terminates at a terminal state or identifies a likely erroneous parse span and passes it with context to the actuator.At termination, the semantic parse is executed and results are returned to the user.Source: §3 · p. 3

Actuator

  • The actuator realizes actions through a natural-language generator, graphical interface, or both.Its role is to present the agent’s questions in a user-friendly form.Source: §3 · p. 3

4 MISP-SQL: An Instantiation of MISP for Text-to-SQL

MISP-SQL instantiates model-based interactive semantic parsing for text-to-SQL with a partial-parse state, uncertainty-driven intervention, natural-language clarification, and feedback-based state updates.

4 MISP-SQL: An Instantiation of MISP for Text-to-SQL

  • MISP-SQL wraps a base text-to-SQL parser with a world model, error detector, and template-based actuator for interactive parsing.The world model changes the parser’s prediction behavior, while the error detector uses prediction uncertainty to trigger clarification.Source: §4 · p. 3

4.1 Agent State

  • Each agent state is a partial SQL query whose predicted components carry uncertainty for introspection and error detection.The state is represented as s_t={o_1, o_2, ..., o_t}, where each o_t is a predicted SQL component.Source: §4.1 · p. 4

4.2 Error Detector

  • The greedy error detector asks for clarification when the latest SQL-component prediction is sufficiently uncertain, using probability or dropout-based uncertainty measures.Probability triggers clarification below p*; dropout triggers it when the standard deviation across perturbations exceeds s*.Source: §4.2 · p. 4

4.3 Actuator: An NL Generator

  • The actuator asks binary questions about predicted SQL components using a rule-based, domain-agnostic lexicon and grammar that can be extended for more complex SQL.Questions are derived from component categories such as aggregators, operators, columns, and generated questions, with clause context providing meaning.Source: §4.3 · pp. 4–5, Table 1 · p. 5

4.4 World Model

  • User feedback either advances decoding after confirmation or constrains the parser to replace a rejected component with an alternative, bounded by K alternatives per component.Unlike DialSQL, MISP-SQL directly uses the base parser to transition states, avoiding additional training.Source: §4.4 · p. 5

5 Experiments

MISP-SQL evaluates model-based interactive semantic parsing across WikiSQL and Spider with simulated and human users. It improves base-parser accuracy through targeted feedback, while detector choice and user understanding constrain performance.

5.2 Simulation Evaluation

  • 94% execution accuracy for SQLova, up from 85%, required only 0.773 questions per query in WikiSQL simulation.Source: §5.2 · p. 6
  • MISP-SQL improves SQLNet and SQLova with targeted interaction, outperforming DialSQL on SQLNet while asking 1.104 versus 2.4 questions per query.The comparison also reports a simpler design without an extra trained model.Source: §5.2 · p. 6
  • Probability-based detection matches dropout-based accuracy with fewer questions for both parsers, with query-match differences of 0.15–0.25 for SQLNet and 0.1–0.15 for SQLova.Source: §5.2 · p. 7, Figure 4 · p. 7
  • Probability detection spends 16.9% of SQLNet questions on correct predictions versus 32.1% for dropout detection at comparable interaction volume.Lowering the probability threshold from 0.8 to 0.5 reduces this share from 23.0% to 16.9%, whereas dropout-based detection remains almost unchanged.Source: §5.2 · p. 7, Table 3 · p. 7

5.3 Extend to Complex SQL Generation

  • On Spider, MISP-SQL improves SyntaxSQLNet by 10% accuracy with roughly 3 questions per query, but probability detection requires more questions than dropout detection.The weaker 20% base accuracy leaves many wrong predictions unsolvable through subsequent component corrections.Source: §5.3 · pp. 7–8, Table 4 · p. 8

5.4 Human Evaluation

  • Human interaction improves every tested base parser, but gains are smaller than in simulation because evaluators may misunderstand ambiguous questions, vague wording, or unfamiliar entities.With full table and gold-SQL information, SQLova performance approaches simulation results; 80% of evaluator–simulator disagreements involved ambiguity or incorrect gold annotations.Source: §5.4 · p. 8, §5.5 · pp. 8–9, Table 5 · p. 8

5.5 Discussion on Future Human Evaluation

  • Using pre-existing test questions creates a gap between evaluator intent and dataset intent, while showing table contents can resolve entities but overexpose evaluators to the system.The paper proposes evaluator-generated questions, partial table views, and larger-scale studies as future alternatives.Source: §5.5 · pp. 8–9

6 Conclusion and Future Work

The paper concludes that MISP provides a unified framework whose components can be improved independently, while identifying lifelong learning and scaling as future directions.

Improving Agent Components

  • MISP’s flexible framework allows its agent components to be improved separately.The error detector could use probability calibration or a reinforced decision policy.Source: §6 · p. 9

Lifelong Learning for Semantic Parsing

  • Learning from user feedback could support lifelong semantic parser improvement but may create a non-stationary environment that destabilizes training.The paper leaves this issue for future work.Source: §6 · p. 9

Scaling Up

  • Scaling MISP to larger data sources requires richer interaction, stronger world models, and policies that improve accuracy while minimizing interventions.The paper names knowledge bases such as Freebase or Wikidata as target settings.Source: §6 · p. 9

A Extension to Complex SQL

The extension adapts MISP-SQL’s lexicon and grammar to generate complex Spider queries involving multiple tables.

A Extension to Complex SQL

  • Spider queries can associate with multiple tables, so columns are named with both column and table information.The notation is COL[col (table tab)], with the table phrase omitted for simplicity when referring to a column.Source: A Extension to Complex SQL · p. 13
  • Table 8 lists the extended lexicon and grammar rules used by the NLG module for complex SQL generation.Source: A Extension to Complex SQL · p. 13
  • The extension uses combined column-table names to support grammar construction for multi-table SQL queries.Source: A Extension to Complex SQL · p. 13

B Simulation Evaluation Results

The supplementary results report complete simulation experiments for MISP-SQL instantiated with SQLNet and SQLova.

B Simulation Evaluation Results

C Error Detector Comparison

The error-detector comparison evaluates different detectors under matched interaction budgets across SQLNet and SQLova settings on WikiSQL.

C Error Detector Comparison

  • Error detectors are compared by tuning thresholds so their actual average question budgets stay within ±0.015 of target budgets.Results are reported for SQLNet and SQLova across WikiSQL development and test settings.Source: C Error Detector Comparison · p. 13
  • Tables 9 and 10 compare error detectors for SQLNet on WikiSQL development and test sets, respectively.Source: Table 9 · p. 14, Table 10 · p. 14
  • Tables 11 and 12 compare error detectors for SQLova on WikiSQL development and test sets, respectively.Source: Table 11 · p. 14, Table 12 · p. 14

Related papers

Loading 1910.05389v1…