PaperlayerView exact revision on arXiv

Source-linked AI summary

How Far are We from Effective Context Modeling? An Exploratory Study on Semantic Parsing in Context

Qian Liu, Bei Chen, Jiaqi Guo, Jian-Guang Lou, Bin Zhou, Dongmei Zhang

arXiv:2002.00652v2cs.CLcs.AI

TL;DR

Semantic parsing in context remains difficult to assess because prior evaluations covered limited scenarios, leaving questions about typical context methods in complex settings and with grammar-based decoding. This paper adapts those methods to a grammar-based parser and evaluates them on two cross-domain datasets; its best model reaches state of the art on both, while simple concatenation remains competitive.

  • Problem

    It remains unclear how typical context-modeling methods perform on complex cross-domain semantic parsing in context, with grammar-based decoding and across contextual phenomena.

    Source: §1 · p. 1

  • Method

    The paper adapts typical context methods to a grammar-based semantic parser and evaluates 13 methods on SPARC and COSQL.

    Source: §1 · p. 2, §3.2 · p. 5

  • Results

    The best model achieves state-of-the-art performance on both datasets, while no evaluated context method is consistently superior and simple concatenation is competitive.

    Source: §1 · p. 2, §3.2 · p. 5, §5 · p. 6

  • Takeaways & Limitations

    The analysis suggests future work should incorporate common sense for pronoun inference and model contextual clues more explicitly.

    Source: §5 · p. 6

Abstract

from arXiv · show

Recently semantic parsing in context has received considerable attention, which is challenging since there are complex contextual phenomena. Previous works verified their proposed methods in limited scenarios, which motivates us to conduct an exploratory study on context modeling methods under real-world semantic parsing in context. We present a grammar-based decoding semantic parser and adapt typical context modeling methods on top of it. We evaluate 13 context modeling methods on two large complex cross-domain datasets, and our best model achieves state-of-the-art performances on both datasets with significant improvements. Furthermore, we summarize the most frequent contextual phenomena, with a fine-grained analysis on representative models, which may shed light on potential research directions. Our code is available at https://github.com/microsoft/ContextualSP.

1 Introduction

Semantic parsing in context must resolve incomplete dialogue questions involving coreference and ellipsis. The paper studies typical context-modeling methods with grammar-based decoding and analyzes them on two complex cross-domain datasets.

1 Introduction

  • Context-dependent questions include coreference, such as pronouns, and ellipsis, where a question omits meaning needed for its SQL query.The example dialogue includes “How about with the max MPG?” and “Show its make!”Source: §1 · p. 1
  • Prior context methods use recent questions, precedent logic forms, or both as context [Iyyer et al., 2017; Suhr et al., 2018; Yu et al., 2019b; Zhang et al., 2019; Yu et al., 2019a].For example, recent questions may be concatenated or encoded hierarchically, while precedent logic forms may be attended to or copied.Source: §1 · p. 1
  • It remains unclear how context methods perform on complex cross-domain tasks, with grammar-based decoding, and across varied contextual phenomena.Prior work had not thoroughly compared typical methods in complex settings or verified them with grammar-based decoding.Source: §1 · pp. 1–2
  • The study adapts typical context-modeling methods to a grammar-based semantic parser and compares them on SPARC and COSQL.It also analyzes frequent contextual phenomena and representative models.Source: §1 · p. 2

2 Methodology

The methodology builds a grammar-based parser for dialogue questions and adapts context modeling methods to use recent questions, preceding SQL, and BERT-enhanced embeddings. It evaluates these methods for question-to-SQL parsing with database schemas, including schemas unseen during training.

2.1 Base Model

  • The base model uses an attention-based sequence-to-sequence architecture with a question encoder and grammar-based SQL decoder.Source: §2.1 · p. 2

Question Encoder

Grammar-based Decoder

  • The decoder emits grammar-rule actions in depth-first order that correspond to the SQL abstract syntax tree, using question-token linking for schema-specific actions in unseen schemas.Schema-specific grammar rules change with the database schema.Source: Grammar-based Decoder · pp. 2–3, Figure 2 · p. 2

2.2 Recent Questions as Context

  • Recent-question context is incorporated by concatenating questions, encoding turns hierarchically, or gating question-level importance in attention.CONCAT extends the encoder input; TURN uses a turn-level encoder; GATE weights attention by question importance.Source: Figure 3 · p. 3, §2.2 · p. 3

2.3 Precedent SQL as Context

  • Preceding SQL informs decoding through attention, compatible-action copying, or subtree copying, which reduces the number of decoding steps.Source: §2.3 · pp. 3–4, Figure 4 · p. 4

2.4 BERT Enhanced Embedding

  • BERT supplies schema-aware question and question-aware schema representations in place of the question embeddings, while the rest of the model remains unchanged.Source: §2.4 · p. 4

3 Experiment & Analysis

The experiments compare context-modeling methods on complex cross-domain semantic parsing, then examine their behavior on out-of-distribution turns and fine-grained contextual phenomena.

3.1 Experimental Setup

  • Evaluation uses SPARC (3,034/422 train/development dialogues) and COSQL (2,164/292), measuring exact-set SQL match with Question, Interaction, and Turn i Match.SPARC and COSQL average 3.0 and 5.2 turns per dialogue, respectively.Source: §3.1 · p. 4

3.2 Model Comparison

  • Our parser achieves state-of-the-art performance on SPARC and COSQL; on SPARC without BERT, it improves over the previous SOTA by 8.8 Ques.Match and 4.2 Int.Match points.Source: §3.2 · p. 5
  • Across 13 methods, none is consistently superior: CONCAT and TURN are competitive, ACTION COPY surpasses TREE COPY and SQL ATTN, and combinations add little.The figure reports Question Match, Interaction Match, and Turn i Match; its first column gives absolute values and the others show improvements over CONCAT.Source: §3.2 · p. 5, Figure 5 · p. 5
  • In the out-of-distribution test, ACTION COPY is comparable to or better than the alternatives, CONCAT is competitive, and TURN is more vulnerable on later turns.The three methods are trained on turns 1–2 and tested on turn 3 and beyond.Source: §3.2 · p. 5, Figure 6 · p. 5

3.3 Fine-grained Analysis

  • The analysis groups questions as semantically complete, coreference, or ellipsis; Table 2 gives fine-grained types, counts, and representative SPARC development examples.Ellipsis is divided into continuation and substitution, with substitution further categorized by explicitness and whether it concerns schema or operator.Source: §3.3 · p. 5, Table 2 · p. 6
  • Models struggle with definite noun phrases and one-anaphora, especially when the antecedent is partial; such antecedents are nearly 40% of these cases, versus 22% for demonstratives.The analysis concludes that current methods do not succeed on pronouns requiring complex contextual inference.Source: §3.3 · pp. 5–6
  • All models perform better on continuation than substitution; explicit substitution unexpectedly trails implicit substitution, while operator substitution matches or exceeds schema substitution.The authors attribute the explicit–implicit gap to contextual clues not being well utilized, and suggest cross-domain schema substitution is harder.Source: §3.3 · p. 6

4 Related Work

Related work includes methods for semantic parsing in context and broader studies of contextual representations, conversational response generation, and contextual phenomena. This work differs by exploring context-modeling methods and analyzing contextual phenomena at a fine-grained level.

4 Related Work

  • SPC methods include context-dependent substitution, search-based sequential parsing, sequence-to-sequence parsing, edit-based SQL reuse, and an auxiliary task.The work is presented as an exploratory study distinct from these prior approaches.Source: §4 · p. 6
  • Other context studies address contextual representations in context-independent parsing, conversational response generation, and summaries or experiments on contextual phenomena.The paper distinguishes its focus on context modeling for semantic parsing and its fine-grained phenomenon analysis.Source: §4 · p. 6

5 Conclusion & Future Work

The study finds that existing context-modeling methods are less effective than expected, while simple concatenation can be competitive. It identifies commonsense pronoun inference and more explicit modeling of contextual clues as future directions.

5 Conclusion & Future Work

  • Existing context-modeling methods are not as effective as expected, and simple concatenation can be much competitive.The conclusion presents this as a finding of the exploratory comparison.Source: §5 · p. 6
  • The proposed future directions are incorporating common sense for better pronoun inference and modeling contextual clues more explicitly.Source: §5 · p. 6
  • The authors say fine-grained analysis can help the community debug models and make further progress.Source: §5 · p. 6
Loading 2002.00652v2…