Source-linked AI summary
Semantic Parsing for Task Oriented Dialog using Hierarchical Representations
Sonal Gupta, Rushin Shah, Mrinal Mohit, Anuj Kumar, Mike Lewis
TL;DR
Task-oriented intent-slot systems have limited ability to represent compositional queries, while logical forms are difficult to annotate and parse. The paper introduces TOP, a hierarchical intent-slot representation with a 44k-query dataset, and adapts constituency parsing models to it. The resulting parsing approach outperforms sequence-to-sequence baselines, while the representation covers nearly all observed queries but excludes a small graph-structured minority.
Problem
Traditional intent-slot systems cannot adequately represent many nested compositional queries, while logical forms are challenging to annotate and parse.
Method
The paper introduces TOP, a hierarchical tree representation with intents nested inside slots, and applies constituency parsing models including RNNG.
Results
RNNG achieves perfect outputs on over 75% of queries and outperforms sequence-to-sequence models, especially on exact-tree accuracy.
Takeaways & Limitations
TOP provides 30% higher coverage than traditional intent-slot tagging while supporting annotation and parsing with standard tree-based methods.
Takeaways & Limitations
Only 0.3% of the dataset was estimated to require a more expressive representation than a tree, such as a graph.
Abstract
from arXiv · showhide
Task oriented dialog systems typically first parse user utterances to semantic frames comprised of intents and slots. Previous work on task oriented intent and slot-filling work has been restricted to one intent per query and one slot label per token, and thus cannot model complex compositional requests. Alternative semantic parsing systems have represented queries as logical forms, but these are challenging to annotate and parse. We propose a hierarchical annotation scheme for semantic parsing that allows the representation of compositional queries, and can be efficiently and accurately parsed by standard constituency parsing models. We release a dataset of 44k annotated queries (fb.me/semanticparsingdialog), and show that parsing models outperform sequence-to-sequence approaches on this dataset.
1 Introduction
The paper introduces TOP, a hierarchical representation for compositional task-oriented queries, and shows that it supports nested semantics while enabling effective parsing. It also releases a dataset and reports that RNNG parsing outperforms sequence-to-sequence baselines.
- Motivation and representation: TOP represents compositional and nested task-oriented queries with a hierarchical intent-slot structure.Traditional intent-slot systems struggle with queries combining intents, whereas TOP is designed to represent such nesting.
- Motivation and representation: The representation is easier to annotate and parse than logical forms or dependency graphs while retaining expressive power.Its structure resembles constituency parses, allowing standard parsing methods to be adapted.
- Contributions: RNNG parsing outperforms CNN-, LSTM-, and Transformer-based sequence-to-sequence models, benefiting from an inductive bias toward well-formed trees.The authors connect the model’s constituency-parsing structure to improved accuracy, especially for exact tree prediction.
- Contributions: The authors release a dataset of 44k annotated requests and report high coverage, high agreement, and learnability with standard algorithms.The dataset is part of the paper’s contribution alongside the representation and parsing results.
2 Representation
TOP generalizes intent-slot tagging into a tree-structured representation that captures nested queries and supports execution through explicit compositional structure. The scheme covers most observed requests while leaving a small minority beyond tree-structured scope.
- Representation: TOP uses a constituency-like tree whose terminals are words, whose nonterminals are intents or slots, and whose root is an intent.Intents may be nested inside slots, allowing one query to compose requests across domains.
- Execution: The tree structure supports compositional execution by extracting an inner result and passing it into an explicitly tagged outer slot.For navigation to an event, the event location can be passed as the destination to the navigation intent.
- Expressiveness: 30% of crowd-sourced queries could not be adequately represented by traditional intent-slot tagging, motivating a more expressive representation.A separate manual analysis found that 30% of 100 sampled queries required compositional representation.
- Scope and trade-offs: Only 0.3% of the dataset was estimated to require a more expressive representation than a tree.More expressive alternatives could provide marginal gains but would add annotation and learning challenges.
- Scope and trade-offs: The authors characterize TOP as a compromise combining higher coverage than traditional tagging with fewer annotation and learning complexities than more general representations.The representation also permits reuse of constituency-parsing models and existing dialog-system infrastructure.
3 Dataset
The dataset contains crowdsourced navigation and event utterances labeled through multi-annotator review and adjudication. It includes 44,783 annotations spanning 25 intents and 36 slots, with substantial structural variation.
- Collection and annotation: Crowdsourced workers generated natural-language requests for navigation and event assistance, which were labeled by two annotators and adjudicated when necessary.Disagreements among three annotators led to discarding the utterance and its annotations.
- Collection and annotation: 63.40% of utterances were resolved with two annotations, while 94.09% were resolved after three annotations.The remaining disagreements were discarded according to the annotation procedure.
- Dataset composition: 44,783 annotations cover 25 intents and 36 slots, split into 31,279 training, 4,462 validation, and 9,042 test utterances.The utterances focus on navigation, events, and navigation-to-events requests.
- Dataset composition: 35% of trees have depth greater than 2, and 4,646 utterances contain both navigation and event intents.The median and mean tree depths are 2 and 2.54, respectively.
4 Models
The paper compares sequence-to-sequence models with RNNG, a syntactic parsing model adapted to construct hierarchical task-oriented parses. RNNG builds well-formed trees through constrained transitions that shift tokens, reduce subtrees, and generate intent or slot labels.
- Model comparison: The experiments compare standard sequence-to-sequence learning with Recurrent Neural Network Grammars adapted from syntactic parsing.RNNG is trained discriminatively rather than generatively to reduce training time.
- RNNG transition system: RNNG constructs each parse tree through SHIFT, REDUCE, and non-terminal-generation actions.SHIFT consumes an input token, REDUCE closes a subtree, and non-terminal actions generate slot or intent labels.
- RNNG transition system: At each step, RNNG restricts prediction to valid actions, such as allowing only REDUCE after all input tokens have been added.These constraints ensure that generated structures remain well formed.
5 Experiments and Results
The experiments evaluate RNNG and CNN, LSTM, and Transformer sequence-to-sequence baselines using structural parsing metrics. RNNG produces perfect outputs for over 75% of queries and outperforms sequence-to-sequence models, especially on exact tree prediction.
- Metrics: The systems are evaluated with exact match accuracy, labeled bracketing F1, Tree-Labeled F1, and tree validity.Exact match is strictest, ordinary F1 is least strict, and Tree-Labeled F1 compares subtree structure rather than only token spans.
- Results: RNNG achieves perfect outputs on over 75% of queries and performs better than sequence-to-sequence models.Its advantage is especially pronounced for predicting exact trees, which the paper identifies as important for task completion.
- Results: 75.63% exact match accuracy is reported for the seq2seq-CNN model with a Limited Output Token Vocabulary, compared with 68.39% for seq2seq-LSTM.The reduced vocabulary was intended to focus learning on tree structure rather than reproducing input tokens.
- Inference: RNNG Top-3 and Top-5 scores reach 90.21 and 92.48, respectively, compared with 78.51 for Top-1.These scores measure whether the correct tree appears among the top k predictions.
- Ablations: Removing the RNNG buffer LSTM reduces exact match to 13.78, while removing the actions or stack LSTM reduces it to 78.08 or 75.31.The ablation results indicate that the buffer component is particularly important for performance.
- Efficiency and inductive bias: RNNG inference has linear time complexity, whereas sequence-to-sequence inference is quadratic because attention is recomputed at every time step.The paper also attributes RNNG’s performance to well-formed-tree guarantees and shorter dependencies between intents and slots.
6 Related Work
The paper positions its tree-structured representation between traditional intent-slot tagging, logical forms, and arbitrary graph representations. It emphasizes a trade-off between expressive power and annotation or parsing difficulty.
- Intent-slot tagging: Traditional task-oriented systems usually identify one intent and fill its relevant slots, limiting representation of nested intents.The paper reports that hierarchical representations improve request coverage in its domains.
- Logical forms: Logical forms are more expressive and can be executed directly, but they are difficult to annotate and lack large-scale datasets.The paper presents its representation as more tightly coupled to the input query than logical forms.
- Design trade-offs: The paper frames annotation-scheme design as a trade-off between expressiveness and ease of annotation, parsing, and execution.Its hierarchical representation is intended to occupy a practical middle ground among these alternatives.
- Graph representations: Abstract Meaning Representation and Alexa Meaning Representation use arbitrary graphs that capture constructions beyond this approach but make parsing significantly more challenging.The paper states that such beyond-scope cases are very rare in its data.
7 Conclusions
The paper introduces a hierarchical generalization of intent-slot representations for nested queries, releases a 44k-request dataset, and reports higher coverage and parsing accuracy than sequence-to-sequence models.
- Conclusions: The hierarchical representation supports complex nested queries and yields 30% higher coverage of user requests.It generalizes traditional intents and slots using a tree-structured representation.
- Conclusions: The released dataset contains 44k annotated requests, and the representation has high annotation agreement and very high request coverage.The paper makes the annotated utterances publicly available.
- Conclusions: The representation enables existing constituency parsing algorithms and results in higher accuracy than sequence-to-sequence models.The paper specifically reports that RNNG outperforms sequence-to-sequence baselines.