Source-linked AI summary
Dialog State Tracking: A Neural Reading Comprehension Approach
Shuyang Gao, Abhishek Sethi, Sanchit Agarwal, Tagyoung Chung, Dilek Hakkani-Tur
TL;DR
Dialog state tracking must estimate belief states while handling slot values that may fall outside fixed ontologies. This paper formulates tracking as reading comprehension, using attention to point to values in dialog context alongside carryover and slot-type models. On MultiWOZ-2.0, the combined approach reaches 47.33% joint-goal accuracy, 11.75% above prior state-of-the-art.
Problem
Fixed-vocabulary dialog state trackers have difficulty handling unseen slot values, motivating a vocabulary-free way to identify values in the conversation.
Method
The method treats each dialog as a passage, asks a slot-specific question, and predicts carryover, slot type, and an answer span with attention-based models.
Results
47.33% joint-goal accuracy on MultiWOZ-2.0 is reported when the method is combined with traditional joint state tracking, exceeding prior state-of-the-art by 11.75%.
Takeaways & Limitations
The approach supports selecting slot values directly from dialog context while contextual embeddings and traditional fixed-vocabulary tracking improve joint-goal accuracy.
Takeaways & Limitations
Imprecise slot references remain an error source when multiple candidates occur, especially after user corrections or when entity order is misleading.
Abstract
from arXiv · showhide
Dialog state tracking is used to estimate the current belief state of a dialog given all the preceding conversation. Machine reading comprehension, on the other hand, focuses on building systems that read passages of text and answer questions that require some understanding of passages. We formulate dialog state tracking as a reading comprehension task to answer the question $what\ is\ the\ state\ of\ the\ current\ dialog?$ after reading conversational context. In contrast to traditional state tracking methods where the dialog state is often predicted as a distribution over a closed set of all the possible slot values within an ontology, our method uses a simple attention-based neural network to point to the slot values within the conversation. Experiments on MultiWOZ-2.0 cross-domain dialog dataset show that our simple system can obtain similar accuracies compared to the previous more complex methods. By exploiting recent advances in contextual word embeddings, adding a model that explicitly tracks whether a slot value should be carried over to the next turn, and combining our method with a traditional joint state tracking method that relies on closed set vocabulary, we can obtain a joint-goal accuracy of $47.33\%$ on the standard test split, exceeding current state-of-the-art by $11.75\%$**.
1 Introduction
Dialog state tracking estimates a conversation’s current belief state, but fixed-ontology methods struggle with unseen slot values. The paper recasts tracking as reading comprehension and combines span extraction with carryover and slot-type decisions, reporting improved MultiWOZ-2.0 accuracy.
- Fixed-ontology trackers often struggle with previously unseen slot mentions because their output space is restricted to predefined values.
- The proposed method treats the dialog as a passage and answers slot-specific questions by pointing to answer spans within the conversation.
- Dialog state tracking is decomposed into binary slot carryover, slot type, and slot span decisions.
- The study also incorporates BERT contextual embeddings and provides an in-depth error analysis of the proposed methods.
- 42.12% joint-goal accuracy represents a 6.5% absolute improvement over previous state-of-the-art methods on MultiWOZ-2.0.
- 47.33% joint-goal accuracy, an 11.75% advance over previous state-of-the-art, is obtained by combining the approach with traditional joint state tracking.
2 Related Work
Related work frames dialog state tracking and reading comprehension as neighboring approaches to selecting slot values. The paper distinguishes its method by pointing to spans in dialog context without relying solely on a fixed vocabulary.
- Traditional dialog state trackers constrain slot outputs with a fixed ontology, limiting handling of unseen values and scalability to large vocabularies.
- Earlier candidate-scoring methods derive possible slot values from predefined ontologies or words and n-grams extracted from prior dialog context.
- Prior reading-comprehension formulations include a relatively complex memory network and may still assume a fixed-set vocabulary.
- Reading comprehension encompasses answer formats including cloze-style, multiple-choice, span, and free-form answers.
- Contextual embeddings such as ELMO and BERT are trained with language-model objectives on large corpora and are applied here to dialog state tracking.
3 Our Approach
The approach reformulates dialog state tracking as reading comprehension over the conversational history, answering slot-specific questions. It combines carryover, slot-type, and span predictions to identify state values, including values expressed directly in the dialog.
- Each dialog prefix is treated as a passage, and each slot is queried with “what is the value for slot i?”
- Dialog Encoding: The model encodes concatenated user and agent utterances with turn markers, pretrained word vectors, and a recurrent network.A bidirectional LSTM produces contextual token representations; BERT contextual embeddings are also evaluated.
- Models: The system sequentially predicts whether to carry over a slot, its type, and, when needed, its start and end span.Slot types are Yes, No, DontCare, or Span; Span triggers extraction of the value from the dialog.
- Slot Carryover Model: The carryover model predicts whether each slot changed from the previous turn, jointly producing decisions across all slots.
- Slot Span Model: For span-valued slots, bilinear interactions between token encodings and the slot question predict the value’s start and end positions.Inference selects the span whose start and end probabilities jointly provide the highest-scoring valid span.
4 Experiments
The experiments evaluate the method on MultiWOZ-2.0, a seven-domain dataset with 37 slots, using standard dataset splits. The dataset contains annotation inconsistencies that affect evaluation and slot carryover, while the models use frozen BERT representations and separate training procedures.
- Data: MultiWOZ-2.0 contains multi-domain conversations across seven domains and 37 slots, including slot types shared across domains.Experiments use the standard training, development, and test splits.
- Data Issues: The dataset has inconsistent ontology and dialog-context values, such as “moderate” versus “moderately.”
- Data Issues: Erroneous delays in state updates can extend across turns and negatively affect the slot carryover model.
- Training: The three models are trained independently, using BERT followed by an affine layer and a one-layer bidirectional LSTM, with BERT parameters frozen.
- Training: Slot carryover predicts a joint binary vector over 37 slots, while slot-type and slot-span models treat each dialog–question pair as a separate task.
5 Results
On MultiWOZ-2.0, the reading-comprehension approach performs competitively, while error analysis identifies slot carryover and context depth as important constraints.
- Results: 42.12% joint goal accuracy is achieved by the ensemble model on the MultiWOZ-2.0 test set.The single model achieves 39.41%.
- Results: The proposed approach performs better on some large-vocabulary slots and also improves over closed-vocabulary tracking on some smaller-vocabulary slots.The authors hypothesize that frequent appearance in user utterances helps the model learn these values from dialog context.
- Results: 46.28% joint-goal accuracy is achieved by a hybrid that selects per-slot predictions from the proposed and JST models.The selection uses slot-type accuracy measured on the development set.
- Ablation Analysis: 60.18% development-set joint goal accuracy results when slot carryover is replaced with an oracle, compared with 41.43% and 45.77% for replacing the other components.The span model already reaches 96% per-slot accuracy, whereas slot carryover reaches 72% per turn.
- Error Analysis: Unanswerable slot errors account for 65.5% of sampled errors, making slot carryover a major barrier even with a perfect span model.Imprecise slot resolution accounts for 12.9% of errors and motivates resolving context words to ontology values.
- Conversation Depth: Performance degrades as conversation turns increase because earlier errors can be carried over into later turns.The authors motivate models that are more robust to conversation depth.
6 Conclusion
The paper frames dialog state tracking as selecting slot values directly from conversational context, addressing fixed-vocabulary limitations. It combines attention-based reading comprehension with slot carryover and contextual embeddings, improving joint goal accuracy on MultiWOZ-2.0.
- Dialog state tracking must select slot values despite vocabularies that can contain thousands of values and millions of possible dialogs.
- Attention-based neural networks provide a vocabulary-free basis for selecting slot values by pointing directly to spans in the dialog context.
- The proposed model encodes dialog context and points to slot values within the conversation using a simple attention-based neural network.
- The model adds a slot carryover component and combines contextual word embeddings with a traditional fixed-vocabulary approach to improve joint goal accuracy on MultiWOZ-2.0.
- Ablation studies identify the slot carryover model as important, motivating further work on predicting whether a slot state is None or not.
- Viewing human conversation as context may allow dialog-related tasks to benefit from advances in modern reading comprehension models.