Source-linked AI summary
Multi-domain Dialog State Tracking using Recurrent Neural Networks
Nikola Mrkšić, Diarmuid Ó Séaghdha, Blaise Thomson, Milica Gašić, Pei-Hao Su, David Vandyke, Tsung-Hsien Wen, Steve Young
TL;DR
Dialog state tracking is commonly designed for a single domain, limiting generalization across different applications. This paper trains and specializes multi-domain RNN belief trackers using shared dialog patterns, then uses them to initialize new-domain trackers. The models outperform most domain-specific trackers and improve adaptation even when substantial in-domain data are available.
Problem
Dialog systems are typically built for a single well-defined domain, while belief tracking across disjoint domains remains underexplored.
Method
A hierarchical procedure trains a shared RNN on delexicalised dialog data from multiple domains and then specializes it for individual domains or new-domain adaptation.
Results
The general model shows robust performance across domains and outperforms most domain-specific models, while initialization improves tracking with any amount of in-domain data.
Takeaways & Limitations
Diverse dialog domains can help RNN trackers capture general dialog dynamics applicable across multiple domains and support adaptation to new ones.
Takeaways & Limitations
Adapting the general model to a new domain still requires a small collection of annotated in-domain dialogs.
Abstract
from arXiv · showhide
Dialog state tracking is a key component of many modern dialog systems, most of which are designed with a single, well-defined domain in mind. This paper shows that dialog data drawn from different dialog domains can be used to train a general belief tracking model which can operate across all of these domains, exhibiting superior performance to each of the domain-specific models. We propose a training procedure which uses out-of-domain data to initialise belief tracking models for entirely new domains. This procedure leads to improvements in belief tracking performance regardless of the amount of in-domain data available for training the model.
1 Introduction
Dialog state trackers are usually built for one domain, but this paper develops RNN models that operate across different domains by learning shared dialog patterns. Cross-domain training produces robust performance and improves adaptation to new domains.
- Most modern dialog systems target a single, well-defined domain, while open-domain conversation remains difficult.
- The paper proposes multi-domain RNN belief trackers that use dialog data from different domains to learn generalizable dialog features.
- The hierarchical procedure first learns general dialog features across domains, then specializes models for individual domains while retaining cross-domain patterns.
- The resulting trackers show robust performance across investigated domains and typically outperform models trained only on target-domain data.
- Cross-domain initialization improves tracking for entirely new domains regardless of the amount of in-domain training data available.
2 Related Work
Prior work established RNN belief tracking as a strong, domain-light alternative to complex semantic decoders, while broader cross-domain belief tracking remained largely unaddressed.
- Data-driven dialog trackers replaced traditional rule-based systems because they are more robust and provide probabilistic state distributions for POMDP managers.
- RNN belief trackers map ASR output directly to belief-state updates without complex semantic decoders while achieving state-of-the-art performance.
- Earlier domain-adaptation work in dialog addressed tagging, generation, or expanded versions of the same topic rather than disjoint dialog domains.
3 Dialog State Tracking using RNNs
The tracker uses a recurrent network to update slot-value beliefs from conversational context and lexical features. Delexicalisation enables transfer across slots and unseen values, although ordinary lexical n-grams do not share information well across domains.
- Belief tracking represents user goals as slot-value constraints, with each domain’s slots and possible values defining its ontology.
- The RNN outputs a distribution over goal slot-value pairs for each utterance while maintaining a memory vector for dialog context.
- Each turn combines ASR hypotheses, the last system action, the current memory vector, and the previous belief state as model input.
- Lexical features capture dialog dynamics but are not ideal for sharing information across different slots and domains.
- Delexicalised n-grams replace slot names and values with generic tags, allowing transfer across slots and operation on unseen values or new slots.
- The model updates a separate distribution for each slot and value while also updating its recurrent memory state.
- Training unrolls the model across turns using backpropagation through time and stochastic gradient descent.
- The architecture omits the original component mapping lexical n-grams to specific slot values to move toward domain independence.
4 Hierarchical Model Training
Hierarchical multi-domain training first learns shared dialog representations from delexicalised data, then preserves general dynamics while learning domain- and slot-specific feature importance.
- Delexicalisation enables transfer between domains because generic features learned for one topic can support tracking in another.
- Shared initialization ties parameters across slots and replaces slot-value occurrences with one generic tag before training a shared RNN.
- The multi-domain shared model is trained on a combined dataset formed by delexicalising slot values across all domains.
- The shared RNN extracts lexical and delexicalised features capturing general dialog dynamics, while later specialization learns their differing relevance to particular slots.
5 Dialog domains considered
The experiments evaluate belief tracking across six domains and construct increasingly general training sets by combining restaurant, related service, and all available dialog data.
- Joint goal accuracy measures whether the tracker correctly infers users’ constraints across all slots in a domain.
- The evaluation covers six domains varying in topic and geographical location, including restaurant, hotel, tourist-information, and laptop dialogs.
- The Laptops dataset is substantially different from the other domains, providing a challenging test of multi-domain models.
- The All Restaurants model combines data from three restaurant domains, while the R+T+H model adds hotels, pubs, and coffee shops.
- The R+T+H+L model is the most general model, trained on all available dialog data.
6 Results
Multi-domain and slot-specialised RNN trackers generally outperform domain-specific baselines, while out-of-domain data provides especially useful initialization when in-domain data is scarce. The broadest model balances performance across six domains, and this benefit persists even with the full in-domain dataset.
- 6.1 Training General Models: Multi-domain models improve over domain-specific models for all but one constituent domain, and the broadest model achieves the highest geometric mean across six domains.The R+T+H model performs better across four domains than R+T+H+L, but the latter balances performance more effectively across all six domains.
- 6.2 Slot-specialising the General Models: Slot-specialisation improves performance in the vast majority of experiments, and all three slot-specialised general models outperform the RNN result reported in DSTC 2.
- 6.3 Out of Domain Initialisation: The hierarchical procedure uses out-of-domain dialogs to initialise improved shared models for new domains.The evaluation compares slot-specialised models trained from in-domain-only and combined in-domain plus out-of-domain shared models.
- 6.3 Out of Domain Initialisation: Out-of-domain initialization improves joint goal accuracy even when the entire in-domain dataset is available, with the strongest starting-point benefit when in-domain data is small.Figure 1 evaluates Michigan Restaurants and Laptops as a function of available in-domain training dialogs.
- 6.3 Out of Domain Initialisation: The Laptops experiment shows a larger and more consistent initialization advantage because out-of-domain dialogs provide delexicalised features absent from the in-domain training data.Additional in-domain dialogs then supply features not present in the out-of-domain data, producing continued improvements.
7 Conclusion
The paper demonstrates that general belief-tracking models can handle multiple dialog domains and that diverse-domain training captures dialog dynamics shared across them. Its hierarchical procedure also adapts these models to new domains with limited annotated data, though annotation is still required.
- General belief-tracking models can address multiple dialog topics at once.
- Training on diverse domains helps models capture dialog dynamics applicable across domains.
- The most general model robustly outperforms most domain-specific models across the investigated domains.
- Hierarchical training adapts the general model to new domains using very small in-domain datasets.
- The adaptation procedure still improves tracking when substantial in-domain data are available, but it requires annotated in-domain dialogs.