Source-linked AI summary
SUMBT: Slot-Utterance Matching for Universal and Scalable Belief Tracking
Hwaran Lee, Jinsik Lee, Tae-Yoon Kim
TL;DR
Belief trackers must handle changing domains, slots, and slot-values, while prior approaches are often domain- or slot-dependent. SUMBT uses contextual semantic encoders, attention-based slot-utterance matching, and non-parametric slot-value prediction. It achieved state-of-the-art joint accuracy on WOZ 2.0 and MultiWOZ.
Problem
Existing trackers have difficulty scaling across domains and slots and adding new slot-values not defined in the ontology.
Method
SUMBT uses contextual BERT representations and attention to match domain-slot types with utterances, then predicts slot-values non-parametrically.
Results
SUMBT achieved state-of-the-art joint accuracy on both WOZ 2.0 and MultiWOZ.
Takeaways & Limitations
A single SUMBT can handle multiple domains and slot-types without increasing model size, while shared learning across domains improves performance.
Abstract
from arXiv · showhide
In goal-oriented dialog systems, belief trackers estimate the probability distribution of slot-values at every dialog turn. Previous neural approaches have modeled domain- and slot-dependent belief trackers, and have difficulty in adding new slot-values, resulting in lack of flexibility of domain ontology configurations. In this paper, we propose a new approach to universal and scalable belief tracker, called slot-utterance matching belief tracker (SUMBT). The model learns the relations between domain-slot-types and slot-values appearing in utterances through attention mechanisms based on contextual semantic vectors. Furthermore, the model predicts slot-value labels in a non-parametric way. From our experiments on two dialog corpora, WOZ 2.0 and MultiWOZ, the proposed model showed performance improvement in comparison with slot-dependent methods and achieved the state-of-the-art joint accuracy.
1 Introduction
Belief tracking updates dialog-state distributions, but existing approaches remain difficult to scale across domains, slots, and newly added slot-values. SUMBT addresses this with a single domain- and slot-independent tracker that matches domain-slot types to utterances and evaluates slot-values non-parametrically.
- Belief tracking estimates dialog-state distributions over user intents, informable slots, and requestable slots as conversations progress.
- Existing neural trackers improve semantic representation learning, but scalability remains limited by domain- or slot-specific modeling and difficulty adding ontology-external slot-values.
- SUMBT uses one belief tracker for any domain and slot-type, making the architecture domain- and slot-independent.
- SUMBT encodes utterances, domain-slot types, and slot-values with BERT, attends to relevant utterance words, and predicts slot-value labels non-parametrically.
- The paper evaluates SUMBT on WOZ 2.0 and MultiWOZ and includes qualitative analysis of how the model works.
2 SUMBT
SUMBT encodes utterances, domain-slot types, and slot-values with contextual BERT representations, then matches slot queries to utterance words. An RNN tracks dialog history, while a distance-based discriminator assigns slot-value probabilities.
- SUMBT comprises BERT encoders, a slot-utterance matching network, an RNN belief tracker, and a non-parametric discriminator.
- Contextual Semantic Encoders: BERT provides contextual word and sequence representations, including embeddings for multiword slot-types and slot-values.
- Contextual Semantic Encoders: The model encodes domain-slot types and slot-values with BERT, using fixed slot-type encoder weights to support new domain and slot-type pairs.
- Slot-Utterance Matching: Multi-head attention uses the domain-slot representation as a query to retrieve relevant information from contextual utterance vectors.
- Belief Tracker: The attended context vector is fed into an RNN, whose output is trained to approach the target slot-value’s semantic vector.
- Training Criteria: Slot-value probabilities use distances between predicted and candidate semantic vectors, while joint training across domain-slot types learns general slot-type–value relations.
3 Experimental Setup
The experiments use WOZ 2.0 and MultiWOZ, covering a single restaurant domain and a larger seven-domain setting. Baselines include contextual encoders, ontology components, and slot-dependent SUMBT variants, with some requiring retraining after ontology changes.
- Datasets: WOZ 2.0 contains one restaurant-reservation domain with three slots, whereas MultiWOZ contains 35 slots across seven domains.
- Baselines: The baselines are BERT+RNN, BERT+RNN+Ontology, and slot-dependent SUMBT.
- Baselines: BERT+RNN and BERT+RNN+Ontology use candidate-dependent linear output layers, so ontology changes require retraining.
- Implementation: The BERT encoder used 12 layers, 784 hidden units, and 12 self-attention heads, with multi-head matching tested using 4 or 8 heads.
4 Experimental Results
On WOZ 2.0 and MultiWOZ, SUMBT was evaluated against prior and baseline trackers using joint accuracy. It achieved 91.0% joint accuracy on WOZ 2.0 and 42.4% on MultiWOZ, while its attention aligned with semantically related utterance phrases.
- Joint Accuracy Performance: The WOZ 2.0 evaluation reports joint goal accuracy for SUMBT alongside baseline and previously proposed models.The contextual semantic encoder models beat all previous models in the reported comparison.
- Joint Accuracy Performance: 91.0% joint accuracy was achieved by slot-independent SUMBT on WOZ 2.0, significantly outperforming the baseline models.The slot-dependent SUMBT and three BERT-based baselines showed no significant performance differences.
- Joint Accuracy Performance: 42.4% joint accuracy was achieved by SUMBT on MultiWOZ, surpassing previous approaches and establishing state-of-the-art performance.MultiWOZ includes more domains and slots than WOZ 2.0.
- Attention Weights Analysis: SUMBT attended to semantically related utterance phrases even when their wording differed from the slot-value label.For the price range value ‘moderate’, attention was high on ‘reasonably priced’.
5 Conclusion
The paper concludes that SUMBT provides a universal and scalable belief tracker by matching domain-slot-types to relevant utterance words. Its results show state-of-the-art joint accuracy on WOZ 2.0 and MultiWOZ, with shared learning across domains improving performance.
- Conclusion: SUMBT uses relevant utterance words, contextual semantic encoders, and a non-parametric discriminator to handle multiple domains and slot-types with one model.The approach is intended to avoid increasing model size as domains and slot-types expand.
- Conclusion: SUMBT achieved state-of-the-art joint accuracy on both the WOZ 2.0 and MultiWOZ corpora.The conclusion also reports improved performance from sharing knowledge across multiple domain data.
- Conclusion: The paper identifies continual learning of new knowledge after domain-ontology updates as future work.This is presented as a planned direction rather than an evaluated capability.