Source-linked AI summary
Transferable Multi-Domain State Generator for Task-Oriented Dialogue Systems
Chien-Sheng Wu, Andrea Madotto, Ehsan Hosseini-Asl, Caiming Xiong, Richard Socher, Pascale Fung
TL;DR
Dialogue state tracking is challenged by dependence on predefined ontologies, unknown slot values, and adaptation to new domains. TRADE uses a shared, copy-based dialogue state generator with a slot gate and encoder, achieving strong MultiWOZ performance while supporting zero-shot and few-shot transfer. The paper also identifies dataset-label incompleteness and challenging unseen slots as boundaries on evaluation and transfer.
Problem
Predefined ontologies are difficult to enumerate and can contain many slot values, while existing approaches struggle with unknown values and adapting to new domains.
Method
TRADE shares an utterance encoder, slot gate, and state generator across domains, using a copy mechanism to generate values without a predefined ontology.
Results
TRADE achieves state-of-the-art joint goal and slot accuracy on five MultiWOZ domains and supports zero-shot and few-shot DST for unseen domains.
Takeaways & Limitations
Domain sharing enables zero-shot DST for unseen domains and rapid few-shot adaptation without forgetting learned domains.
Takeaways & Limitations
Evaluation can be incorrect when labels for the (hotel, type) pair are missing, and zero-shot transfer is extremely challenging when the target slot is absent from the source domains.
Abstract
from arXiv · showhide
Over-dependence on domain ontology and lack of knowledge sharing across domains are two practical and yet less studied problems of dialogue state tracking. Existing approaches generally fall short in tracking unknown slot values during inference and often have difficulties in adapting to new domains. In this paper, we propose a Transferable Dialogue State Generator (TRADE) that generates dialogue states from utterances using a copy mechanism, facilitating knowledge transfer when predicting (domain, slot, value) triplets not encountered during training. Our model is composed of an utterance encoder, a slot gate, and a state generator, which are shared across domains. Empirical results demonstrate that TRADE achieves state-of-the-art joint goal accuracy of 48.62% for the five domains of MultiWOZ, a human-human dialogue dataset. In addition, we show its transferring ability by simulating zero-shot and few-shot dialogue state tracking for unseen domains. TRADE achieves 60.58% joint goal accuracy in one of the zero-shot domains, and is able to adapt to few-shot cases without forgetting already trained domains.
1 Introduction
Dialogue state tracking must extract user goals as slot–value states across domains and turns, but predefined ontologies limit coverage of unknown values. TRADE addresses this with shared, transferable tracking mechanisms for multi-domain and unseen-domain settings.
- Motivation: Dialogue state tracking extracts user goals as compact sets of slots and corresponding values from conversation.Examples include (price, cheap) and (area, centre).
- Motivation: Predefined ontologies are difficult to obtain completely and may contain too many possible slot values for neural classification.Industry databases may be exposed only through externally maintained APIs, while restaurant names and train departure times can have many values.
- Motivation: MultiWOZ introduces mixed-domain conversations requiring prediction of domain, slot, and value across 30 pairs and over 4,500 possible slot values.A conversation may move from restaurant reservation to attraction information and taxi booking.
- Motivation: Sharing tracking knowledge across domains is motivated by overlapping slot names and values, including area across restaurant, attraction, and taxi domains.The restaurant name value can also correspond to the taxi departure slot, motivating transfer to unseen domains.
- Approach: TRADE proposes a transferable dialogue state generator for multi-domain task-oriented dialogue state tracking.Its stated advantages are simplicity and improved performance.
- Approach: TRADE uses a context-enhanced slot gate and copy mechanism to address multi-turn mapping and track slot values.The approach shares parameters across domains and does not require a predefined ontology.
- Transfer: TRADE supports zero-shot DST for unseen domains and few-shot adaptation without forgetting previously learned domains.These capabilities are attributed to leveraging and sharing knowledge across domains.
2 TRADE Model
TRADE shares an utterance encoder, slot gate, and state generator across domains while directly generating slot values with a copy mechanism. The slot gate identifies triggered pairs, and joint training optimizes gating and value generation.
- Architecture: TRADE shares an utterance encoder, slot gate, and state generator across domains, using domain-slot-specific start tokens for value generation.The state generator decodes all possible (domain, slot) pairs independently.
- Utterance Encoder: The utterance encoder represents a recent dialogue history of length l rather than only the current utterance.The encoder uses bidirectional GRUs to encode concatenated utterances and responses.
- State Generator: Soft-gated pointer-generator copying combines vocabulary and dialogue-history distributions, allowing words outside the predefined vocabulary.This mechanism supports generating slot values from text in the input source.
- Slot Gate: The context-enhanced slot gate predicts ptr, none, or dontcare for each domain-slot pair and determines whether generated values are retained.Pairs predicted as none or dontcare are filled as “not-mentioned” or “does not care”.
- Optimization: Training minimizes a weighted sum of slot-gate and value-generation cross-entropy losses across domain-slot pairs and decoding steps.The weighting uses hyper-parameters α and β.
3 Unseen Domain DST
TRADE is evaluated for transferring dialogue state tracking to unseen domains under zero-shot and few-shot settings. Few-shot adaptation uses limited target-domain data and continual-learning methods to preserve performance on previously learned domains.
- Transfer Settings: Zero-shot transfer assumes no training data in the new domain, whereas few-shot expansion uses around 20 to 30 dialogues, or 1% of the original training data.The evaluation targets generalization and adaptation to unseen domains.
- Zero-shot DST: A zero-shot model must generate target-domain slot values from context, domain, and slot inputs without target-domain training samples.The setting is especially challenging when target slots never appeared among source slots.
- Few-shot Domain Expansion: Few-shot domain expansion aims to adapt quickly with limited data while avoiding forgetting previously learned domains and retraining on all prior data.This addresses the cost and availability constraints of collecting and retaining large datasets.
- Continual Learning: The few-shot baselines and methods include unconstrained fine-tuning, elastic weight consolidation, and gradient episodic memory.EWC regularizes adaptation using source-domain Fisher information, while GEM constrains gradients using stored source samples.
4 Experiments
Experiments evaluate TRADE on MultiWOZ for multi-domain tracking, zero-shot transfer, and few-shot domain expansion. Results show strong joint and slot accuracy, transfer to unseen domains, and improved retention or adaptation with continual-learning strategies.
- Dataset: MultiWOZ contains 8,438 multi-turn dialogues across seven domains, with 30 domain-slot pairs and over 4,500 possible values; experiments use five domains.The hospital and police domains are excluded because they have few dialogues and appear only in training.
- Zero-shot transfer: 60.58% joint goal accuracy is achieved for taxi in zero-shot evaluation without taxi training samples.Taxi performance is attributed to overlap between its slot values and corresponding train-domain slots; other unseen domains achieve around 50–65% slot accuracy.
- Few-shot domain expansion: GEM outperforms naive and EWC fine-tuning in preserving performance on the original four domains during domain expansion.For hotel, GEM reduces joint accuracy from 58.98% to 53.54% (-5.44%), while naive fine-tuning reduces it to 36.08% (-22.9%).
- Few-shot domain expansion: 59.83% joint accuracy is obtained after fine-tuning with 1% of Train-domain data, compared with 44.24% when training from scratch on the same amount.Fine-tuning from the four-domain base model generally outperforms training from scratch on the new domain.
- Few-shot domain expansion: GEM reaches 34.73% joint accuracy on the new attraction domain, versus 29.39% for naive fine-tuning.The comparison concerns domain expansion where hotel and attraction are treated as new domains.
5 Error Analysis
TRADE’s errors vary substantially by slot and domain, with name slots particularly difficult because they have many possible values. Cross-domain transfer succeeds for several shared slots but struggles on slots unique to one domain, while missing labels also create apparent errors.
- Restaurant, attraction, and hotel name slots have the highest error rates: 8.50%, 8.17%, and 7.86%, respectively.The paper attributes this difficulty to their large numbers of possible values.
- 8.50% is the highest slot error rate, for the restaurant name slot; taxi arrive by has the lowest at 1.33%.
- Hotel type has a high error rate despite only two ontology values because labels for the (hotel, type) pair are usually missing.Missing labels can make predictions incorrect even when the slot should be predicted.
- People, area, price range, and day slots transfer successfully to unseen hotel and restaurant domains.The analysis excludes labels marked “none” because predicting them is relatively easier.
- Slots appearing in only one domain are very difficult for TRADE to track correctly.
6 Related Work
Related dialogue-state-tracking work relies on handcrafted features, domain-specific lexicons, ontology matching, or varied parameter-sharing strategies. Prior research also explores multi-domain, zero-shot, and few-shot learning across dialogue-system components.
- Traditional DST approaches combine extracted semantics with dialogue-state estimation or jointly learn speech understanding, often using handcrafted features and domain-specific lexicons.
- Mrkˇsi´c et al. do not share parameters across slots, whereas Nouri and Hosseini-Asl use global modules and Zhong et al. use slot-specific local modules.
- Multi-domain approaches use bi-GRU or bi-LSTM architectures, with some relying on delexicalization and others matching utterances to ontology terms.
- Zero/Few-Shot and Continual Learning: Zero-shot methods have been studied for intention classification, slot filling, dialogue policy, machine translation, and dialogue generation.
- Zero/Few-Shot and Continual Learning: Few-shot learning has also been applied to semantic parsing and machine translation.
7 Conclusion
TRADE tracks dialogue states without a predefined domain ontology by sharing parameters across domains. It achieves strong MultiWOZ performance, supports zero-shot transfer and few-shot expansion, and remains a basis for future resource-based transfer and meta-learning.
- TRADE learns dialogue-state tracking without any predefined domain ontology.
- TRADE shares all parameters across domains and achieves state-of-the-art joint goal and slot accuracy on five MultiWOZ domains.
- Domain sharing enables zero-shot DST for unseen domains and few-shot adaptation without forgetting learned domains.
- Future work includes transferring knowledge from other resources and collecting larger multi-domain datasets for meta-learning.