Source-linked AI summary
Find or Classify? Dual Strategy for Slot-Value Predictions on Multi-Domain Dialog State Tracking
Jian-Guo Zhang, Kazuma Hashimoto, Chien-Sheng Wu, Yao Wan, Philip S. Yu, Richard Socher, Caiming Xiong
TL;DR
DST must handle slot values that may either come from candidate ontologies or appear as spans in dialog context, while the best slot handling remains unclear. The paper introduces a single BERT-style model that jointly uses both strategies with direct context–slot interactions, achieving state-of-the-art results on MultiWOZ 2.0 and competitive results on MultiWOZ 2.1. Its analysis also examines slot-specific strategy effects and evaluation limitations.
Problem
Existing ontology-based and ontology-free DST methods have complementary limitations, and it remains unclear which slots each strategy should handle.
Method
DS-DST adapts a single BERT question-answering model to select categorical values from partial ontologies and extract non-categorical values from dialog context with direct context–slot interactions.
Results
The model achieves state-of-the-art results on MultiWOZ 2.0 and competitive results on MultiWOZ 2.1.
Takeaways & Limitations
Comprehensive slot-level error analysis shows that dual-strategy effects vary by slot and is intended to facilitate future DST research.
Takeaways & Limitations
MultiWOZ evaluation is constrained by annotation errors, ambiguity, inconsistency, and nonstandard preprocessing that can make comparisons unfair.
Abstract
from arXiv · showhide
Dialog state tracking (DST) is a core component in task-oriented dialog systems. Existing approaches for DST mainly fall into one of two categories, namely, ontology-based and ontology-free methods. An ontology-based method selects a value from a candidate-value list for each target slot, while an ontology-free method extracts spans from dialog contexts. Recent work introduced a BERT-based model to strike a balance between the two methods by pre-defining categorical and non-categorical slots. However, it is not clear enough which slots are better handled by either of the two slot types, and the way to use the pre-trained model has not been well investigated. In this paper, we propose a simple yet effective dual-strategy model for DST, by adapting a single BERT-style reading comprehension model to jointly handle both the categorical and non-categorical slots. Our experiments on the MultiWOZ datasets show that our method significantly outperforms the BERT-based counterpart, finding that the key is a deep interaction between the domain-slot and context information. When evaluated on noisy (MultiWOZ 2.0) and cleaner (MultiWOZ 2.1) settings, our method performs competitively and robustly across the two different settings. Our method sets the new state of the art in the noisy setting, while performing more robustly than the best model in the cleaner setting. We also conduct a comprehensive error analysis on the dataset, including the effects of the dual strategy for each slot, to facilitate future research.
1 Introduction
Dialog state tracking estimates users’ goals from conversation history by maintaining domain-slot-value states across turns. The paper proposes a dual-strategy BERT model that combines categorical and non-categorical slot handling with direct context–slot interactions.
- DST tracks accumulated < domain, slot, value > triplets across conversational turns to estimate users’ goals and intentions.
- Ontology-based methods classify values from candidate lists but require complete ontologies and can be expensive for large, diverse slot vocabularies.
- Ontology-free methods extract values from dialog spans without ontologies, but struggle when values are absent from context or described variably.
- DS-DST adapts one BERT question-answering model to jointly process categorical slots through partial ontologies and non-categorical slots through context spans.
- The model achieves state of the art on MultiWOZ 2.0 and competitive performance on MultiWOZ 2.1, while also providing comprehensive error analysis.
2 Related Work
Prior multi-domain DST methods rely on handcrafted features, fixed ontologies, separate context and slot encodings, or generation. This work emphasizes stronger context–domain-slot interactions and analyzes slot-type decisions on MultiWOZ.
- Traditional multi-domain DST methods use handcrafted features or domain-specific lexicons, making adaptation to new domains difficult.
- BERT-DST encodes user and previous system turns, whereas this work jointly models dialog context and domain-slot pairs through stronger interactions.
- The paper investigates how to decide slot types and conducts comprehensive analysis of the MultiWOZ datasets.
- Generative DST avoids fixed vocabularies and spans but can produce ill-formatted strings, especially for long values such as hotel addresses.
3 DS-DST: a Dual Strategy for DST
DS-DST uses direct interactions between each domain-slot pair and the dialogue context, then applies a slot gate to route categorical slots to candidate-value selection and non-categorical slots to span extraction. Its modules are jointly trained with shared BERT parameters and distinct losses for gating, spans, and picklist matching.
- 3.1 Slot-Context Encoder: DS-DST encodes each domain-slot pair together with the dialogue context to produce contextualized representations for slot-value prediction.The BERT encoder is fine-tuned during training, and its outputs include an aggregate representation and token-level representations.
- 3.2 Slot-Gate Classification: A slot gate classifies each domain-slot pair at each turn as none, dontcare, or prediction.The gate uses the [CLS] representation and a softmax classifier with learnable weights and bias.
- 3.3 Non-Categorical Slot-Value Prediction: For non-categorical slots, DS-DST predicts value spans by applying a two-way linear mapping to dialogue-context token representations and learning start and end positions.The non-categorical loss combines the start-position and end-position losses.
- 3.4 Categorical Slot-Value Prediction: For categorical slots, a fixed BERT feature extractor represents candidate values, which are ranked against the slot-context representation using cosine similarity and a hinge loss.The picklist module requires only a partial ontology for DS-DST.
- 3.5 Training Objective: The model jointly optimizes gate, span, and picklist losses while sharing BERT parameters across modules.For none and dontcare slots, span and picklist losses are set to zero, leaving only gate classification to be optimized.
4 Experimental Setup
The experiments evaluate DST models on MultiWOZ 2.0 and 2.1 using joint accuracy, comparing ontology-based, ontology-free, and dual-strategy variants under dataset-specific preprocessing. The proposed DS-DST jointly handles categorical and non-categorical slots, while baseline variants treat all slots as one type or combine span extraction with categorical handling.
- Datasets and evaluation: Experiments use MultiWOZ 2.0 and 2.1, with seven-domain data described for MultiWOZ 2.0 and corrections reducing noise in MultiWOZ 2.1.The experiments follow prior work by focusing on five domains and excluding hospital and police because they contain few dialogues and appear only in training.
- Models and evaluation: The study compares models using joint accuracy, including SpanPtr, Ptr-DST, DSTreader, DSTQA, TRADE, COMER, BERT-DST, and SST-2.The comparison also includes the proposed DS-DST, DS-Span, and DS-Picklist variants.
- Comparison scope: Comparisons with existing systems require care because some baselines use different dataset preprocessing, and the authors therefore exclude Lee et al. and Shan et al.Table 2 reports joint accuracy on MultiWOZ 2.0 and 2.1, while Table 3 reports joint accuracy on MultiWOZ 2.1 under different BERT input settings.
- Model design: The proposed model uses BERT with domain-slot and dialogue-context information, then applies span prediction for non-categorical slots and picklist selection for categorical slots.BERT-DST and related baselines provide comparison points for schema encoding, copying, and slot-interaction approaches.
- Proposed variants: DS-DST jointly handles categorical and non-categorical slots, assigning time- and number-related slots to the non-categorical type and the remaining slots to the categorical type.DS-Span treats every domain-slot pair as non-categorical, whereas DS-Picklist treats every pair as categorical and assumes a full ontology is available.
5 Experimental Results
DS-DST achieves strong and comparatively robust performance across MultiWOZ settings by jointly handling categorical and non-categorical slots, with evidence that slot-context interaction and slot-specific strategy selection drive gains.
- Evaluation: Joint accuracy is 1.0 only when every domain-slot-value triplet is correct, making the metric stringent for multi-domain tracking.The score is averaged across evaluation turns.
- Ablation and strategy design: Jointly training categorical and non-categorical modules improves DS-DST by 1.90% on MultiWOZ 2.1 compared with separate training.When the full ontology is available, DS-Picklist improves further, although such access may be impractical in real scenarios.
- BERT-based comparisons: DS-DST outperforms BERT-DST by 7.81% on MultiWOZ 2.1, supporting strong interactions between dialog context and domain-slot pairs.DS-Span also outperforms SpanPtr by 10.91%, while DS-DST outperforms BERT-DST, which separately encodes context and domain-slot pairs.
- Slot-level analysis: Categorical handling improves slots whose values are expressed differently or absent from the dialog, while either strategy performs similarly when values are explicitly mentioned.Examples include hotel-type, attraction-type, attraction-name, hotel-internet, and hotel-parking.
- Error analysis: Replacing the slot-gate classifier with an oracle raises development-set joint accuracy from 55.23% to 86.10%, indicating substantial room for better gate designs.The analysis identifies slot-gate classification as an important remaining performance bottleneck.
6 Conclusion
The paper proposes a dual-strategy model for multi-domain dialog state tracking that combines partial-ontology selection for categorical slots with context span extraction for non-categorical slots. It reports state-of-the-art results on MultiWOZ 2.0, competitive results on MultiWOZ 2.1, and comprehensive dataset analysis.
- The model uses strong interactions between dialog context and domain-slot pairs.
- DS-DST predicts categorical slot values from a partial ontology and non-categorical values from the dialog context.
- The models achieve state-of-the-art results on MultiWOZ 2.0 and competitive results on MultiWOZ 2.1.
- The paper provides a comprehensive analysis of the dataset to facilitate future research.
A.1 Training Details
Training uses a pretrained bert-base-uncased BERT model, updates all parameters with BertAdam, and selects the best checkpoint using validation joint accuracy. The experiments use fixed sequence, optimization, batching, and epoch settings.
- The model uses pretrained bert-base-uncased BERT and updates all parameters during fine-tuning with BertAdam.
- The maximum WordPiece-tokenized input sequence length is 512.
- The constant margin λ is 0.5 for DS-DST and DS-Picklist.
- The initial learning rate is 1e−4, warmup proportion is 0.1, batch size is 16, and maximum training length is 5 epochs.
- The best checkpoint is selected using validation joint accuracy checked every 1,000 iterations for final test evaluation.
A.2 Sample Output
The sample-output appendix compares DS-Span and DS-DST predictions across train, hotel, taxi, attraction, and restaurant domains in two MultiWOZ 2.1 validation dialogues. The examples illustrate span-extraction failures when values are implicit or differently phrased.
- The appendix presents DS-Span and DS-DST outputs for all turns in validation dialogues MUL0729 and PMUL2428.
- DS-Span cannot match hotel internet value yes when the dialog uses the description free wifi.
- Table 7 covers train and hotel domains in dialogue MUL0729, while Table 8 covers taxi, attraction, and restaurant domains in PMUL2428.