Source-linked AI summary
An End-to-end Approach for Handling Unknown Slot Values in Dialogue State Tracking
Puyang Xu, Qi Hu
TL;DR
Dialogue state trackers often depend on predefined candidate lists, limiting their ability to output unknown slot values, especially without an SLU module. This paper uses a pointer-network-based end-to-end tracker with targeted feature dropout; it reports state-of-the-art DSTC2 accuracy and improved unknown-value performance.
Problem
Many DST approaches assume predefined candidate lists and are not designed to output unknown slot values, particularly when end-to-end systems lack an SLU module.
Source: §1 · pp. 1–2Method
Results
The hybrid tracker achieves state-of-the-art accuracy on DSTC2; with 5% dropout, unknown-food accuracy rises from 11.6% to 34.4%.
Source: §7 · p. 8, §6.4 · p. 8Takeaways & Limitations
The approach can extract unknown slot values that are explicitly mentioned in dialogue while maintaining state-of-the-art accuracy on DSTC2.
Source: §7 · p. 8Extracted surface forms may lack canonical forms required for backend database lookup and can make retrieval and accuracy computation harder.
Source: §4 · p. 4
Abstract
from arXiv · showhide
We highlight a practical yet rarely discussed problem in dialogue state tracking (DST), namely handling unknown slot values. Previous approaches generally assume predefined candidate lists and thus are not designed to output unknown values, especially when the spoken language understanding (SLU) module is absent as in many end-to-end (E2E) systems. We describe in this paper an E2E architecture based on the pointer network (PtrNet) that can effectively extract unknown slot values while still obtains state-of-the-art accuracy on the standard DSTC2 benchmark. We also provide extensive empirical evidence to show that tracking unknown values can be challenging and our approach can bring significant improvement with the help of an effective feature dropout technique.
1 Introduction
DST maintains dialogue states for a dialogue manager, but fixed ontologies can be unavailable, large, or dynamic. The paper proposes an end-to-end pointer-network tracker for unknown slot values and targeted dropout to improve their recall.
1 Introduction
- Fixed ontologies may be inaccessible or impractical to enumerate when partner databases are hidden, value lists are large, or values change frequently.These conditions challenge approaches that classify over a fixed ontology or score enumerated candidates.Source: §1 · p. 1
- The paper targets unknown slot values when a comprehensive domain ontology is absent, a problem the authors describe as rarely investigated.DST monitors user intent and provides compact dialogue states for the dialogue manager.Source: §1 · p. 1
- The proposed end-to-end tracker uses a pointer network to handle unknown values without a separate SLU module.The pointer network extracts values from the dialogue text, unlike trackers that require candidate lists.Source: §1 · p. 2
- Rastogi et al. (2018) also address dynamic value sets, but their candidate-generation approach relies on SLU and adds a module that can propagate errors.The proposed technique is similar in motivation but does not rely on SLU-generated value candidates.Source: §1 · p. 2
- Targeted dropout is introduced as a training technique that drastically improves recall of unknown slot values.The introduction also notes that it can be essential for generalization on some datasets.Source: §1 · p. 2
2 Dialogue State Tracking
DST represents user goals and requests as slot values tracked from utterances and dialogue context. Prior methods often rely on enumerable value sets, while unknown values may be unavailable to trackers, particularly end-to-end systems without SLU.
2 Dialogue State Tracking
- DST tracks informable search constraints and requestable attributes using the current utterance and dialogue context.The values are conventionally specified in advance through an ontology, an assumption that motivates the paper.Source: §2 · p. 2
- Generative DST models face scalability issues from large state spaces, while discriminative methods model state posteriors and have achieved state-of-the-art benchmark results.Discriminative systems commonly use multi-class classification or binary prediction over candidates from an ontology or SLU.Source: §2 · pp. 2–3
- End-to-end DST removes the separate SLU module, reducing development cost and alleviating error propagation, but commonly learns from raw dialogue history without fine-grained semantic features.Such systems typically use RNNs or memory networks to learn features from the history.Source: §2 · p. 3
- Most prior DST approaches, especially end-to-end ones, are not designed for values unknown to the tracker, and classification or candidate scoring can be difficult when values are not enumerable.Delexicalization can generalize to infrequent or unseen values, but still requires values to be known through an ontology or supplied by SLU.Source: §2 · p. 3
3 Pointer Network
A pointer network predicts positions in the dialogue input, making slot values that appear explicitly in the conversation extractable as spans. The paper applies this mechanism to DST and notes that predicted span boundaries are not inherently ordered.
3 Pointer Network
- The pointer network replaces prediction over predefined candidates with attention-normalized output positions in the input sequence.The encoder produces position-specific hidden states, and the decoder uses attention over them to select an input position at each step.Source: §3 · p. 3
- DST values are usually input word segments, so the decoder predicts their start and end indices rather than generating longer output sequences.This reformulates DST as a pointing task similar to machine comprehension.Source: §3 · p. 3
- Attention scores against the decoder state determine the starting position, and a second decoding step uses the selected start word's embedding to predict the ending position.The ending-position step does not guarantee that its index follows the starting position.Source: §3 · p. 4
- When the predicted end precedes the start, this often indicates that no answer exists in the input; a task-dependent constraint can enforce end-after-start ordering.The authors use this backoff interpretation for the none slot in DSTC2.Source: §3 · p. 4
- Explicitly mentioned unknown values can be extracted from dialogue text, giving DST an extractive perspective akin to slot tagging.The model points to the slot value in its surface form rather than requiring a candidate list.Source: §3 · p. 4
- In Figure 1, the classifier's “other” output routes the decision to the pointer network, whose decoder predicts the value's ending word using attention over encoded states.The decoder predicts the ending word given the predicted starting word.Source: Figure 1 · p. 4
4 Rephrasing and Non-pointable Values
The PtrNet copies expressed slot values from dialogue history, but surface forms may mismatch database-canonical values, and some values have no pointable span. A classifier handles non-pointable values alongside the pointer network.
4 Rephrasing and Non-pointable Values
- The PtrNet pinpoints slot values in their expressed surface forms, which may not match canonical backend values.An extra canonicalization step before database lookup can remedy the rephrasing mismatch.Source: §4 · p. 4
- Values such as none and dontcare are non-pointable because no specific dialogue span expresses them.The classifier chooses whether a slot takes a non-pointable value or should be processed by the PtrNet.Source: §4 · p. 5
- A multi-class classifier handles non-pointable values alongside the PtrNet.The classifier uses the dialogue encoder's final forward state and is trained jointly with the PtrNet using cross-entropy loss.Source: §4 · p. 5
- Copying mechanisms let dialogue-state tracking handle unknown slot values.The paper formulates DST to incorporate copying from the conversation history.Source: §4 · p. 5
5 Targeted Feature Dropout
Targeted feature dropout zeros embeddings for slot-value mentions during training, encouraging the model to learn contextual cues. It improves generalization to unknown values without sacrificing accuracy on known and observed values.
5 Targeted Feature Dropout
- Frequent or dictionary-covered slot values can leave contextual features undertrained, limiting generalization to unknown values.Targeted feature dropout was introduced to address feature co-adaptation, including insufficient training of word-context features in slot filling.Source: §5 · p. 5
- In DSTC2, most food types occur more than 10 times in training, encouraging memorization of frequent values over contextual patterns.The paper points to the food-type histogram as evidence of this frequency pattern.Source: §5 · p. 5
- During training, the model zeros slot-value word embeddings with some probability when the value is the labeled user goal.For example, it can zero the embedding for “italian” when that value is mentioned.Source: §5 · p. 6
- Targeted dropout improves generalization to unknown slot values without sacrificing accuracy on known and observed ones.Source: §5 · p. 6
6 Experiments and Results
The experiments evaluate unknown-value tracking on DSTC2 and bAbI, including a modified DSTC2 set with withheld food types. They test a PtrNet-based tracker and targeted dropout, comparing standard benchmark performance and unknown-value accuracy.
6.1 Datasets
- Experiments use DSTC2 and bAbI; the modified DSTC2 set treats 26 of 74 food types as unknown by removing their training instances.About 27% of training instances are discarded, while the test set remains unchanged.Source: §6.1 · p. 6, Table 2 · p. 6
6.2 Model and Training Details
- The classifier routes DSTC2 values into none, dontcare, or other, while the PtrNet handles pointable values; non-pointable values from ASR errors receive no special treatment.Training uses the last exact-match occurrence of a reference value when available; spelling variations “moderate”/“moderately” and “center”/“centre” are treated as equivalent.Source: §6.2 · p. 7
6.4 Results
- The PtrNet model achieved state-of-the-art results in the standard DSTC2 setting, where slot values are known and observed in training.Table 3 compares joint goal accuracy on the DSTC2 test set with previously reported approaches.Source: §6.3 · p. 7, §6.4 · pp. 7–8, Table 3 · p. 8
6.3 Evaluation Setup
- Unknown-value accuracy lacks established baselines, so the evaluation demonstrates the difficulty of the task and the effect of targeted dropout rather than comparing unknown-value systems.The regular DSTC2 comparison instead measures joint goal accuracy when values are known and observed during training.Source: §6.3 · p. 7
6.4 Results
- Unknown-food accuracy rose from 11.6% to 34.4% with 5% targeted dropout on modified DSTC2, while accuracy on known values remained roughly the same.The result is shown for the modified dataset with a reduced training set.Source: §6.4 · p. 8, Figure 3 · p. 8
- On bAbI, OOV API parameter values are harder to predict than regular values, and targeted dropout bridges the accuracy gap entirely.The dataset is described as artificial, with perfect accuracy often achievable on the regular test set.Source: §6.4 · p. 8, Table 4 · p. 8
7 Conclusion
The paper presents a pointer-network-based end-to-end tracker with a jointly trained classifier, targeting unknown slot values while achieving state-of-the-art accuracy on DSTC2.
7 Conclusion
- The hybrid tracker combines extractive pointer-network output with a jointly trained classification component.The pointer network extracts slot values, while the classifier handles non-pointable values.Source: §7 · p. 8
- The approach achieves state-of-the-art accuracy on DSTC2 while handling unknown slot values.Source: §7 · p. 8
- Feature dropout is reported as an effective technique.Source: §7 · p. 8