Source-linked AI summary
Neural Belief Tracker: Data-Driven Dialogue State Tracking
Nikola Mrkšić, Diarmuid Ó Séaghdha, Tsung-Hsien Wen, Blaise Thomson, Steve Young
TL;DR
Belief tracking must scale beyond approaches that require extensive SLU annotation or hand-crafted semantic lexicons. The paper proposes Neural Belief Tracking, which composes pre-trained word vectors with dialogue context; across two datasets, it matches lexicon-based models and outperforms them without those resources.
Problem
Existing belief trackers depend on substantial domain-specific SLU annotation or hand-crafted semantic dictionaries, limiting scalability to larger dialogue domains.
Method
NBT couples SLU and DST by composing pre-trained word vectors from user utterances, system output, and candidate slot-value pairs into representations for intent decisions.
Results
NBT matches delexicalisation-based models with hand-crafted semantic lexicons and significantly outperforms them when those resources are unavailable.
Takeaways & Limitations
NBT provides a belief-tracking framework that achieves state-of-the-art performance without hand-crafted semantic lexicons.
Takeaways & Limitations
The authors identify better ASR compensation as necessary for deployment in environments with challenging acoustics.
Abstract
from arXiv · showhide
One of the core components of modern spoken dialogue systems is the belief tracker, which estimates the user's goal at every step of the dialogue. However, most current approaches have difficulty scaling to larger, more complex dialogue domains. This is due to their dependency on either: a) Spoken Language Understanding models that require large amounts of annotated training data; or b) hand-crafted lexicons for capturing some of the linguistic variation in users' language. We propose a novel Neural Belief Tracking (NBT) framework which overcomes these problems by building on recent advances in representation learning. NBT models reason over pre-trained word vectors, learning to compose them into distributed representations of user utterances and dialogue context. Our evaluation on two datasets shows that this approach surpasses past limitations, matching the performance of state-of-the-art models which rely on hand-crafted semantic lexicons and outperforming them when such lexicons are not provided.
1 Introduction
Dialogue state tracking must interpret user goals from speech and dialogue context, but existing approaches depend on costly annotations or hand-crafted semantic dictionaries. The Neural Belief Tracker instead composes pre-trained word vectors and matches lexicon-based performance while outperforming models without those resources.
- Belief tracking maintains a probability distribution over dialogue states so the dialogue manager can choose the next system action.
- Dialogue state tracking must identify user constraints and requests from each utterance while considering the preceding dialogue context.
- Traditional approaches require either substantial domain-specific SLU annotation or manually constructed semantic dictionaries for lexical and morphological variation.
- NBT couples SLU and DST, composing pre-trained word vectors from the user utterance and preceding system output to classify ontology-defined intents.
- NBT models match delexicalisation-based models using hand-crafted semantic lexicons and significantly outperform them when those resources are unavailable.
2 Background
Prior belief trackers either separate SLU from DST or couple them through delexicalisation, whose semantic dictionaries create scalability constraints. NBT addresses this dependence by learning shared semantic representations and domain-specific variation from available data.
- Belief tracking models estimate user goals and requested entity attributes within an ontology-based evaluation setting.
- Separate SLU: Separate SLU pipelines detect slot-value pairs from ASR output before downstream DST combines them with dialogue context.
- Joint SLU/DST: Joint SLU/DST models generate belief states from ASR predictions, but commonly rely on delexicalisation and semantic dictionaries to identify slot-value mentions.
- Motivation: The NBT motivation is to overcome prior belief-tracking limitations without relying on hand-crafted resources.
- Motivation: NBT learns semantic information from pre-trained word vectors while sharing parameters across ontology values and learning domain-specific paraphrasings.
3 Neural Belief Tracker
The Neural Belief Tracker detects ontology-defined slot-value pairs by composing pre-trained word vectors for the utterance, dialogue context, and candidate pair. Its DNN and CNN variants model utterances differently, while semantic decoding and contextual gating support intent decisions.
- Neural Belief Tracker: NBT iterates over ontology-defined candidate slot-value pairs to decide which user goals were expressed.Its inputs include preceding system dialogue acts, the user utterance, and one candidate pair.
- Neural Belief Tracker: The model learns representations for the utterance, candidate pair, and system dialogue acts before semantic decoding and final intent classification.Intermediate interaction summaries are passed to a decision-making module.
- Representation Learning: NBT uses fixed, semantically specialised pre-trained word vectors so related unseen words can be recognised and parameters shared across ontology values.The paper uses Paragram-SL999 vectors and describes sharing across values or slots.
- Representation Learning: NBT-DNN sums unigram, bigram, and trigram word-vector representations, maps them through hidden layers, and sums the results into utterance representation r.Its cumulative n-grams are unweighted sums of the utterance’s word vectors.
- Representation Learning: NBT-CNN applies 300 filters to 1-, 2-, and 3-gram sequences, then uses ReLU and max-pooling before summing summary representations into r.Direct interaction with utterance subsequences makes this variant better suited to longer utterances than NBT-DNN.
- Semantic Decoding and Context Modelling: Semantic decoding compares utterance and candidate representations, while context gating incorporates system requests and confirmations into the binary decision.For confirmations, the model uses a three-way interaction among the utterance, candidate pair, and system-offered slot-value pair.
4 Belief State Update Mechanism
The NBT updates belief states from ASR hypotheses by combining turn-level estimates with accumulated dialogue history, weighting current and prior evidence.
- Turn-level estimation: The model combines predictions across the N ASR hypotheses to obtain a turn-level belief state estimate.
- Turn-level estimation: NBT models estimate the probability that each slot-value pair was expressed in each ASR hypothesis and preceding system output.Predictions are made separately for each hypothesis, slot, and slot value.
- Belief-state update: The turn-level estimate is combined with the cumulative belief state from earlier turns to produce the updated belief state.
- Belief-state update: λ controls the relative weight assigned to the current-turn and previous-turn belief-state estimates.For each slot, detected values at the current turn are then determined from the resulting estimate.
- Belief-state interpretation: For informable slots, the highest-probability value is selected as the current goal, while requestable slots require no cross-turn belief tracking.
5 Experiments
Experiments evaluate NBT on two Cambridge restaurant-dialogue datasets using joint goals and request accuracy, comparing neural variants with delexicalisation-based baselines and dictionary-augmented systems.
- Datasets: The DSTC2 and WOZ 2.0 corpora share a restaurant-search ontology with FOOD, AREA, and PRICE informable slots.Users can also request information such as phone number and address.
- Datasets: DSTC2 contains 2,207 training and 1,117 test dialogues, while WOZ 2.0 is divided into 600 training, 200 validation, and 400 test dialogues.DSTC2 includes transcriptions, ASR hypotheses, and turn-level semantic labels; WOZ users typed instead of speaking.
- Evaluation: The evaluation measures joint goal accuracy and turn-level request accuracy.Joint goal accuracy requires all user search constraints to be identified correctly on a dialogue turn.
- NBT models: NBT-DNN and NBT-CNN are trained with Adam and cross-entropy loss while keeping pre-trained word vectors fixed.Models are trained separately for each slot, with positive examples included in each mini-batch to address class imbalance.
- Baselines: The comparisons include dataset-specific delexicalisation baselines, versions supplemented with task-specific semantic dictionaries, and alternative delexicalised neural architectures.The DSTC2 dictionary contains only three rephrasings, yet these produce substantial DST gains.
6 Results
NBT models outperform baselines on joint goals and requests while matching lexicon-supplemented systems, and their performance depends on the semantic quality of word vectors and dataset conditions.
- Belief tracking performance: NBT models outperform baseline systems on both joint goal and request accuracies across DSTC2 and WOZ 2.0.Goal gains are always statistically significant with paired t-tests at p < 0.05.
- Belief tracking performance: NBT performance does not differ significantly from lexicon-supplemented models, indicating that it handles relations otherwise encoded in semantic dictionaries.
- Dataset conditions: The improvement over baseline is greater on WOZ 2.0, whose longer sentences and richer vocabulary make linguistic variation more important.DSTC2 is less linguistically rich, with ASR errors presenting the main challenge.
- Dataset conditions: 0.96 goal accuracy is achieved on DSTC2 when the NBT models receive test-set transcriptions instead of ASR output.The authors connect this result to the need for better ASR compensation in acoustically challenging deployment environments.
- Word vector spaces: Paragram-SL999 vectors significantly outperform GloVe and XAVIER vectors for goal tracking on both datasets.Pre-trained vectors are especially beneficial on noisy DSTC2, while GloVe does not improve over random initialization on clean WOZ 2.0.
- Word vector spaces: NBT-DNN shows the same word-vector trends as NBT-CNN, although Table 2 reports only NBT-CNN figures.
7 Conclusion
The paper presents NBT as a lexicon-free framework for coupled SLU and DST that achieves strong belief-tracking performance and benefits from semantically specialized word vectors.
- Conclusion: NBT couples SLU and DST without hand-crafted semantic lexicons while matching lexicon-based models and outperforming them when lexicons are unavailable.
- Conclusion: NBT performance improves with the semantic quality of its underlying word vectors.The authors describe this as evidence that semantic specialization improves downstream task performance.
- Future work: Future work will examine NBT for multi-domain dialogue and languages requiring complex morphological variation.