Source-linked AI summary
Recent Neural Methods on Slot Filling and Intent Classification for Task-Oriented Dialogue Systems: A Survey
Samuel Louvan, Bernardo Magnini
TL;DR
Task-oriented dialogue systems need to identify users’ intents and the slot values that specify their requests, yet adapting models to new domains and languages with limited labeled data remains challenging. This survey reviews independent, joint, and transfer-learning neural approaches to slot filling and intent classification. It concludes that joint models perform relatively better than independent models, while widely used datasets can be nearly solved with sufficient in-domain data.
Problem
Models designed for a single domain require relatively large labeled datasets, while adding new intents and slots makes domain expansion data- and time-intensive; portability to new domains and languages with limited labels remains challenging.
Source: §5 · p. 8, §6 · p. 10Method
The survey reviews independent, joint, and transfer-learning neural models for slot filling and intent classification and compares their performance.
Source: §1 · p. 1, §7 · p. 10Results
Joint models show relatively better performance than independent models, and most nearly solve ATIS and SNIPS when sufficient in-domain training data are available.
Source: §7 · p. 10Takeaways & Limitations
Improving model scalability to new domains and languages when labeled data are limited remains a challenge for slot filling and intent classification.
Source: §7 · p. 10Most models are evaluated in English, and more work is needed on transfer learning with limited or no labeled data for target domains.
Source: §6 · p. 10
Abstract
from arXiv · showhide
In recent years, fostered by deep learning technologies and by the high demand for conversational AI, various approaches have been proposed that address the capacity to elicit and understand user's needs in task-oriented dialogue systems. We focus on two core tasks, slot filling (SF) and intent classification (IC), and survey how neural-based models have rapidly evolved to address natural language understanding in dialogue systems. We introduce three neural architectures: independent model, which model SF and IC separately, joint models, which exploit the mutual benefit of the two tasks simultaneously, and transfer learning models, that scale the model to new domains. We discuss the current state of the research in SF and IC and highlight challenges that still require attention.
1 Introduction
Understanding a user’s intent and the values they mention is central to task-oriented dialogue. This survey maps recent neural approaches to slot filling and intent classification, compares their performance, and discusses open issues.
1 Introduction
- Natural language understanding represents a user’s needs through intent classification, which identifies the frame, and slot filling, which extracts its values.For “I want to listen to Hey Jude by The Beatles,” the intent is to play a song and the slots identify its title and artist.Source: §1 · p. 1
- Neural approaches to slot filling and intent classification have progressed from RNNs and attention to transformers and contextualized word embeddings.The survey notes that these developments address dialogue phenomena such as context, out-of-vocabulary words, and long-distance dependencies.Source: §1 · p. 1
- The survey aims to map recent neural models for both tasks, compare their performance, and highlight open issues.It organizes the approaches as independent, joint, and transfer-learning models.Source: §1 · pp. 1–2
2 Slot Filling and Intent Classification
The survey focuses on natural-language understanding in task-oriented dialogue, where a semantic frame represents a user’s intent and associated slot values. It defines slot filling and intent classification, sets the survey’s scope, and reviews datasets and evaluation metrics.
2.1 Background
- Task-oriented dialogue systems help users accomplish tasks through natural-language conversation, with an NLU component interpreting their needs within a broader processing pipeline.The described pipeline also includes speech recognition, dialogue state tracking, dialogue policy, and response generation.Source: §2.1 · p. 2
- The NLU output is treated as a partially filled semantic frame containing an intent and slot-value pairs; intent identifies a task or action, while slots represent relevant domain information.Users may supply slot values in one turn or across several turns.Source: §2.1 · p. 2
2.2 Scope of the Survey
- The survey focuses on slot filling as sequence tagging rather than dialogue-state-tracking classification, which requires slot-value pairs to be predefined in an ontology.The survey notes that this ontology requirement limits the applicability of the latter approach.Source: §2.2 · pp. 2–3
2.3 Task Definition
- Slot filling assigns a label to each utterance token, while intent classification assigns one intent label to the whole utterance.The survey describes models that estimate the joint probability of the intent and slot labels given the utterance.Source: §2.3 · p. 3
2.4 Datasets for SF and IC
- The reviewed datasets pair utterances with intent labels and slot annotations; ATIS and SNIPS are prominent benchmarks, while MEDIA and MIT cover other settings.ATIS contains about 5K airline-domain utterances, has a skewed intent distribution, and is dominated by location slots; MEDIA is French and has slot labels only.Source: §2.4 · p. 3
2.5 Evaluation Metrics
- Intent classification is evaluated by utterance-level accuracy, while slot filling uses exact-match entity-level F1, which requires the correct slot label and token boundaries.Partial matches do not count as correct slot predictions.Source: §2.5 · p. 4
3 Independent Models for SF and IC
Independent models train slot filling and intent classification separately, using neural encoders and task-specific outputs. The survey describes their architectural choices and reports that context and bidirectional encoding can improve performance, while separate training misses interaction between the tasks.
3 Independent Models for SF and IC
- Independent models typically use RNN encoders, predicting slot labels from each hidden state and intent from the final hidden state, with separate models trained for each task.The most common evaluation dataset for these models is ATIS.Source: §3 · p. 4
- Reported comparisons favor LSTM over Elman and Jordan RNNs for slot filling, and find LSTM better for longer utterances while vanilla RNN works best for shorter ones.A bidirectional Jordan RNN achieved 93.89 F1 for slot filling on ATIS, about +1 absolute F1 over CRF; CNN performed worse than RNN for slot filling on ATIS.Source: §3 · p. 4
- R-CRF combines RNN-learned features with CRF transition scores and outperforms CRF and vanilla RNN on ATIS and the Bing query understanding dataset.The survey identifies softmax as a typical output-layer prediction function.Source: §3 · p. 4
- Independent models do not exploit interaction between intent and slots and may introduce error propagation when used in a pipeline.The survey identifies this as their main disadvantage.Source: §3 · p. 5
4 Joint Models for SF and IC
Joint models address the relationship between slot filling and intent classification through two-way information sharing. The survey describes parameter and state sharing alongside explicit slot-intent gating.
4.1 Parameter and State Sharing
- Shared encoders and task representations support joint prediction, while other designs exchange hidden states asynchronously or use token-level intent predictions to guide slot filling.Examples include shared Bi-GRU, LSTM, and self-attentive encoders, as well as BERT hidden states used for slot and intent labels.Source: §4.1 · p. 6
4.2 Slot-Intent Gate Mechanism
- The slot-intent gate uses slot and intent context to weight token representations, explicitly modeling their correlation for slot prediction.Related variants add an intent gate and iterative exchange between slot-filling and intent-detection networks, or use self-attention to compute slot context.Source: §4.2 · pp. 6–7
- +0.5% SF improvement over Liu and Lane (2016) is reported for the slot-gate mechanism; Zhang et al. (2019) add gating to BERT.The section also reports that SF-ID performs better than Goo et al. (2018) on both SF and IC on ATIS and SNIPS.Source: §4.2 · p. 7
- Joint models are competitive with independent models; hybrid parameter/state sharing plus intent gating yields the best performance.The survey also identifies fine-tuning a pre-trained model such as BERT as a route to maximum SF and IC performance when computational power is available.Source: §4.2 · p. 7
5 Scaling to New Domains
The domain scaling problem arises because existing models are designed for a single domain and need substantial labeled data, while new domains require methods that work with limited or no labels. The survey reviews transfer-learning strategies for that setting.
5.1 Transfer Learning Models for SF and IC
- Data-driven transfer combines source and target data or pre-trains on source domains before fine-tuning on the target, while model-driven adaptation reuses expert models for common slots.Model-driven target training time scales with target-domain data size rather than the combined source and target data size.Source: §5.1 · pp. 8–9
5.2 Zero-shot Models for SF and IC
- Zero-shot models use natural-language slot descriptions to recognize semantically similar, unseen labels, and some approaches combine descriptions with slot-value examples.Per-slot prediction can create overlapping predictions; a coarse-to-fine method first detects entity tokens and then predicts a slot type using similarity to its description.Source: §5.2 · p. 9
5.1 Transfer Learning Models for SF and IC
- With limited target-domain data, MTL and pre-train fine-tuning improve performance; for many domains, fine-tuning or model-driven approaches are preferable for training time.Fine-tuning is more prone to forgetting than MTL; zero-shot methods suit target domains without annotations but require slot or intent meta-information.Source: §5.2 · p. 9
6 State of the Art and Beyond
Neural models perform strongly on ATIS and SNIPS, but this does not mean slot filling and intent classification are solved. The survey highlights data efficiency, portability, live data, generative modeling, and more complex evaluation as open challenges.
State of the Art and Beyond
- Neural models perform strongly on ATIS and SNIPS, but these datasets do not establish that slot filling and intent classification are solved.ATIS is described as relatively simple and close to saturation, with a similar trend noted for SNIPS.Source: §6 · p. 9
Portable and Data Efficient Models
- Models need stronger transfer performance with limited or no target-domain labels and across languages, where evaluation remains heavily English-focused.Designing data-efficient models portable across domains and languages remains challenging.Source: §6 · pp. 9–10
Leveraging unlabeled data from live traffic
- Unlabeled live traffic could supplement in-house annotated data, but its diverse, noisy, and irrelevant utterances challenge existing semi-supervised and data-selection methods.Academic access to live data is itself a bottleneck.Source: §6 · p. 10
Generative Models
- Generative models have shown an advantage over discriminative models when data is scarce, while synthetic data augmentation must preserve semantics and produce diverse, fluent utterances.The cited works explore augmentation as a potential use of generative models.Source: §6 · p. 10
Evaluation of SF and IC on more complex dataset
- Evaluation should extend beyond single-intent utterances to multiple intents, multi-turn dialogue, and hierarchical representations.The passage identifies multi-turn datasets and datasets with hierarchical annotations as available examples.Source: §6 · p. 10
7 Conclusion
The survey reviews independent, joint, and transfer-learning models for slot filling and intent classification. Joint models perform relatively better than independent models, while scalability with limited labeled data remains a challenge.
Conclusion
- The survey examines independent, joint, and transfer-learning models for slot filling and intent classification in task-oriented dialogue systems.Source: §7 · p. 10
- Joint models that exploit the relationship between slot filling and intent classification perform relatively better than independent models.Source: §7 · p. 10
- Most joint models nearly solve ATIS and SNIPS with sufficient in-domain training data, but scalability to new domains and languages with limited labels remains challenging.Source: §7 · p. 10
Appendix A. SF and IC Dataset
Table 4 presents statistics for single-turn datasets.
Appendix A. SF and IC Dataset
- Table 4 presents statistics for single-turn datasets.Source: Table 4 · p. 17