Source-linked AI summary
Deep Unknown Intent Detection with Margin Loss
Ting-En Lin, Hua Xu
TL;DR
Unknown intents are difficult to detect because they lack training examples and may overlap semantically with known intents. The paper combines BiLSTM feature extraction with LMCL and LOF-based novelty detection, yielding consistent improvements over baselines on two benchmark datasets. These results support using discriminative feature learning with novelty detection for unknown-intent detection.
Problem
Detecting user intents absent from the training set is challenging because unknown intents lack examples and may be semantically similar to known intents.
Method
The method trains a BiLSTM feature extractor with LMCL, which promotes inter-class separation and intra-class compactness, then applies LOF to detect unknown intents.
Results
Consistent improvements over baseline methods are reported on two benchmark datasets, including macro f1-score gains over DOC on SNIPS of 6.7%, 16.2%, and 14.9% in the 25%, 50%, and 75% settings.
Takeaways & Limitations
The results support combining margin-based discriminative features with density-based novelty detection for unknown-intent detection.
Takeaways & Limitations
On ATIS, unknown-intent detection performance drops sharply as the number of known intents increases because semantically similar intents overlap.
Abstract
from arXiv · showhide
Identifying the unknown (novel) user intents that have never appeared in the training set is a challenging task in the dialogue system. In this paper, we present a two-stage method for detecting unknown intents. We use bidirectional long short-term memory (BiLSTM) network with the margin loss as the feature extractor. With margin loss, we can learn discriminative deep features by forcing the network to maximize inter-class variance and to minimize intra-class variance. Then, we feed the feature vectors to the density-based novelty detection algorithm, local outlier factor (LOF), to detect unknown intents. Experiments on two benchmark datasets show that our method can yield consistent improvements compared with the baseline methods.
1 Introduction
Unknown-intent detection matters for discovering intents absent from training data, but limited examples, unknown class counts, and semantic overlap make it difficult. The paper addresses these challenges by learning discriminative BiLSTM features with margin loss and applying LOF for detection.
- Unknown-intent detection can reveal potential business opportunities, guide developers, and accelerate dialogue-system development.
- The task is difficult because unknown intents lack examples, their exact number is hard to estimate, and intent semantics depend on context.
- Existing approaches require out-of-domain samples, struggle to generate useful text examples, or underuse known-intent information during clustering.
- The method leverages known-intent labels by treating unknown intents as a single additional class in an open-world classification problem.
- The proposed system replaces softmax loss with LMCL to learn compact, separated BiLSTM features, then applies LOF to detect unknown intents.
- Experiments on two benchmark dialogue datasets support the effectiveness of the proposed two-stage method.
2 Proposed Method
The proposed method uses BiLSTM to extract sentence representations, LMCL to make intent features more discriminative, and LOF to detect unknown intents from local density.
- 2.1 BiLSTM: BiLSTM converts word embeddings into forward and backward sequence representations, then concatenates boundary outputs into a sentence representation for the next stage.The resulting representation captures high-level semantic concepts learned by the model.
- 2.2 Large Margin Cosine Loss (LMCL): LMCL replaces BiLSTM’s softmax loss and applies normalized cosine classification with an angular decision margin.The formulation uses a scaling factor, cosine margin, class weights, and angles between class weights and feature vectors.
- Experiments: The experiments use the publicly available SNIPS and ATIS benchmark dialogue datasets.Table 1 reports dataset statistics, with # denoting the total number of utterances.
- 2.2 Large Margin Cosine Loss (LMCL): LMCL forces the model to maximize inter-class variance and minimize intra-class variance, producing discriminative intent representations.These representations are used as feature vectors for novelty detection.
- 2.3 Local Outlier Factor (LOF): LOF detects unknown intents by evaluating whether an example’s local density is substantially lower than the density of its k-nearest neighbors.The method computes local reachability density from reachability distances and neighbor relationships.
3 Experiments
Experiments evaluate unknown-intent detection on SNIPS and ATIS under varying proportions of known classes, comparing the proposed method with baselines and a softmax ablation. The method consistently improves results, while performance declines on semantically overlapping ATIS intents as known-intent coverage increases.
- 3.2 Baselines: The comparison includes MSP, DOC, DOC (Softmax), and LOF (Softmax), alongside the proposed method.LOF (Softmax) is the ablation that replaces LMCL with softmax loss for feature extraction.
- 3.1 Datasets: Experiments use SNIPS and ATIS, treating 25%, 50%, or 75% of classes as known during training and evaluating with all classes.SNIPS spans seven intent types across domains, while ATIS contains 18 intent types in the flight domain.
- 3.4 Results and Discussion: 6.7%, 16.2% and 14.9% macro f1-score improvements over DOC occur on SNIPS at the 25%, 50%, and 75% settings, respectively.The paper reports these gains as consistent improvements over all baselines in all settings.
- 3.4 Results and Discussion: LMCL features are intra-class compact and inter-class separable, benefiting novelty detection algorithms based on local density.Figure 2 visualizes deep features learned with softmax and LMCL on SNIPS.
- 3.4 Results and Discussion: On ATIS, unknown-intent detection performance drops dramatically as known intents increase because unknown and known intents can overlap semantically.The paper attributes this difficulty to the shared flight domain and similar intents such as flight and flight no.
- 3.4 Results and Discussion: The approach improves more on SNIPS than on ATIS, where intents originate from different domains and are less semantically similar.The paper connects this setting with more robust feature extraction from margin loss.
4 Conclusion
The paper proposes a two-stage unknown-intent detection method and reports consistent improvements over baseline methods on two benchmark datasets. It also identifies end-to-end unknown-intent identification and clustering as future work.
- 4 Conclusion: The proposed two-stage method trains a BiLSTM feature extractor with margin loss, then detects unknown intents using novelty detection.Margin loss encourages inter-class variance maximization and intra-class variance minimization.
- 4 Conclusion: Experiments on two benchmark datasets show consistent improvements compared with baseline methods.
- 4 Conclusion: Future work will identify unknown intents from known intents and cluster unknown intents end to end.