Source-linked AI summary
Zero-shot User Intent Detection via Capsule Neural Networks
Congying Xia, Chenwei Zhang, Xiaohui Yan, Yi Chang, Philip S. Yu
TL;DR
The paper addresses intent detection when emerging intents lack labeled utterances and diverse expressions challenge direct intent mapping. It proposes capsule-based architectures that extract and aggregate semantic features, then transfer knowledge from existing intents; experiments on two real-world datasets report effective detection of existing and emerging intents.
Problem
Existing intent detectors require labeled utterances for predefined intents, while emerging intents lack labels and diverse expressions complicate zero-shot mapping.
Method
INTENTCAPSNET hierarchically extracts semantic features and aggregates them with dynamic routing, while INTENTCAPSNET-ZSL transfers existing-intent knowledge to emerging intents.
Results
Experiments on two real-world datasets show effectiveness for discriminating existing intents and emerging intents without labeled utterances.
Takeaways & Limitations
Capsule-based hierarchical representations provide a model for intent detection that accommodates diverse expressions and zero-shot emerging-intent discrimination.
Takeaways & Limitations
The model sometimes confuses emerging intents with inherently similar display-configuration intents.
Abstract
from arXiv · showhide
User intent detection plays a critical role in question-answering and dialog systems. Most previous works treat intent detection as a classification problem where utterances are labeled with predefined intents. However, it is labor-intensive and time-consuming to label users' utterances as intents are diversely expressed and novel intents will continually be involved. Instead, we study the zero-shot intent detection problem, which aims to detect emerging user intents where no labeled utterances are currently available. We propose two capsule-based architectures: INTENT-CAPSNET that extracts semantic features from utterances and aggregates them to discriminate existing intents, and INTENTCAPSNET-ZSL which gives INTENTCAPSNET the zero-shot learning ability to discriminate emerging intents via knowledge transfer from existing intents. Experiments on two real-world datasets show that our model not only can better discriminate diversely expressed existing intents, but is also able to discriminate emerging intents when no labeled utterances are available.
1 Introduction
Existing intent detection relies on labeled utterances and struggles with diverse expressions and emerging intents without annotations. The paper proposes capsule-based models that hierarchically represent semantics and transfer knowledge from existing intents for zero-shot detection.
- Existing supervised classifiers handle predefined intents but cannot detect emerging intents without labeled utterances.Annotating emerging intents and retraining the detector are labor-intensive and time-consuming.
- Zero-shot intent detection expands the intent space from labeled existing intents to emerging intents with no labeled utterances.
- Diverse semantic expressions can impede methods that learn a direct utterance-to-intent mapping for zero-shot detection.
- INTENTCAPSNET-ZSL constructs zero-shot DetectionCaps using extracted semantic features, existing routing agreements, and similarities between emerging and existing intent labels.
- INTENTCAPSNET extracts interpretable semantic features and aggregates them hierarchically through dynamic routing-by-agreement for intent detection.Semantic features such as actions, times, and city names collectively contribute to higher-level intents.
- INTENTCAPSNET performs consistently better than TF-IDF bag-of-word classifiers and neural text-classification models on two real-world datasets.
2 Problem Formulation
The paper distinguishes ordinary intent detection among labeled existing classes from zero-shot detection of disjoint emerging classes without labeled examples.
- An intent is a purpose or goal underlying a user-generated utterance, and this work considers the single-intent case.Utterances with multiple intents are left for future work after segmentation into single-intent snippets.
- Intent detection assigns an utterance to one of K existing intent classes using labeled training examples.
- Zero-shot intent detection assigns an unlabeled emerging utterance to one of L emerging intents disjoint from the existing intent set.
3 Approach
The proposed capsule architectures extract multiple semantic features, route them into intent representations, and transfer existing-intent knowledge to detect emerging intents without labeled utterances.
- Architecture: INTENTCAPSNET uses SemanticCaps to extract semantic features, DetectionCaps to aggregate them, and activation vectors to discriminate existing intents.The architecture contains SemanticCaps, DetectionCaps, and Zero-shot DetectionCaps for extending detection to emerging intents.
- SemanticCaps: A bidirectional LSTM encodes each utterance into word-level hidden states that provide inputs to multi-head self-attention.Each word’s forward and backward hidden states are concatenated into the hidden-state matrix H.
- SemanticCaps: Each self-attention head focuses on a distinct semantic feature, producing R vectorized semantic representations from the utterance.The model encourages heads to attend to different features, such as an action or city name in GetWeather.
- DetectionCaps: DetectionCaps transform semantic features into intent-specific prediction vectors and combine them through dynamic routing-by-agreement.Coupling coefficients represent how informative each semantic feature is for an intent and are updated iteratively during routing.
- DetectionCaps: The existing-intent objective combines max-margin classification with a regularizer encouraging different attention heads to capture different semantic features.The loss uses positive and negative margins and an attention-diversity term weighted by α.
- Zero-shot DetectionCaps: INTENTCAPSNET-ZSL transfers existing-intent vote vectors to emerging intents using semantic-extraction behavior, routing information, and intent-label similarities.The transferred vectors are used in zero-shot dynamic routing to construct activation vectors for emerging intents.
4 Experiment Setup
The experiments evaluate capsule-based intent detection on SNIPS-NLU and CVA, comparing INTENTCAPSNET and INTENTCAPSNET-ZSL with classification and zero-shot baselines.
- Experimental Tasks: INTENTCAPSNET is evaluated on existing-intent detection, while INTENTCAPSNET-ZSL is evaluated on emerging-intent detection.The experiments use the two proposed architectures for their respective intent-detection settings.
- Evaluation Measures: Tables 1 and 4 report Accuracy, Precision, Recall, and F1 as support-weighted averages across classes.Table 1 covers existing-intent detection, whereas Table 4 covers zero-shot intent detection.
- Datasets: The evaluation uses SNIPS-NLU, an English crowdsourced corpus, and CVA, an anonymously collected Chinese voice-assistant corpus.Both datasets are used across the intent-detection tasks.
- Baselines: Existing-intent detection compares INTENTCAPSNET with TF-IDF classifiers, CNN, recurrent neural networks, and self-attention BiLSTM.The recurrent baselines include RNN, GRU, LSTM, and BiLSTM variants.
- Baselines: Zero-shot detection compares INTENTCAPSNET-ZSL with DeViSE and CMT compatibility-based strategies.The baselines learn compatibility functions between utterances and emerging intent labels.
- Implementation: Hyperparameters are selected with three-fold cross-validation, using averaged word embeddings for intent representations and dynamic routing for three iterations.Adam optimization is used, with λ = 0.5 and margins m+ = 0.9 and m- = 0.1.
5 Results
INTENTCAPSNET outperforms several text-classification baselines for existing intents, and INTENTCAPSNET-ZSL outperforms alternative zero-shot strategies for emerging intents. The learned representations generally separate intents while revealing confusion among semantically similar display-related intents.
- Existing-Intent Detection: INTENTCAPSNET consistently outperforms TF-IDF classifiers and neural text-classification models on two existing-intent detection datasets.The reported metrics are presented in Table 1.
- Zero-Shot Detection: INTENTCAPSNET-ZSL outperforms baselines using different zero-shot learning strategies on the zero-shot intent-detection task.CMT has higher precision but lower accuracy and recall on SNIPS-NLU, while CDSSM fails on CVA.
- Ablation Study: On SNIPS-NLU, removing self-attention, Bi-LSTM, or the regularizer each reduces F1 by approximately 2–3%.The ablation results indicate comparable contributions from the three modules on SNIPS-NLU.
- Ablation Study: On CVA, self-attention provides a 5.2% improvement in F1, making it the most important ablated module on that dataset.The comparison is reported in the lower part of Table 4.
- Emerging-Intent Representations: t-SNE visualization shows that utterances with different emerging intents tend to have different activation-vector orientations, while same-intent utterances remain proximate despite expression nuances.The visualization uses normalized activation vectors for 20 emerging CVA intents, colored by ground-truth labels.
- Emerging-Intent Representations: The model confuses DecreaseScreenBrightness, ReduceFontSize, and SetColdColor, whose representations cluster near (15, -5).The paper attributes these errors to the inherent similarity of the three display-configuration intents.
6 Interpretability
The interpretability analysis shows that SemanticCaps extract distinct, generalizable semantic features and transfer extraction behavior across existing and emerging intents. Similarity variance also helps explain when knowledge transfer supports zero-shot detection accuracy.
- Extracting Meaningful Semantic Features: Self-attention heads usually focus on distinct semantic features, such as actions, musician names, book names, or rating scores.On SNIPS-NLU, heads adapt to diverse expressions, including untagged musician names and multiple formulations of search actions.
- Extraction-behavior Transfer by SemanticCaps: SemanticCaps transfer extraction behavior from existing PlayMusic utterances to the emerging RateABook intent.A head attending to “play” in PlayMusic also attends to rating expressions such as rate, add the rating, and give.
- Extraction-behavior Transfer by SemanticCaps: The observed behavior indicates an intent-independent extractor that provides generalizable semantic features for existing and emerging intent representations.The model also transfers extraction behaviors among utterances belonging to different intents.
- Knowledge Transfer via Intent Similarity: 13 of 20 emerging CVA intents with var(q_l) > 0.005 achieved Accuracy≥0.83.Higher variance indicates that some existing intents are more similar than others, giving the model clearer sources for similarity-based knowledge transfer.
- Knowledge Transfer via Intent Similarity: Low similarity variance does not always reduce performance because some emerging intents can transfer knowledge more evenly from several existing intents.This transfer may occur with less confidence from any individual existing intent.
7 Conclusions
The paper introduces capsule-based architectures for intent detection and zero-shot recognition of emerging intents. INTENTCAPSNET extracts and aggregates semantic features, while INTENTCAPSNET-ZSL transfers knowledge from existing intents; experiments on two real-world datasets show effectiveness and interpretability.
- Conclusions: INTENTCAPSNET extracts semantic features with self-attention and aggregates them through dynamic routing-by-agreement into utterance-level intent representations.The architecture applies capsule-based hierarchical learning to text modeling.
- Conclusions: INTENTCAPSNET-ZSL adds zero-shot learning through knowledge transfer from existing intents to discriminate emerging intents without labeled utterances or excessive external resources.The model extends the capsule architecture to emerging intents lacking direct training examples.
- Conclusions: Experiments on two real-world datasets show the effectiveness and interpretability of the proposed models.The conclusion reports both performance and interpretability outcomes for the proposed architectures.