Source-linked AI summary
Joint Slot Filling and Intent Detection via Capsule Neural Networks
Chenwei Zhang, Yaliang Li, Nan Du, Wei Fan, Philip S. Yu
TL;DR
The paper addresses the gap between word-level slot filling and utterance-level intent detection in models that do not explicitly preserve their hierarchy. It introduces CAPSULE-NLU, which uses dynamic routing among word, slot, and intent capsules plus intent-informed re-routing, and reports effectiveness on two real-world datasets against alternative models and commercial NLU services.
Problem
Existing systems either separate slot filling and intent detection or jointly model them without explicitly preserving the hierarchy among words, slots, and intents.
Method
CAPSULE-NLU uses a hierarchical capsule network with dynamic routing for slot filling and intent detection, then re-routes using the inferred intent representation.
Results
The model performs consistently better than current joint-learning schemes and intent-focused capsule models on intent detection across two datasets.
Takeaways & Limitations
The model explicitly harnesses relationships among words, slots, and intents for joint slot filling and intent detection.
Takeaways & Limitations
Existing joint approaches risk losing context when longer sequences are compressed into a single recurrent vector.
Abstract
from arXiv · showhide
Being able to recognize words as slots and detect the intent of an utterance has been a keen issue in natural language understanding. The existing works either treat slot filling and intent detection separately in a pipeline manner, or adopt joint models which sequentially label slots while summarizing the utterance-level intent without explicitly preserving the hierarchical relationship among words, slots, and intents. To exploit the semantic hierarchy for effective modeling, we propose a capsule-based neural network model which accomplishes slot filling and intent detection via a dynamic routing-by-agreement schema. A re-routing schema is proposed to further synergize the slot filling performance using the inferred intent representation. Experiments on two real-world datasets show the effectiveness of our model when compared with other alternative model architectures, as well as existing natural language understanding services.
1 Introduction
Slot filling and intent detection are complementary NLU tasks, but existing joint models do not explicitly preserve the hierarchy among words, slots, and intents. CAPSULE-NLU addresses this gap with capsule routing and re-routing that connect word-level slots to utterance-level intent.
- Motivation: Slot filling labels words with slot types, while intent detection assigns an intent to the utterance.The paper illustrates artist, playlist owner, and playlist name slots alongside an AddToPlaylist intent.
- Motivation: Recognized slots can provide clues about the utterance-level intent, creating a potential synergistic relationship between the tasks.For example, recognizing Sungmin as an artist makes AddToPlaylist more likely than unrelated intents.
- Proposed Approach: Dynamic routing assigns words to appropriate slot capsules and aggregates slot representations to infer the utterance intent.The routing weights encode how strongly each word belongs to a slot type.
- Limitations of Existing Models: Existing joint approaches sequentially label words and summarize intent without explicitly modeling the hierarchy among words, slots, and intents.The paper notes that compressing longer sequences into a single recurrent representation can be risky.
- Proposed Approach: CAPSULE-NLU represents the word, slot, and intent hierarchy with a hierarchical capsule neural network.Capsules route low-level features to higher-level capsules when their predictions agree.
- Proposed Approach: Re-routing uses the inferred intent representation to refine word-level slot filling.The authors propose that intent information can help recover slots otherwise neglected during initial routing.
2 Approach
CAPSULE-NLU models words, slots, and utterance intents as a hierarchy of capsules, using dynamic routing to assign words to slots and slots to intents. Intent representations are also fed back through re-routing to update slot representations and routing information.
- Architecture: CAPSULE-NLU uses WordCaps, SlotCaps, and IntentCaps to represent context-aware words, slot types, and utterance-level intents hierarchically.SlotCaps aggregate words for each slot type, while IntentCaps determine the utterance intent from slot representations.
- Slot Filling: Dynamic routing replaces sequential slot labeling by routing each word toward slot capsules according to learned agreement values.Prediction vectors are iteratively aggregated into slot representations, and agreement logits are updated from prediction–representation similarity.
- Slot Filling: The routing algorithm initializes word–slot logits, repeatedly aggregates prediction vectors, squashes the sums into slot representations, and updates logits by agreement.The resulting agreement values determine how WordCaps outputs are sent to subsequent SlotCaps.
- Intent Detection: IntentCaps consume SlotCaps outputs and apply the same dynamic routing-by-agreement mechanism to determine the utterance-level intent.Each slot representation is transformed with respect to each intent type before routing.
- Intent Detection: Intent activation-vector length determines the predicted intent, while vector orientation represents discriminative intent properties.The predicted intent is selected by the activation vector with the largest norm.
- Re-Routing: Re-routing incorporates the most-likely intent activation vector into SlotCaps updates, combining slot agreement with inferred utterance intent.This updates routing information and slot representations, allowing intent information to strengthen relevant slots during slot filling.
3 Experiment Setup
The experiments evaluate CAPSULE-NLU on SNIPS-NLU and ATIS, comparing it with alternative architectures and commercial NLU services. Dataset statistics, hyperparameter settings, and slot-filling and intent-detection results are reported across the evaluation.
- Datasets: CAPSULE-NLU is evaluated on the SNIPS-NLU and ATIS datasets.SNIPS-NLU benchmarks voice assistants, while ATIS contains flight-reservation utterances.
- Baselines: The evaluation compares CAPSULE-NLU with alternative joint slot-filling and intent-detection models and commercial NLU services.Commercial comparisons include api.ai, Watson Assistant, Luis, wit.ai, snips.ai, recast.ai, and Amazon Lex.
- Experimental materials: The experiments report dataset statistics, hyperparameter settings, and CAPSULE-NLU results for slot filling and intent detection.These materials are presented in Tables 1–3.
- Evaluation protocol: Stratified 5-fold cross validation is used for benchmarking with existing NLU services on SNIPS-NLU.The figure includes black bars indicating standard deviation.
- Implementation details: Validation data is used for hyperparameter selection, with Xavier-initialized trainable word embeddings and dropout applied to the LSTM layer.The dropout rate is 0.2; RMSProp is used for optimization.
4 Results
CAPSULE-NLU is evaluated against alternative joint models, natural language understanding services, and visualized routing behavior on SNIPS-NLU. The results show competitive performance and progressively more confident, context-sensitive routing across capsule layers.
- CAPSULE-NLU performs consistently better than current joint slot-filling and intent-detection schemes and intent-focused capsule models on two datasets.
- CAPSULE-NLU is highly competitive with off-the-shelf natural language understanding systems, despite training word embeddings from scratch.The comparison uses stratified 5-fold cross validation because the original data split was unavailable.
- After the second routing iteration, WordCaps-to-SlotCaps agreement probabilities become more concentrated near 0 or 1.This indicates more confident routing of each word toward its most appropriate slot.
- For the unseen value operetta, rerouting becomes more certain that the phrase operetta for the theatre organ forms the B-object name and I-object name slots.Figure 5 compares agreement values after the first and second routing iterations, showing how context changes the routing decision.
- After the second iteration, I-object name and B-object type contribute more to the correct SearchCreativeWork intent capsule.The comparison concerns the same sample utterance used in Figure 5.
5 Related Works
Prior work commonly models intent detection and slot filling with neural sequence or attention-based architectures, while capsule models aggregate word features for intent classification. CAPSULE-NLU instead explicitly connects word-level slots and utterance-level intent through hierarchical capsule routing.
- Earlier systems use RNNs, CNNs, CRFs, and encoder-decoder architectures for slot filling and intent detection.These approaches may derive intent from recurrent final states or attention-weighted hidden representations.
- Existing joint approaches do not explicitly model the hierarchical relationships among words, slots, and intents.
- CAPSULE-NLU routes words to slot types and slot representations to intent representations using dynamic routing-by-agreement.
6 Conclusions
CAPSULE-NLU models the hierarchy among words, slots, and intents for joint slot filling and intent detection. It assigns words to slots through dynamic routing and aggregates learned slot representations into an utterance-level intent representation.
- CAPSULE-NLU harnesses hierarchical relationships among words, slots, and intents for joint slot filling and intent detection.