Source-linked AI summary
A Stack-Propagation Framework with Token-Level Intent Detection for Spoken Language Understanding
Libo Qin, Wanxiang Che, Yangming Li, Haoyang Wen, Ting Liu
TL;DR
Intent detection and slot filling are correlated SLU tasks, but prior joint models do not explicitly pass intent information to slot filling. The paper introduces Stack-Propagation with token-level intent detection and reports state-of-the-art results on two datasets, with further gains from BERT.
Problem
Prior joint SLU models mainly capture intent–slot relationships through parameter sharing, despite slots often depending strongly on intent.
Method
The paper uses a Stack-Propagation joint model that feeds intent-detection outputs into slot filling and adds token-level intent prediction.
Results
The framework achieves state-of-the-art performance on SNIPS and ATIS, with BERT further raising the reported results to a new state-of-the-art level.
Takeaways & Limitations
Explicit intent information can guide slot prediction within a jointly trained SLU framework, while token-level prediction is used to ease error propagation.
Takeaways & Limitations
A directly comparable evaluation setting still performs token-level intent detection, limiting how independently the Stack-Propagation effect is isolated.
Abstract
from arXiv · showhide
Intent detection and slot filling are two main tasks for building a spoken language understanding (SLU) system. The two tasks are closely tied and the slots often highly depend on the intent. In this paper, we propose a novel framework for SLU to better incorporate the intent information, which further guides the slot filling. In our framework, we adopt a joint model with Stack-Propagation which can directly use the intent information as input for slot filling, thus to capture the intent semantic knowledge. In addition, to further alleviate the error propagation, we perform the token-level intent detection for the Stack-Propagation framework. Experiments on two publicly datasets show that our model achieves the state-of-the-art performance and outperforms other previous methods by a large margin. Finally, we use the Bidirectional Encoder Representation from Transformer (BERT) model in our framework, which further boost our performance in SLU task.
1 Introduction
The paper frames intent detection and slot filling as correlated SLU tasks and proposes Stack-Propagation to pass intent information directly into slot filling. Token-level intent detection and BERT further improve robustness and reported benchmark performance.
- Motivation: SLU combines utterance-level intent detection with token-level slot filling, and slot labels can depend strongly on the detected intent.For example, a WatchMovie intent makes a movie-name slot more likely than a music-name slot.
- Related work: Existing joint models share parameters between intent detection and slot filling but do not explicitly pass upstream intent features to the downstream task.The paper positions explicit intent incorporation as the unresolved limitation of earlier multi-task approaches.
- Proposed framework: Stack-Propagation directly uses intent-detection outputs as slot-filling inputs, while token-level intent prediction supplies localized intent information and helps alleviate error propagation.The framework also supports oracle-intent experiments to examine how intent information enhances slot filling.
- Results: BERT is incorporated and further boosts the framework’s SLU performance.The paper explicitly explores the effect of incorporating BERT in SLU tasks.
2 Background
The background distinguishes intent classification from slot sequence labeling and contrasts shared-encoder multi-task learning with Stack-Propagation. Stack-Propagation explicitly passes upstream task features to a downstream task while retaining joint differentiable learning.
- Intent Detection and Slot Filling: Intent detection classifies an utterance into an intent label, whereas slot filling maps an input word sequence to a sequence of slot labels.The paper formalizes the two tasks as utterance-level classification and sequence labeling.
- Multi-task Framework vs. Stack-Propagation: A basic multi-task framework learns task correlations through a shared encoder but does not explicitly provide upstream features to the downstream task.This distinction motivates the Stack-Propagation comparison.
- Multi-task Framework vs. Stack-Propagation: Stack-Propagation lets the downstream task leverage upstream features without breaking differentiability, while joint learning promotes both tasks.The framework is presented as a way to combine explicit feature transfer with end-to-end joint optimization.
- Stack-Propagation Framework: The proposed SLU architecture uses one shared self-attentive encoder and two decoders, concatenating intent outputs with encoder representations for slot filling.The intent decoder performs token-level intent detection before the slot-filling decoder uses the combined representation.
3 Approach
The framework shares a self-attentive encoder, predicts intent at token level, and feeds explicit intent information into slot filling through Stack-Propagation. Joint training combines both tasks while token-level voting produces the utterance intent.
- 3.1 Self-Attentive Encoder: A shared encoder represents the utterance for both intent detection and slot filling, combining BiLSTM temporal features with self-attention contextual information.The self-attentive encoder produces contextual token representations, while the shared architecture captures knowledge common to both tasks.
- 3.2 Token-Level Intent Detection Decoder: Token-level intent detection maps each utterance to intent labels for individual tokens, using the sentence intent as every token’s training label.The final utterance intent is computed by voting over token-level predictions.
- 3.2 Token-Level Intent Detection Decoder: Token-level intent features provide slot filling with information at each token, reducing the impact of an incorrect whole-sentence intent prediction.Correct token-level predictions can remain useful for corresponding slots even when some token predictions are wrong.
- 3.3 Stack-propagation for Slot Filling: The slot decoder directly concatenates each token’s intent output distribution with its aligned encoder representation to guide slot prediction.This explicit intent information constrains slots toward a specific intent and alleviates the slot-filling decoder’s burden.
- 3.4 Joint Training: The framework converts sentence-level intent classification into token-level prediction and optimizes intent and slot objectives jointly through a combined loss.The joint loss allows shared encoder representations to consider both tasks together and further ease error propagation compared with pipeline models.
4.1 Experimental Settings
Experiments use the ATIS and SNIPS benchmark datasets with a shared self-attentive encoder and fixed regularization, dropout, and optimization settings.
- Datasets: Experiments use ATIS flight-reservation recordings and the SNIPS custom-intent-engines dataset, following the format and partition of Goo et al. (2018).Word embeddings have dimensionality 256 for ATIS and 512 for SNIPS.
- Model and Regularization: The self-attentive encoder uses 256 hidden units, with L2 regularization of 1 × 10^-6 and dropout ratio 0.4.These settings are used to reduce overfitting while training the model.
- Optimization and Evaluation: The model is optimized with Adam using its suggested hyperparameters, and the best development-set model is evaluated on the test set.Model selection is based on development-set performance.
4.2 Baselines
The evaluation compares the proposed model with joint, attention-based, gated, self-attentive, capsule, and directly connected slot-intent baselines.
- Joint Models: Joint Seq jointly models domain detection, intent detection, and slot filling in a single recurrent neural network architecture.
- Attention and Gating: Attention BiRNN uses attention to learn the relationship between slot filling and intent detection, while Slot-Gated Atten explores their correlation with a slot-gated joint model.
- Intent-Augmented Models: The Self-Attentive Model uses an intent-augmented gate mechanism, and Bi-Model models the cross-impact between intent and slot filling.
- Other Baselines: CAPSULE-NLU uses capsule networks with dynamic routing, while SF-ID Network establishes direct connections between slot filling and intent detection.
4.3 Overall Results
On SNIPS and ATIS, the proposed model outperforms prior baselines across slot filling, intent detection, and sentence-level semantic frame accuracy. Oracle-intent results further support using intent information to guide slot prediction.
- Evaluation: The evaluation reports slot filling with F1, intent prediction with accuracy, and sentence-level semantic frame parsing with overall accuracy.Table 2 presents results on the SNIPS and ATIS datasets.
- SNIPS: 0.7% improvement on SNIPS Slot (F1), 0.8% on Intent (Acc), and 3.1% on Overall (Acc) over Bi-Model.The cited comparison is against the best prior joint work.
- ATIS: 0.4% improvement on ATIS Slot (F1), 0.5% on Intent (Acc), and 0.8% on Overall (Acc) over the reported best prior joint work.The model achieves the state-of-the-art performance on the evaluated datasets.
- Statistical reporting: The reported improvements over all baselines are statistically significant at p < 0.05 under a t-test when marked with an asterisk.The table caption defines the meaning of the asterisk.
- Oracle intent: Using gold intent information produces better slot filling performance, supporting intent information as guidance for slot prediction.The paper presents this comparison in the oracle row of Table 2.
4.4 Analysis
Ablation studies attribute performance gains to Stack-Propagation, token-level intent detection, shared modeling, and self-attention. Removing these design elements degrades one or more SLU metrics, while the remaining framework still matches the cited state-of-the-art comparison.
- Effect of Stack-Propagation: Without Stack-Propagation, the gate-mechanism ablation significantly reduces slot filling (F1), intent detection (Acc), and overall accuracy (Acc).The comparison tests direct intent incorporation through a gate mechanism instead of Stack-Propagation.
- Effect of Stack-Propagation: Without a shared encoder, the pipelined model declines significantly on all metrics.The authors associate the shared Stack-Propagation model with learning task-correlation knowledge and easing error propagation.
- Token-level intent detection: Token-level intent detection performs better than utterance-level intent detection in the reported comparison.The paper attributes this to token-level intent prediction reducing predicted variance and providing more useful information for slot filling.
- Self-attention: Replacing self-attention with BiLSTM harms intent detection and slot filling, while self-attention further improves SLU performance.The authors attribute the gain to capturing contextual information for each token.
- Self-attention: Even without self-attention, the framework still performs the state-of-the-art Bi-Model comparison cited by the authors.This result is presented as evidence for the effectiveness and robustness of the other framework components.
4.5 Effect of BERT
Replacing the self-attentive encoder with fine-tuned BERT substantially strengthens the framework on ATIS and SNIPS. The joint BERT model also outperforms separately trained BERT-based slot filling and intent detection models.
- BERT results: The BERT-based model achieves new state-of-the-art performance on both ATIS and SNIPS.The authors attribute this result to rich semantic features from the pretrained model.
- Evaluation: Table 4 reports SLU performance for BERT-based models on ATIS and SNIPS.The table is specifically described as covering BERT-based model performance on the two datasets.
- BERT results: The model with BERT outperforms BERT SLU on overall accuracy on both datasets.BERT SLU jointly models the two tasks without explicit interaction between intent detection and slot filling.
- Joint versus separate BERT models: The joint BERT model yields higher slot filling (F1) and intent detection accuracy than separate BERT-based models.Separate models classify intent from [CLS] and slot tags from each token’s final hidden representation.
5 Related Work
Earlier SLU systems commonly treated slot filling and intent detection separately, while later joint models learned their correlation. More recent approaches explicitly inject intent information into slot filling through gating mechanisms, whereas this paper uses Stack-Propagation.
- Task formulations: Slot filling is formulated as sequence labeling, while intent detection is formulated as utterance classification.Prior approaches include CRF and RNN methods for slot filling and SVM and RNN methods for intent detection.
- Joint models: Pipeline approaches motivate joint models because they can suffer from error propagation between intent detection and slot filling.Earlier joint work uses RNNs, a single recurrent network, or attention-based modeling.
- Joint models: Earlier joint models modeled task correlation primarily through shared parameters rather than explicitly modeling intent information for slots.The paper distinguishes this parameter-sharing approach from explicit intent incorporation.
- Intent-aware models: Goo et al. and Li et al. introduced gate-based mechanisms to incorporate intent information into slot filling.These methods are described respectively as slot-gated and intent-augmented gate mechanisms.
- Intent-aware models: The paper’s framework differs from these approaches by using Stack-Propagation to incorporate intent information for slot filling.The related-work passage introduces this distinction, while the method passage identifies Stack-Propagation as the proposed framework.
6 Conclusion
The paper proposes a Stack-Propagation joint model that incorporates intent information into slot filling, with token-level intent detection to ease error propagation. Experiments show effectiveness and state-of-the-art performance, further improved by BERT.
- Stack-Propagation directly incorporates intent information into slot filling within a joint SLU model.The framework uses intent information to better guide slot prediction.
- Token-level intent detection further eases error propagation in the proposed framework.
- Experiments on two datasets show the proposed models achieve state-of-the-art performance.
- Using BERT in the framework reaches a new state-of-the-art level for SLU.