Source-linked AI summary
Attention-Based Recurrent Neural Network Models for Joint Intent Detection and Slot Filling
Bing Liu, Ian Lane
TL;DR
Spoken language understanding requires intent detection and slot filling, but slot filling has explicit alignment unlike many sequence-to-sequence tasks. The paper combines alignment information with attention-based encoder-decoder and RNN models, achieving state-of-the-art independent-task results and further gains from joint training.
Problem
SLU requires both intent detection and slot filling, while the usefulness of combining explicit slot alignment with attention-based and alignment-based sequence models remains under investigation.
Method
The paper integrates explicit alignment into attention-based encoder-decoder models and adds attention-based context vectors to bidirectional RNN slot-label prediction, with a shared encoder for joint intent and slot modeling.
Results
Independent models achieve state-of-the-art intent detection and slot filling on ATIS, while joint training yields 0.56% absolute (23.8% relative) intent detection improvement and 0.23% absolute slot filling gain over independent models.
Takeaways & Limitations
Joint modeling improves both intent detection and slot filling over independent training and requires only one model to be trained and deployed.
Abstract
from arXiv · showhide
Attention-based encoder-decoder neural network models have recently shown promising results in machine translation and speech recognition. In this work, we propose an attention-based neural network model for joint intent detection and slot filling, both of which are critical steps for many speech understanding and dialog systems. Unlike in machine translation and speech recognition, alignment is explicit in slot filling. We explore different strategies in incorporating this alignment information to the encoder-decoder framework. Learning from the attention mechanism in encoder-decoder model, we further propose introducing attention to the alignment-based RNN models. Such attentions provide additional information to the intent classification and slot label prediction. Our independent task models achieve state-of-the-art intent detection error rate and slot filling F1 score on the benchmark ATIS task. Our joint training model further obtains 0.56% absolute (23.8% relative) error reduction on intent detection and 0.23% absolute gain on slot filling over the independent task models.
1. Introduction
Spoken language understanding identifies a speaker’s intent and extracts semantic constituents through intent detection and slot filling. This work investigates combining explicit slot-alignment information with attention-based and alignment-based sequence models for these tasks.
- SLU systems support spoken dialogue by identifying speaker intent and extracting semantic constituents from natural-language queries.
- Intent detection is commonly treated as utterance classification, while slot filling is commonly treated as sequence labeling.
- Existing approaches often process intent detection and slot filling separately, although joint models have also been proposed.
- Encoder-decoder models encode an input sequence into a dense vector and generate an output sequence, while attention enables learned alignment during decoding.
- The paper explores how explicit slot-filling alignment can be incorporated into encoder-decoder models and whether attention can improve alignment-based RNN slot filling.
2. Background
Slot filling maps input words to aligned slot-label sequences, whereas encoder-decoder models can map sequences of different lengths without explicit alignment. RNNs model slot labels from input and previously emitted outputs, and attention-based encoder-decoder models learn soft alignment.
- Slot filling is formulated as sequence labeling, mapping an input sequence to a label sequence of the same length.
- Equal input and label sequence lengths provide explicit alignment between words and slot labels.
- At each time step, an RNN reads a word and predicts its slot label using available input information and emitted output sequences.
- The RNN encoder-decoder encodes the full source sequence into a vector and generates the target sequence, allowing different source and target lengths.
- Attention extends encoder-decoder models by learning a soft alignment while decoding the output sequence.
3. Proposed Methods
The paper develops two attention-based approaches for jointly modeling intent detection and slot filling, incorporating explicit slot-label alignment into encoder-decoder models and adding attention to alignment-based RNNs.
- 3.1. Encoder-Decoder Model with Aligned Inputs: The encoder-decoder approach integrates aligned encoder states into slot filling while using a bidirectional LSTM encoder and a unidirectional LSTM decoder.The backward encoder’s last state initializes the decoder, while decoding uses the previous state, previous label, aligned encoder state, and context vector.
- 3.1. Encoder-Decoder Model with Aligned Inputs: Joint modeling adds an intent decoder that shares the slot-filling encoder, with both decoder costs back-propagated to the encoder.The intent decoder produces one sentence-level intent distribution and uses a context vector because intent classification does not require alignment.
- 3.2. Attention-Based RNN Model: The attention-based RNN combines each time-aligned bidirectional hidden state with a weighted context vector to predict slot labels.The context vector aggregates RNN hidden states, supplementing local hidden-state information with information from other input positions.
- 3.2. Attention-Based RNN Model: The attention-based RNN reuses bidirectional hidden states for intent classification, using mean-pooling without attention or a weighted temporal average with attention.This design shares the sequence representation between slot filling and intent prediction.
- 3.2. Attention-Based RNN Model: The attention-based RNN is more computationally efficient than the aligned-input attention-based encoder-decoder because it reads the input sequence once instead of twice during training.The comparison concerns encoder input passes during model training.
4. Experiments
Experiments on ATIS evaluate independent and joint models for slot filling and intent detection, including attention-based encoder-decoder and bidirectional RNN architectures. The proposed models improve over prior results, while joint training further improves both tasks.
- Experiment setup: The experiments use the ATIS corpus with 4,978 training utterances, 893 test utterances, 127 slot labels, and 18 intent types.An additional corpus contains 5,138 annotated utterances with 110 slot labels and 21 intent types, evaluated using 10-fold cross validation.
- Experiment setup: The models use one-layer LSTMs with 128 units, 128-dimensional trainable word embeddings, dropout rate 0.5, gradient clipping at 5, and Adam optimization.
- Independent slot filling: Pure attention-based slot filling without explicit alignment performs poorly, whereas attention with aligned inputs gives slightly better average and best F1 scores than the non-attention model.The learned attention is often distributed across the source sequence, with a few cases showing useful supporting information for slot prediction.
- Independent slot filling: For bidirectional RNN slot filling models, attention also slightly improves F1, although the context vector contributes little when the average ATIS sentence length is 11.The bidirectional hidden state appears to encode most information needed for slot-label prediction at this sequence length.
- Independent results: Both proposed model architectures advance the best previously reported slot-filling F1 scores, while the proposed intent models outperform prior state-of-the-art intent error rates by a large margin.The attention-based encoder-decoder intent model outperforms the bidirectional RNN intent model.
- Joint model results: Joint training improves the encoder-decoder model by 0.09% absolute on slot filling and 0.45% absolute on intent detection, or 22.2% relative improvement.For the attention-based bidirectional RNN, joint training yields 0.23% absolute slot-filling gain and 0.56% absolute intent-detection gain, or 23.8% relative improvement; both joint approaches outperform the best reported joint results.
5. Conclusions
The paper explores how explicit alignment can be used in attention-based encoder-decoder models and proposes attention-based bidirectional RNNs for joint intent detection and slot filling. Independent models achieve state-of-the-art ATIS performance, while joint models improve both tasks further.
- The work explores strategies for incorporating explicit alignment information into attention-based encoder-decoder models.
- The paper proposes an attention-based bidirectional RNN model for jointly detecting intent and filling slots.
- Independent training models achieve state-of-the-art performance for both intent detection and slot filling on the benchmark ATIS task.
- Joint training further improves intent detection accuracy and slot-filling F1 over the independent training models.