Source-linked AI summary
Leveraging Sentence-level Information with Encoder LSTM for Semantic Slot Filling
Gakuto Kurata, Bing Xiang, Bowen Zhou, Mo Yu
TL;DR
Slot filling assigns semantic labels to words, and this paper extends LSTM-based sequence labeling to account for the whole input sentence. Its encoder-labeler LSTM encodes the sentence to initialize a labeler, and the approach achieves a state-of-the-art 95.66% F1-score on standard ATIS.
Problem
Slot filling requires estimating semantic slot labels for each word, while the paper identifies sentence-level information as an additional consideration for LSTM-based slot filling.
Source: §1 · p. 1Method
The encoder-labeler LSTM encodes the input sentence into a fixed-length vector that initializes a labeler LSTM, with a variant also modeling label dependencies.
Source: §1 · p. 1, §2.3 · p. 2Results
95.66% F1-score was reported as state of the art for slot filling on the standard ATIS corpus.
Source: §1 · p. 2Takeaways & Limitations
Explicit label-dependency modeling was not beneficial in these experiments and the authors identify it for further investigation.
Source: §4 · p. 5Absolute F1-scores cannot be compared with Liu et al. (2013b) because capitalization policy and training-data size differed.
Source: §4 · p. 5
Abstract
from arXiv · showhide
Recurrent Neural Network (RNN) and one of its specific architectures, Long Short-Term Memory (LSTM), have been widely used for sequence labeling. In this paper, we first enhance LSTM-based sequence labeling to explicitly model label dependencies. Then we propose another enhancement to incorporate the global information spanning over the whole input sequence. The latter proposed method, encoder-labeler LSTM, first encodes the whole input sequence into a fixed length vector with the encoder LSTM, and then uses this encoded vector as the initial state of another LSTM for sequence labeling. Combining these methods, we can predict the label sequence with considering label dependencies and information of whole input sequence. In the experiments of a slot filling task, which is an essential component of natural language understanding, with using the standard ATIS corpus, we achieved the state-of-the-art F1-score of 95.66%.
1 Introduction
The paper extends LSTM-based slot filling to use sentence-level information, alongside explicit modeling of label dependencies. On standard ATIS, the proposed approach achieves a state-of-the-art F1-score of 95.66%.
1 Introduction
- Slot filling is framed as estimating semantic slot labels for each word in the input sequence.The paper focuses on slot filling as a component of natural language understanding.Source: §1 · p. 1
- Prior RNN/LSTM slot-filling work captured temporal dependencies and was extended to model label dependencies explicitly.The paper builds on this line of work to incorporate sentence-level information.Source: §1 · p. 1
- The encoder-labeler LSTM encodes the input sentence into a fixed-length vector and uses it to initialize the labeler LSTM.The labeler predicts the slot sequence while taking sentence-level information into consideration.Source: §1 · pp. 1–2
- 95.66% F1-score was state of the art for slot filling on the standard ATIS corpus.The paper reports this result for its encoder-labeler deep LSTM(W).Source: §1 · p. 2
2 Proposed Method
The proposed method combines word-based slot labeling with explicit label-dependency modeling and sentence-level information encoded from the input. Its encoder-labeler architecture initializes a labeler with the encoder’s representation of the whole sentence.
2 Proposed Method
- The method section first revisits LSTM slot filling, adds explicit label-dependency modeling, and then introduces the encoder-labeler LSTM.These are the two enhancements to LSTM-based sequence labeling described in the section.Source: §2 · p. 2
2.1 LSTM for Slot Filling
- The labeler LSTM(W) feeds words to the LSTM and produces a hidden state at each time step for slot-label prediction.A context window can include preceding and succeeding words; a softmax layer estimates label probabilities from each hidden state.Source: §2.1 · p. 2
2.2 Explicit Modeling of Label Dependency
- In labeler LSTM(W+L), the previous time step’s label is fed to the current hidden state alongside the words.Ground-truth previous labels are used for training, while evaluation uses left-to-right beam search.Source: §2.2 · p. 2
2.3 Encoder-labeler LSTM for Slot Filling
- The encoder reads the sentence backward, and its last hidden state initializes a labeler that predicts slots conditioned on the encoded sentence.The W+L variant combines this sentence-level conditioning with label-dependency modeling; labeler errors are propagated to the encoder during training.Source: §2.3 · p. 2
- Unlike the usual encoder-decoder LSTM, the encoder-labeler accepts the same input sequence twice and shares its word embedding matrix across the encoder and labeler.The encoding and labeling LSTMs are different.Source: §2.3 · p. 2
2.4 Related Work on Considering Sentence-level Information
- Unlike bidirectional RNN/LSTMs that model preceding and succeeding context at each word, the encoder-labeler explicitly encodes the whole sentence.The paper also discusses attention-based slot filling and directly feeds input words to the labeler using a context window.Source: §2.4 · p. 3, Figure 1 · p. 3
3 Experiments
The experiments evaluate encoder-labeler LSTMs on standard ATIS and a merged multi-domain dataset, testing sentence-level information, label dependencies, and published-result comparisons.
3.1.1 Experimental Setup
- ATIS results used 80% of the training sentences for model training and the remaining 20% as heldout data for hyperparameter selection.The corpus has 4,978 training and 893 evaluation sentences; reported F1 is on evaluation data using hyperparameters selected on heldout data.Source: §3.1.1 · p. 3
3.1.2 Improvement by Encoder-labeler LSTM
- Both encoder-labeler LSTM variants outperformed their corresponding labeler LSTM baselines, but adding label dependencies did not improve F1 over encoder-labeler LSTM(W).After the first label prediction error, accuracy fell from 84.0% to 82.6% when using label dependencies; a two-layer encoder-labeler LSTM(W) improved F1 from 94.91% to 95.47%.Source: §3.1.2 · p. 4
3.1.3 Comparison with Published Results
- The comparison with published ATIS methods distinguishes encoder-labeler LSTM by its use of an encoder LSTM for sentence-level information.Some prior results used all training data, while these experiments used a randomly selected 80% for training and 20% for hyperparameter tuning.Source: §3.1.3 · p. 5
- 95.66% F1-score versus the previously published best of 95.56% was achieved by encoder-labeler deep LSTM(W) on ATIS.The proposed methods were further tuned with random search; the result is reported in Table 2.Source: §3.1.3 · p. 5, Table 2 · p. 5
3.2 Large-scale Experiment
- On the merged dataset, encoder-labeler LSTM(W) reached 74.41% F1-score, compared with 72.80% for labeler LSTM(W).The dataset contains 30,229 training and 6,810 evaluation sentences, 191 slot labels, and a vocabulary of 16,049 words.Source: §3.2 · p. 5
4 Conclusion
The paper proposes an encoder-labeler LSTM for slot filling conditioned on sentence-level information and reports a state-of-the-art result on standard ATIS. Explicitly modeling label dependencies was not beneficial in the experiments; joint training is suggested for future exploration.
4 Conclusion
- The encoder-labeler LSTM conditions slot filling on encoded sentence-level information and achieved a state-of-the-art F1-score on standard ATIS.The paper also reports that explicitly modeling label dependencies was not beneficial in its experiments.Source: §4 · p. 5
- The authors suggest exploring encoder-labeler LSTM in joint slot-filling and intent-classification training.They note prior reports that joint training boosted accuracy on both tasks.Source: §4 · p. 5
- Absolute F1-scores cannot be compared with Liu et al. (2013b) because capitalization policy and training-data size differed.Source: §4 · p. 5