PaperlayerView exact revision on arXiv

Source-linked AI summary

BERT for Joint Intent Classification and Slot Filling

Qian Chen, Zhu Zhuo, Wen Wang

arXiv:1902.10909v1cs.CL

TL;DR

Limited labeled data makes intent classification and slot filling difficult to generalize, especially for rare words, while BERT offers pre-trained bidirectional representations. The paper fine-tunes BERT in a joint model for both tasks and reports significant gains across intent accuracy, slot-filling F1, and sentence-level semantic frame accuracy on ATIS and Snips.

  • Problem

    Limited human-labeled NLU data leads to poor generalization capability, especially for rare words.

  • Method

    The paper fine-tunes a BERT-based joint model that predicts intent from [CLS] and slot labels from token hidden states.

  • Results

    The joint BERT model significantly improves intent classification accuracy, slot filling F1, and sentence-level semantic frame accuracy over previous state-of-the-art models on ATIS and Snips.

  • Takeaways & Limitations

    Joint BERT is effective for exploiting the relationship between intent classification and slot filling in these benchmark evaluations.

Abstract

from arXiv · show

Intent classification and slot filling are two essential tasks for natural language understanding. They often suffer from small-scale human-labeled training data, resulting in poor generalization capability, especially for rare words. Recently a new language representation model, BERT (Bidirectional Encoder Representations from Transformers), facilitates pre-training deep bidirectional representations on large-scale unlabeled corpora, and has created state-of-the-art models for a wide variety of natural language processing tasks after simple fine-tuning. However, there has not been much effort on exploring BERT for natural language understanding. In this work, we propose a joint intent classification and slot filling model based on BERT. Experimental results demonstrate that our proposed model achieves significant improvement on intent classification accuracy, slot filling F1, and sentence-level semantic frame accuracy on several public benchmark datasets, compared to the attention-based recurrent neural network models and slot-gated models.

1 Introduction

NLU for goal-oriented dialogue typically combines intent classification with slot filling, but limited labeled data weakens generalization. This work explores BERT-based joint modeling to address that gap and improve all three reported evaluation measures.

  • NLU tasks: NLU forms semantic parses by predicting an utterance’s intent and extracting its semantic concepts as slots.Intent classification is treated as classification, while slot filling is treated as sequence labeling.
  • Prior approaches: Joint learning methods exploit dependencies between intent classification and slot filling to improve over independent models.Prior joint approaches include recurrent, attention-based, and slot-gated models.
  • Motivation: Limited human-labeled data creates a generalization challenge for NLU and other NLP tasks, motivating pre-training on large-scale unlabeled text.ELMo, GPT, and BERT are presented as approaches for learning general-purpose language representations.
  • Motivation: BERT uses bidirectional pre-training and has produced state-of-the-art models for diverse NLP tasks after fine-tuning.The cited examples include question answering and natural language inference.
  • Contribution: The paper proposes a joint BERT model and reports significant gains in intent accuracy, slot-filling F1, and sentence-level semantic frame accuracy.The comparisons are against attention-based RNN and slot-gated models on public benchmark datasets.

2 Related work

Related work distinguishes NLU systems that model intent classification and slot filling independently from systems that model them jointly. The literature spans convolutional, recurrent, attention-based, adversarial, and structured-prediction approaches.

  • Modeling paradigms: NLU models are categorized according to whether intent classification and slot filling are modeled separately or jointly.This distinction organizes the related approaches reviewed in the paper.
  • Independent modeling: Intent classification has been addressed with CNN, LSTM, attention-based CNN, hierarchical attention, and adversarial multi-task models.The review also lists other approaches beyond these examples.
  • Independent modeling: Slot filling has been studied with CNN, deep LSTM, RNN-EM, encoder-labeler deep LSTM, and joint pointer-and-attention models.These methods represent sequence-labeling approaches surveyed by the paper.
  • Joint modeling: Joint approaches include CNN-CRF, RecNN, joint RNN-LSTM, attention-based BiRNN, and slot-gated attention-based models.These methods explicitly belong to the paper’s joint-modeling category.

3 Proposed Approach

The proposed approach fine-tunes BERT for joint intent classification and slot filling. It uses the [CLS] representation for intent prediction, token representations for slot labels, and a joint conditional-probability objective, with CRF investigated for label dependencies.

  • 3.1 BERT: BERT is a multi-layer bidirectional Transformer encoder whose input combines WordPiece, positional, and segment embeddings.For single-sentence classification and tagging, [CLS] starts the sequence and [SEP] ends it.
  • 3.1 BERT: Pre-trained BERT supplies context-dependent sentence representations that can be fine-tuned for intent classification and slot filling.Its pre-training uses masked language modeling and next sentence prediction on large-scale unlabeled text.
  • 3.2 Joint Intent Classification and Slot Filling: The joint model predicts intent from the hidden state of [CLS] and predicts slot labels from the final hidden states of the other tokens.For WordPiece inputs, the first sub-token representation is used for each tokenized word.
  • 3.2 Joint Intent Classification and Slot Filling: The slot-label equation applies a softmax classifier to each word’s first-sub-token hidden state.The prose identifies h_n as the hidden state corresponding to the first sub-token of word x_n.
  • 3.2 Joint Intent Classification and Slot Filling: Joint training maximizes p(y_i, y_s|x) and fine-tunes the model end-to-end by minimizing cross-entropy loss.The objective is designed to model intent and slot filling together.
  • 3.3 Conditional Random Field: The paper investigates adding a CRF layer because slot-label predictions depend on surrounding-word predictions.This tests structured modeling of slot-label dependencies on top of joint BERT.

4 Experiments and Analysis

The proposed joint BERT model is evaluated on ATIS and Snips, where it improves intent classification, slot filling, and sentence-level semantic frame accuracy over baseline joint models. Ablation and case-study results further examine the contribution of joint learning and BERT’s pretrained representations.

  • Results: Joint BERT significantly outperforms attention-based RNN, BiLSTM, and slot-gated baselines on ATIS and Snips.The evaluation reports slot filling F1, intent classification accuracy, and sentence-level semantic frame accuracy.
  • Results: 98.6% intent accuracy, 97.0% slot filling F1, and 92.8% semantic frame accuracy are achieved on Snips, versus 97.0%, 88.8%, and 75.5%.
  • Results: 97.5% intent accuracy, 96.1% slot filling F1, and 88.2% semantic frame accuracy are achieved on ATIS, versus 94.1%, 95.2%, and 82.6%.
  • Analysis: 22.9% relative improvement in Snips sentence-level semantic frame accuracy exceeds the 6.8% relative improvement on ATIS.The paper relates the larger Snips gain to its multiple domains and larger vocabulary.
  • Ablation Analysis: Without joint learning, Snips intent accuracy falls to 98.0% and slot filling F1 falls to 95.8%; one fine-tuning epoch still outperforms the baseline group.
  • Case Study: A rare movie phrase is correctly assigned its slot labels and intent by joint BERT but misclassified by the slot-gated model.The authors suggest Wikipedia pretraining may have provided information about the phrase.

5 Conclusion

The paper proposes a joint BERT model for intent classification and slot filling to address poor generalization in traditional NLU models. On ATIS and Snips, it improves multiple evaluation metrics over previous state-of-the-art models, while future work targets larger datasets and external knowledge.

  • The proposed model jointly performs intent classification and slot filling using BERT to address poor generalization in traditional NLU models.
  • Joint BERT improves intent accuracy, slot filling F1, and sentence-level semantic frame accuracy over previous state-of-the-art models on ATIS and Snips.
  • Future work includes evaluation on larger, more complex NLU datasets and combining external knowledge with BERT.
Loading 1902.10909v1…