PaperlayerView exact revision on arXiv

Source-linked AI summary

Dialogue Act Sequence Labeling using Hierarchical encoder with CRF

Harshit Kumar, Arvind Agarwal, Riddhiman Dasgupta, Sachindra Joshi, Arun Kumar

arXiv:1709.04250v2cs.CL

TL;DR

Dialogue-act recognition assigns semantic labels to utterances, but prior approaches do not consistently capture dependencies among both utterances and labels. This paper uses a hierarchical Bi-LSTM encoder with a CRF to model those dependencies, and reports higher accuracy than prior state-of-the-art methods on SwDA and MRDA.

  • Problem

    Dialogue-act recognition must account for dependencies among consecutive utterances and labels, which independent classifiers and some structured models handle incompletely.

    Source: Introduction · p. 1

  • Method

    A hierarchical bidirectional LSTM encodes words, utterances, and conversation, and a linear-chain CRF models dependencies among dialogue-act labels.

    Source: Introduction · p. 2, Linear Chain CRF · p. 4

  • Results

    79.2% accuracy versus 77% on SwDA, and 90.9% versus 86.8% on MRDA, outperforming the reported state-of-the-art on both datasets.

    Source: Introduction · p. 2

  • Takeaways & Limitations

    The reported benchmark results support combining utterance-level representations with label-sequence modeling for dialogue-act recognition on SwDA and MRDA.

    Source: Introduction · p. 2

    SwDA labels include subjective or indistinguishable cases, and the dataset’s reported inter-labeler agreement is 84%.

    Source: Results and Discussion · p. 6

Abstract

from arXiv · show

Dialogue Act recognition associate dialogue acts (i.e., semantic labels) to utterances in a conversation. The problem of associating semantic labels to utterances can be treated as a sequence labeling problem. In this work, we build a hierarchical recurrent neural network using bidirectional LSTM as a base unit and the conditional random field (CRF) as the top layer to classify each utterance into its corresponding dialogue act. The hierarchical network learns representations at multiple levels, i.e., word level, utterance level, and conversation level. The conversation level representations are input to the CRF layer, which takes into account not only all previous utterances but also their dialogue acts, thus modeling the dependency among both, labels and utterances, an important consideration of natural dialogue. We validate our approach on two different benchmark data sets, Switchboard and Meeting Recorder Dialogue Act, and show performance improvement over the state-of-the-art methods by $2.2\%$ and $4.1\%$ absolute points, respectively. It is worth noting that the inter-annotator agreement on Switchboard data set is $84\%$, and our method is able to achieve the accuracy of about $79\%$ despite being trained on the noisy data.

Introduction

Dialogue acts characterize speakers’ intentions and can support dialogue systems and conversation summarization. The paper proposes a hierarchical Bi-LSTM-CRF to model dependencies among utterances and dialogue-act labels, improving results on Switchboard and MRDA.

Introduction

  • Dialogue acts characterize speakers’ intentions and can help dialogue systems predict replies and summarizers select utterances by act.A prior Greeting makes another Greeting more likely; the paper cites dialogue-system work by Higashinaka et al. 2014.Source: Introduction · p. 1
  • Independent classifiers ignore dependencies, while HMMs account for label dependencies but not dependencies among utterances; handcrafted features also limit scaling across datasets.Earlier SwDA results included 66% accuracy for Naïve Bayes, 71% for HMM, and 74.32% for SVM-HMM, versus 84% reported human agreement.Source: Introduction · p. 1
  • The proposed Hierarchical Bi-LSTM-CRF captures dependencies among both utterances and dialogue-act labels, unlike prior approaches that capture only one kind.Its hierarchy represents words, utterances, and conversation, with a CRF classification layer.Source: Introduction · p. 2
  • 79.2% accuracy on SwDA versus 77% state-of-the-art, and 90.9% on MRDA versus 86.8%.On SwDA, the result is closer to the reported human inter-annotator agreement of 84%.Source: Introduction · p. 2
  • Adding linguistic features or intra-attention context did not improve performance, although additional context helped model convergence during training.The context and linguistic-feature variants are described as analyses atop the proposed model.Source: Introduction · p. 2

Related Work

Prior dialogue-act recognition work includes sequence prediction and independent utterance classification, with deep models and structured prediction addressing different parts of the task. The related work contrasts their treatment of utterance and label dependencies.

Related Work

  • The literature groups dialogue-act recognition into sequence prediction for whole conversations and independent prediction for each utterance.The passage identifies both categories with prior sequence-labeling and independent-classification work.Source: Related Work · p. 2
  • 71% accuracy on SwDA was reported for HMM, while MRDA’s cited naïve Bayesian result was 82%.The SwDA HMM used hand-crafted features with contextual and lexical information.Source: Related Work · p. 2
  • Deep models vary in dependency handling: CNN/RNN context models use preceding texts, HCNN-RNN predicts utterances individually, and LVRNN uses prior context without directly modeling label dependencies.The cited studies are Lee and Dernoncourt 2016, Blunsom and Kalchbrenner 2013, and Ji, Haffari, and Eisenstein 2016.Source: Related Work · p. 2
  • Earlier LSTM-CRF sequence taggers model label dependencies but do not use language’s hierarchical structure to capture utterance dependencies in a principled way.The cited systems are Huang, Xu, and Yu 2015 and Ma and Hovy 2016.Source: Related Work · p. 2

Methodology

The method represents conversations as utterances of words, then uses a hierarchical bidirectional recurrent encoder and linear-chain CRF to predict dialogue acts. Its hierarchy encodes utterance and conversation context, while the CRF models label dependencies.

Methodology

  • A flat chain of all conversation words would be extremely long and make backpropagation impractical, motivating separate utterance- and conversation-level encoders.The conversation-level encoder processes utterance representations to capture dependencies among utterances.Source: Methodology · pp. 2–3
  • The model combines embedding, utterance-encoder, conversation-encoder, and CRF-classifier layers, and trains end-to-end across word, utterance, and conversation representations.It predicts one dialogue act for each utterance.Source: Figure 1 · p. 3, Methodology · p. 3

Hierarchical Recurrent Encoder

  • Word embeddings feed a word-level bidirectional LSTM; the final word-step representation forms each utterance vector, which a second bidirectional LSTM contextualizes across the conversation.The authors omit additional character- or subword-level granularity to keep model complexity manageable.Source: Hierarchical Recurrent Encoder · pp. 3–4

Linear Chain CRF

  • A linear-chain CRF models neighboring-label dependencies with pairwise transition potentials and encoder-derived unary features; maximum-likelihood training and Viterbi decoding produce the label sequence.The transition matrix is position-independent, and the model restricts potentials to pairwise terms.Source: Linear Chain CRF · p. 4

Experiments

Experiments evaluate the hierarchical Bi-LSTM-CRF on Switchboard and MRDA, testing benchmark performance and the effects of hierarchy and added features. The full hierarchy with a CRF performs best, while added POS and context representations do not improve final accuracy.

Experiments

Datasets

  • SwDA contains 1,155 telephone conversations with 42 dialogue-act classes, while MRDA comprises 72 hours of meetings mapped to five classes.The datasets are imbalanced: non-opinion and backchannel exceed half of SwDA utterances, while statements exceed half of MRDA utterances.Source: Datasets · pp. 4–5

Hyperparameter Tuning

  • Hyperparameters were tuned one at a time on the SwDA validation set, then the selected values were used to train models on both datasets.Increasing the number of stacked LSTM layers reduced accuracy, so the authors used one layer.Source: Hyperparameter Tuning · p. 5, Table 3 · p. 5

Results and Discussion

Effect of Hierarchy and Label Dependency

  • With a CRF, WE, WE+UL, and WE+UL+CL achieve accuracies of 72.2%, 72.7%, and 79.2%, respectively, versus 71.4%, 72.2%, and 74.1% with LR.Adding the conversation layer improves accuracy by approximately 2 absolute points with LR and 6 with CRF; its vectors represent the utterance and preceding utterances.Source: Effect of Hierarchy and Label Dependency · pp. 6–7, Table 9 · p. 7

Effect of Linguistic Features and Context

  • POS features reduce accuracy by approximately 1%, and adding POS or context representations does not improve final model performance.Attention combines the current utterance vector with a weighted sum of previous utterances; modeling longer dependencies reduces performance. Added context improves first-epoch accuracy on SwDA to 68.8% from 65.1% and on MRDA to 88% from 87%.Source: Effect of Linguistic Features and… · p. 7, Table 10 · p. 7

CONCLUSION

The Hierarchical Bi-LSTM-CRF outperforms the state of the art on SwDA and MRDA. It represents utterances using word- and conversation-level context, then uses a CRF to model dependencies between dialogue-act labels and utterance representations.

  • The proposed model outperforms the state of the art on both SwDA and MRDA.Source: CONCLUSION · p. 7
  • Bidirectional LSTMs capture long-term dependencies between words within utterances and across utterances, producing a vector representation for each utterance.Source: CONCLUSION · p. 7
  • A CRF classifies the utterance-vector sequence while modeling dependencies between dialogue-act labels and utterance representations.Source: CONCLUSION · p. 7
Loading 1709.04250v2…