Source-linked AI summary
Recurrent Convolutional Neural Networks for Discourse Compositionality
Nal Kalchbrenner, Phil Blunsom
TL;DR
The paper addresses how meaning composes beyond individual sentences, where principled discourse compositionality approaches have been limited. It introduces hierarchical convolutional sentence composition and a speaker-conditioned recurrent discourse model, achieving state-of-the-art dialogue act classification without feature engineering or pretraining.
Problem
Principled approaches to discourse compositionality have largely been unexplored despite the need to model how discourse meaning composes from sentences and words.
Method
The paper combines a hierarchical convolutional neural network for sentence vectors with a recurrent discourse model conditioned on the current sentence and speaker.
Results
The coupled model achieves state-of-the-art dialogue act classification performance without feature engineering or pretraining, using simple greedy decoding.
Takeaways & Limitations
The discourse model captures sentence sequentiality and speaker interaction while producing discourse representations sensitive to discourse structure.
Takeaways & Limitations
Multiple merged HCNNs are described as potentially useful but are not used in the experiment.
Abstract
from arXiv · showhide
The compositionality of meaning extends beyond the single sentence. Just as words combine to form the meaning of sentences, so do sentences combine to form the meaning of paragraphs, dialogues and general discourse. We introduce both a sentence model and a discourse model corresponding to the two levels of compositionality. The sentence model adopts convolution as the central operation for composing semantic vectors and is based on a novel hierarchical convolutional neural network. The discourse model extends the sentence model and is based on a recurrent neural network that is conditioned in a novel way both on the current sentence and on the current speaker. The discourse model is able to capture both the sequentiality of sentences and the interaction between different speakers. Without feature engineering or pretraining and with simple greedy decoding, the discourse model coupled to the sentence model obtains state of the art performance on a dialogue act classification experiment.
1 Introduction
Meaning is composed at both sentence and discourse levels, but principled discourse compositionality has remained largely unexplored. The paper addresses these levels with hierarchical convolutional and speaker-conditioned recurrent models, evaluated on dialogue act classification.
- Motivation: Discourse compositionality models how sentence meanings combine into paragraphs, documents, or dialogues, ultimately relating discourse meaning to words.
- Motivation: Existing discourse approaches largely use bag-of-n-grams, sentence averaging, or non-compositional language models coupled to hidden Markov models.
- Sentence model: The HCNN composes word vectors with feature-wise one-dimensional convolutions arranged in a hierarchy of increasing kernel sizes.
- Discourse model: The RCNN models discourse sequentiality and speaker interaction by conditioning recurrent and output weights on speakers while consuming the current sentence vector.
- Evaluation: The coupled models are evaluated on dialogue act recognition using 1134 telephone conversations and about 200K utterances, with random initialization, supervised training, and greedy decoding.
- Evaluation: The paper reports state-of-the-art dialogue act classification performance without feature engineering or pretraining.
2 Sentence Model
The sentence model represents a sentence through hierarchical, feature-wise one-dimensional convolutions over its word-vector matrix. Increasing kernel sizes compose local word patterns into a single sentence vector while keeping convolution weights feature- and layer-dependent rather than word-specific.
- 2 Sentence Model: The model computes a sentence vector from a word-vector matrix using hierarchical one-dimensional convolutions applied separately to each feature.Each matrix row represents one feature across the sentence's word vectors.
- 2.1 Sentential compositionality: Feature-wise composition initially combines values of the same feature across word vectors, before discourse-level transformations can mix sentence-vector features.The sentence model preserves feature-wise processing until the sentence vector is computed.
- 2.1 Sentential compositionality: Increasing convolution kernel sizes model increasingly global effects, progressing from neighboring words toward the whole sentence.The hierarchy uses kernels that span an increasing number of words.
- 2.2 Sentence Composition: A one-dimensional convolution applies the same kernel weights across each feature's subsequences, while different features use different kernels.Word-specific parameters enter through learned word vectors rather than separately learned word-specific operations.
- 2.2.3 Composition Operation in a HCNN: The hierarchy recursively applies feature- and layer-specific kernels to produce matrices whose top layer is a single vector representing the sentence.The resulting architecture is a hierarchical convolutional neural network.
- 2.2.4 Multiple merged HCNNs: Multiple parallel HCNNs can optionally be merged at the top sentence-vector layer or intermediate layers, but this option is not used in the experiment.Word-vector weights may be tied across the parallel HCNNs.
3 Discourse Model
The discourse model extends an RNN with sentence representations and speaker-conditioned weights to model sentence order and speaker interactions.
- The model targets two discourse properties: utterance sequentiality and interactions between different speakers.
- The discourse model couples an RNN to a hierarchical convolutional sentence model, using sentence vectors as inputs.
- Recurrent and output weights are conditioned on the speaker associated with each utterance.
- At each step, the RCNN models a probability distribution for the current label from the discourse sequence and speaker-conditioned recurrent computation.
- The hidden layer h_i represents the discourse up to step i.
4 Predicting Dialogue Acts
The model is evaluated for dialogue-act prediction on Switchboard conversations using transcripts, act labels, and speaker labels. Its learned discourse vectors reflect pragmatic similarity across utterances, speakers, and act sequences.
- The experiment recognizes dialogue acts in the Switchboard corpus, where dialogue acts encode an utterance’s pragmatic role and speaker intentions.
- The model uses each utterance’s transcript, dialogue-act label, and speaker label, without other annotations.
- Depth d = 2 makes each prediction depend on the previous two utterances, speakers, and dialogue acts; greater depths yielded no improvement.
- The RCNN predicts dialogue acts greedily from preceding predicted acts, while requiring only utterances available up to the current step.
- The discourse-vector analysis reports nearest neighbours for short dialogues and finds pragmatic similarity despite semantically different components.
5 Conclusion
The paper combines novel convolutional sentence composition with speaker-conditioned recurrent discourse modeling. It reports state-of-the-art dialogue-act classification and discourse representations sensitive to discourse structure.
- The paper introduces a hierarchical convolutional sentence model and a recurrent discourse model for composing meaning across sentences and discourse.
- The coupled models achieve state-of-the-art dialogue-act classification without feature engineering or pretraining and with greedy decoding.
- The discourse model produces representations sensitive to discourse structure and potentially useful for further semantic data and unsupervised pretraining.