Source-linked AI summary
A Latent Variable Recurrent Neural Network for Discourse Relation Language Models
Yangfeng Ji, Gholamreza Haffari, Jacob Eisenstein
TL;DR
The paper addresses how to combine predictive neural language modeling with probabilistic treatment of discourse relations between adjacent sentences. It introduces a recurrent neural model with latent discourse relations, and reports gains on two discourse-classification tasks and language modeling. Its modeling choices leave sequential dependencies between discourse relations uncoupled, with more complex coupling posing training and inference challenges.
Problem
Neural architectures learn predictive distributed representations but lack flexibility to marginalize unavailable annotations, while graphical models become difficult to train with many layers of latent variables.
Source: §1 · p. 1Method
The model combines a recurrent neural language model with latent variables for shallow discourse relations between adjacent sentences, using objectives for discourse prediction or joint text-and-relation modeling.
Source: §1 · p. 1, §3.1 · p. 3, §3.3 · p. 4Results
The model outperforms state-of-the-art systems on two discourse relation detection tasks and, when discourse relations are marginalized at test time, outperforms language models that do not account for discourse structure.
Source: §1 · pp. 1–2, §7 · p. 9Takeaways & Limitations
The model provides a probabilistic representation of shallow discourse relations while retaining discriminatively trained vector representations, and can also be used as a discourse-informed language model.
Source: §7 · p. 9The model decouples discourse relations after conditioning on words; adding sequential dependencies would increase training costs and require dynamic programming.
Source: §3.4 · p. 5
Abstract
from arXiv · showhide
This paper presents a novel latent variable recurrent neural network architecture for jointly modeling sequences of words and (possibly latent) discourse relations between adjacent sentences. A recurrent neural network generates individual words, thus reaping the benefits of discriminatively-trained vector representations. The discourse relations are represented with a latent variable, which can be predicted or marginalized, depending on the task. The resulting model can therefore employ a training objective that includes not only discourse relation classification, but also word prediction. As a result, it outperforms state-of-the-art alternatives for two tasks: implicit discourse relation classification in the Penn Discourse Treebank, and dialog act classification in the Switchboard corpus. Furthermore, by marginalizing over latent discourse relations at test time, we obtain a discourse informed language model, which improves over a strong LSTM baseline.
1 Introduction
The paper combines a recurrent neural language model with latent variables for relations between adjacent sentences. This hybrid supports discourse classification and discourse-informed language modeling, and outperforms state-of-the-art alternatives on two discourse-classification tasks.
1 Introduction
- The hybrid addresses neural architectures’ limited flexibility and graphical models’ difficulty training with many layers of latent variables.Graphical models can marginalize over unavailable annotations, while neural architectures learn predictive distributed representations.Source: §1 · p. 1
- The model combines discriminatively trained neural representations with a probabilistic treatment of shallow discourse relations between adjacent sentences.This lets it act as both a discourse relation classifier and a language model.Source: §1 · p. 1
- It outperforms state-of-the-art methods on implicit discourse relation classification in the Penn Discourse Treebank and dialog act classification in Switchboard.Unlike previous recursive neural architectures that learn only from annotated discourse relations, it learns from discourse annotations and a language-modeling objective.Source: §1 · pp. 1–2
- Marginalizing over discourse relations at test time lets the jointly trained model outperform language models that do not account for discourse structure.Source: §1 · p. 2
- The model requires only minimal modifications to existing recurrent neural network architectures and avoids complex variational autoencoders for representing latent-variable uncertainty.Source: §1 · p. 2
- The paper focuses on shallow relations between adjacent sentences or utterances, including contrast and question-answer relations.These relations are used in implicit discourse relation classification and dialog act classification.Source: §1 · p. 2
2 Background
The background describes recurrent neural language models and a document-context extension for incorporating information across sentences. The proposed model builds on recurrent word prediction while adding discourse information between adjacent sentences.
2 Background
- An RNN language model decomposes sentence probability across tokens, conditioning each word on preceding words through a recurrent summary vector.The hidden state is computed recurrently from the previous state and the preceding word’s embedding.Source: §2 · p. 2
- The paper uses Long Short-Term Memory units in place of simple hidden-state units in its RNN language model.The passage describes LSTM units as yielding stronger language-modeling performance.Source: §2 · p. 2
- The Document Context Language Model adds an inter-sentence context vector to the word-level context when generating each word.Its context vector is set to the last hidden state of the previous sentence; the cited work reports improved language-model perplexity.Source: §2 · pp. 2–3
- Figure 1 illustrates discourse information flow: the previous sentence affects the distribution of the relation variable, which then affects word prediction in the current sentence.Source: Figure 1 · p. 3
3 Discourse Relation Language Models
The model combines recurrent word generation with latent variables for discourse relations between adjacent sentences. It supports inference over relations or words, with training objectives chosen for discourse prediction or language modeling.
3.1 The Model
- The model generates each relation from the preceding sentence's context, then generates the next sentence conditioned on that relation and the preceding sentence.When relations are unobserved, they act as linguistically meaningful latent variables that can be predicted or marginalized.Source: §3.1 · p. 3
3.2 Inference
- Relation inference estimates each relation from its adjacent sentences, while discourse-informed language modeling marginalizes relations to compute sentence-sequence probability.Observed words decouple neighboring relations, allowing the marginal likelihood to be computed as a product of local marginals.Source: §3.2 · p. 4
3.3 Learning
- Joint training optimizes language modeling and relation prediction together, whereas conditional training prioritizes relation prediction and treats language modeling as auxiliary.The language-modeling term can dominate the joint objective because vocabularies and token counts are large.Source: §3.3 · pp. 4–5
3.4 Modeling limitations
- Discourse relations are decoupled after conditioning on words to avoid dynamic programming that would raise time complexity quadratically in the number of relations.An alternative that sends context into later hidden states obtains slightly better perplexity but couples a relation to all subsequent sentences, complicating inference.Source: §3.4 · p. 5
4 Data and Implementation
The experiments use the PDTB for implicit discourse relation classification and Switchboard for dialog act classification. The section describes corpus-specific settings and the shared model implementation and tuning procedure.
4 Data and Implementation
- PDTB experiments focus on challenging implicit relations between adjacent sentences, leaving explicit relations for future work and collapsing other relation types into a dummy class.The task uses multiway classification, for which the paper says document-level training makes balancing instances by relation type impossible.Source: §4 · pp. 5–6
The Switchboard Dialog Act Corpus (SWDA)
- Switchboard labels utterances with 42 speech acts across 1,155 telephone conversations; 1,115 were used for training and 19 for testing.Nineteen training conversations were randomly selected for development, and the standard split was adopted.Source: The Switchboard Dialog Act Corpus (SWDA) · p. 6
4.1 Implementation
- The models use a single-layer LSTM; word-representation dimension K and LSTM hidden dimension H are selected from {32, 48, 64, 96, 128} by development-set grid search for each corpus.Training uses norm clipping with threshold 5.0 and value dropout at rate 0.5 on input X, context vector c, and hidden state h. In the experiments, 4 to 5 epochs were enough.Source: §4.1 · p. 6
5 Experiments
The experiments evaluate discourse-relation prediction on the PDTB and Switchboard, and test whether discourse-aware language modeling improves perplexity. Across the classification tasks, conditionally trained DRLM outperforms alternatives; marginalizing relations at test time improves language modeling on both datasets.
5 Experiments
- The experiments assess discourse prediction on the PDTB and SWDA, and evaluate language modeling to test the effect of incorporating training-time discourse annotations.PDTB relation prediction is evaluated as four-way classification, following Rutherford and Xue (2015).Source: §5 · p. 6, §5.1 · p. 6
5.1 Implicit discourse relation prediction on the PDTB
- Conditionally trained DRLM outperforms alternatives on both PDTB relation-prediction metrics.The comparison includes the multiclass first-level relation-identification results in Table 1.Source: Table 1 · p. 7, §5.1 · p. 7
5.2 Dialogue Act tagging
- Conditionally trained DRLM outperforms competitive systems on Switchboard dialogue-act tagging.The result is significantly better than the previous state of the art; the evaluation reports accuracy because Macro-F1 scores are unavailable.Source: Table 2 · p. 7, §5.2 · p. 7
5.3 Discourse-aware language modeling
- Adding discourse-relation information further improves language-model perplexity on both PDTB and SWDA.Table 3 reports perplexities, with lower values better; test-time discourse relations are marginalized, so test annotations are not required.Source: §5.3 · pp. 7–8, Table 3 · p. 8
- The language-modeling approach requires discourse annotations for training and does not scale to the large datasets typically used for language modeling.The authors describe the resulting language-modeling results as somewhat academic from a practical perspective.Source: §5.3 · p. 8
6 Related Work
Related work spans discourse and dialogue modeling, language models that incorporate document context, and neural models with latent variables. The paper positions its approach as a probabilistic model over text with linguistically interpretable discourse variables.
Discourse and dialog modeling
- Earlier PDTB discourse classifiers use handcrafted features, while representation-learning approaches learn jointly with classifiers or parsers; these systems use the 16,000 annotated training relations, whereas this paper models the entire text probabilistically.Relevant neural approaches include Ji and Eisenstein (2015), Zhang et al. (2015), and Braud and Denis (2015).Source: §6 · p. 8
- For dialogue-act tagging, the paper contrasts its conditionally trained recurrent language model with HMMs using act-specific n-gram emissions and a combined convolutional-recurrent neural architecture.The authors characterize their framework as simpler, relying on a latent-variable parameterization of a purely recurrent architecture.Source: §6 · p. 8
Language modeling
- Prior document-context language models use topics, convolutional context vectors, or bag-of-words representations, while this paper models context and the current sentence in a unified framework.The paper identifies the Document Context Language Model (Ji et al., 2015) as the most relevant work and reports better held-out perplexity by adding discourse information.Source: §6 · p. 8
Latent variable neural networks
- Unlike prior latent-variable neural models with continuous variables and complex inference, this model uses discrete discourse variables that are easy to sum and maximize over.The variables carry linguistic interpretations and are at least partially observed.Source: §6 · p. 8
7 Conclusion
The conclusion presents a probabilistic neural model of words and shallow discourse relations between adjacent sequences. It reports gains on two discourse-relation tasks and application to language modeling with test-time marginalization over relations.
7 Conclusion
- The model combines discriminatively trained vector representations with a probabilistic representation of shallow discourse relations.The authors report that it outperforms state-of-the-art systems on two discourse-relation detection tasks and can marginalize relations at language-model test time.Source: §7 · p. 9