PaperlayerView exact revision on arXiv

Source-linked AI summary

Neural Net Models for Open-Domain Discourse Coherence

Jiwei Li, Dan Jurafsky

arXiv:1606.01545v3cs.CL

TL;DR

Prior coherence models address individual aspects and score existing sentences, leaving open the challenge of generating coherent discourse. This paper introduces discriminative neural classifiers and latent-variable generative models for discourse relationships across sentences. The generative models obtain the best result in a large open-domain setting, including full-paragraph sentence ordering, and the work marks an initial step toward generating coherent text from context.

  • Problem

    Existing coherence frameworks model individual aspects and focus on scoring existing sentences rather than generating coherent discourse.

    Source: §1 · p. 1

  • Method

    The paper introduces discriminative neural models and encoder-decoder generative models with latent variables that model discourse relationships across sentences.

    Source: §1 · p. 1

  • Results

    Generative models obtain the best result in a large open-domain setting, including reconstructing every sentence’s order in a paragraph, and the latent-variable model significantly improves generated-text coherence.

    Source: §1 · p. 1

  • Takeaways & Limitations

    The work marks an initial step toward generating coherent text given discourse contexts and learning latent discourse-level features for discourse-aware NLP.

    Source: §1 · p. 1, §5 · p. 9

    Discriminative models can score given text but cannot generate coherent text, while neighbor-conditioned SEQ2SEQ models lack longer-term discourse dependencies.

    Source: §2 · p. 2, §3.1 · p. 3

Abstract

from arXiv · show

Discourse coherence is strongly associated with text quality, making it important to natural language generation and understanding. Yet existing models of coherence focus on measuring individual aspects of coherence (lexical overlap, rhetorical structure, entity centering) in narrow domains. In this paper, we describe domain-independent neural models of discourse coherence that are capable of measuring multiple aspects of coherence in existing sentences and can maintain coherence while generating new sentences. We study both discriminative models that learn to distinguish coherent from incoherent discourse, and generative models that produce coherent text, including a novel neural latent-variable Markovian generative model that captures the latent discourse dependencies between sentences in a text. Our work achieves state-of-the-art performance on multiple coherence evaluations, and marks an initial step in generating coherent texts given discourse contexts.

1 Introduction

The paper introduces neural models for discourse coherence to address the limits of prior approaches, which model individual aspects and score existing text rather than generate coherent discourse. Its generative models perform best in a large open-domain setting, while its discriminative model excels on a standard domain-specific ordering task.

1 Introduction

  • Existing frameworks model individual aspects of coherence, while scoring existing sentences rather than generating coherent discourse for tasks such as abstractive summarization.These approaches include lexical cohesion, rhetorical relations, and entity focus; entity-grid extensions add coreference, named entities, discourse relations, and entity graphs.Source: §1 · p. 1
  • The discriminative models achieve 96% accuracy on the standard domain-specific sentence-pair-ordering dataset but suffer in a larger open-domain setting.The passage attributes the open-domain weakness to negative sampling covering only a small semantic space.Source: §1 · p. 1
  • The paper introduces discriminative and generative neural models for discourse coherence.The generative models augment encoder-decoder models with latent variables to model discourse relationships across sentences.Source: §1 · p. 1
  • The generative models obtain the best result in a large open-domain setting, including the difficult task of reconstructing every sentence’s order in a paragraph.The latent-variable generative model also significantly improves the coherence of text generated by the model.Source: §1 · p. 1
  • The work presents an initial step toward evaluating open-domain coherence and generating coherent text given discourse contexts.Source: §1 · p. 1

2 The Discriminative Model

The discriminative model learns from coherent sentence cliques and cliques made incoherent by replacing their center sentence. It represents sentences with LSTMs and classifies the concatenated clique representation, but negative sampling is limited in open-domain settings and the model cannot generate text.

2 The Discriminative Model

  • The model treats original-article cliques as coherent examples and cliques with randomly replaced center sentences as incoherent examples.It is described as an extension of Li and Hovy’s (2014) model that is practical at large scale.Source: §2 · p. 2
  • LSTMs represent each sentence in the clique, and a neural network maps the concatenated representations to the probability that the clique is coherent.Each clique contains 2L + 1 sentences, with each sentence represented by the LSTM output at its final time step.Source: §2 · p. 2
  • Negative sampling covers only a tiny proportion of possible open-domain negative candidates, and the discriminative model cannot generate coherent text.It can instead support tasks such as extractive summarization by reordering sentences; the passage contrasts this with dialogue generation and abstractive summarization.Source: §2 · p. 2
  • Li and Hovy’s (2014) recursive neural model operates on parse trees, which do not support batched computation and are hard to scale.Source: §2 · p. 2

3 The Generative Model

This section develops generative coherence models that inject discourse-level information into sentence generation. It moves from neighboring-sentence SEQ2SEQ models toward topic-based and learned Markovian latent states.

3.1 Model 1: the SEQ2SEQ Model and its Variations

  • Bidirectional coherence scoring combines separate forward and backward conditional models, while language-model correction focuses the score on context information.The corrected score has the same form as mutual information between adjacent sentences.Source: §3.1 · pp. 2–3
  • SEQ2SEQ predicts a sentence from its neighboring context, but its coherence scores require a completed sentence and may be used to rerank generated N-best lists.The forward score predicts the next sentence and is normalized by its length.Source: §3.1 · pp. 2–3
  • SEQ2SEQ generation loses global discourse information as its hidden state evolves, and conditioning on only one neighboring sentence misses longer-range dependencies.The proposed remedies provide a global meaning signal at each word-generation step.Source: §3.1 · p. 3

3.2 HMM-LDA based Generative Models (HMM-LDA-GM)

  • The HMM-LDA generative model draws each sentence’s topic based on the preceding sentence’s topic rather than a document-level topic distribution.Its implementation assigns all words in a sentence to the same topic.Source: §3.2 · p. 3
  • HMM-LDA-GM converts topic distributions into discourse-state vectors that guide every decoding step and model Markov dependencies across sentences.The topic representation is computed from topic vectors and incorporated into decoder hidden-state updates.Source: §3.2 · pp. 3–4
  • Topic models capture topical coherence but their coarse-grained topics do not capture all finer-grained discourse relationships.The paper attributes this coarseness to topic models’ focus on document-level word co-occurrences and the limited number of topic clusters.Source: §3.2 · p. 4

3.3 Variational Latent Variable Generative Models (VLV-GM)

  • VLV-GM learns each sentence’s stochastic discourse state from data, conditioning it on prior sentences and the preceding latent state rather than using an upstream LDA model.The latent state is used alongside the LSTM representation to predict each word.Source: §3.3 · p. 4
  • Because the true posterior over latent states is intractable, VLV-GM uses a variational approximate posterior and maximizes a lower bound on the data.The objective combines posterior approximation with sentence-token prediction using the latent state.Source: §3.3 · pp. 4–5

4 Experimental Results

The experiments evaluate sentence ordering, paragraph reconstruction, and coherent text generation. Results vary by setting: discriminative models excel on domain-specific data, while generative models lead on open-domain ordering and VLV-GM performs best in reconstruction and generation evaluations.

4.1 Sentence Ordering, Domain-specific Data

  • 93% accuracy on the accident dataset; the extended discriminative model also achieves almost perfect earthquake accuracy and significantly beats recursive and non-neural baselines.Generative models do not perform competitively on this domain-specific dataset; the authors conjecture that the small dataset causes overfitting.Source: §4.1 · p. 6

4.2.1 Binary Permutation Classification

  • Generative models significantly outperform all baselines on open-domain binary classification; the discriminative model is only slightly better than the entity grid.The authors attribute the discriminative model’s weaker open-domain results to sentence-level negative sampling covering only a small part of the semantic space.Source: §4.2.1 · p. 6, Table 2 · p. 6

4.2.2 Paragraph Reconstruction

  • Given permuted sentences, generative models reconstruct paragraph order; VLV-GM performs best, evaluated with Kendall’s τ to preserve relative-order coherence.The task starts with a bag of constituent sentences; the first sentence is given, and beam search selects subsequent sentences.Source: §4.2.2 · p. 7, Table 3 · p. 7

4.3 Adversarial evaluation on Text Generation Quality

  • 0.081 is the best adversarial success on adver-3; VLV-GM is most human-like overall, but the evaluator distinguishes every model with over 90% accuracy.Performance declines as turns increase, which the authors associate with accumulating errors and difficulty modeling long-term sentence dependencies.Source: Table 4 · p. 7, §4.3 · p. 8

4.4 Qualitative Analysis

  • In miniature discourses, scores favor temporal and causal ordering in simple cases, while rough shifts lower coherence and suggest sensitivity to entity focus.The authors caution that these qualitative evaluations are suggestive; understanding the discourse embedding space likely requires more sophisticated visualizations.Source: §4.4 · p. 8

5 Conclusion

The conclusion reports that the traditional small-domain pair-ordering task is completely solvable by the discriminative models and recommends harder open-domain full-paragraph ordering. It presents Markov generative models as an initial step toward generating coherent text from context and learning latent discourse features.

5 Conclusion

  • The paper investigates open-domain discourse coherence with discriminative models trained on coherent texts versus permutations and Markov generative models that predict sentences from neighbors.Source: §5 · pp. 8–9
  • The traditional small-domain sentence-pair-ordering task is completely solvable by the discriminative models, motivating open-domain full-paragraph sentence ordering.Source: §5 · p. 9
  • The latent-variable neural models offer a new way to learn latent discourse-level features, suggesting directions for discourse-aware NLP tasks.Source: §5 · p. 9
  • The proposed models are an initial step toward generating coherent texts given contexts, with potential benefits for NLP generation tasks.Source: §5 · p. 9

6 Supplemental Material

The supplemental material describes the domain-specific dataset and training setup, document-level scoring, evaluation and inference procedures, and qualitative inspection of generated-model coherence scores.

6 Supplemental Material

  • The domain-specific corpus contains 200 articles each from airplane accident and earthquake reports, with original-order versus random-permutation document pairs.Source: §6 · p. 11
  • For the domain-specific comparison, reduced models are used; the discriminative model uses fixed 300-dimensional GloVe embeddings, while the generative model is a one-layer seq2seq model with 100-dimensional words and hidden units.Source: §6 · p. 11
  • Document coherence is scored by splitting documents into consecutive-sentence cliques and averaging their scores.Source: §6 · p. 12
  • Kendall’s τ is computed from ranking inversions; N is the original document’s sentence count, and inversions count consecutive-element interchanges needed to recover its order.Source: §6 · p. 12
  • The variational-inference derivation computes the KL divergence between approximate and true latent-variable distributions, then backpropagates the resulting error through the hierarchical neural models.Source: §6 · p. 12
  • The qualitative analysis examines examples with MMI generative-model log-likelihood coherence scores to inspect the kinds of coherence represented.Source: §6 · p. 12

Related papers

Loading 1606.01545v3…