Discourse-Based Objectives for Fast Unsupervised Sentence Representation Learning

1 Introduction

• Sentence encoders construct distributed representations and underpin neural approaches to translation, summarization, and classification.They are commonly implemented as convolutional, recursive, or recurrent networks over words or characters.Source: §1 · p. 1
• Unsupervised pretraining with unlabeled data has shown success in improving encoder performance and extending methods to lower-resource settings.Earlier sentence-encoder successes largely used ample training data and fully supervised end-to-end training.Source: §1 · p. 1
• Discourse coherence depends on sentences being interpretable in context and relevant to a topic, linking sentence meaning with local context and world knowledge.The paper argues that recognizing coherence therefore requires understanding sentence meanings and relating them to relevant knowledge about the world.Source: §1 · p. 1
• Earlier coherence-based encoder methods used generative objectives that compute word likelihoods, making an epoch on a typical 76M-sentence corpus take weeks.The authors say this cost made further research difficult and scaling to all available unlabeled English text nearly impossible.Source: §1 · p. 2
• The paper proposes alternative objectives that exploit much of the same coherence information at greatly reduced cost.The proposed approach makes it possible to learn broadly useful sentence representations in hours.Source: §1 · p. 2

2 Related Work

• Skip Thought uses paragraph-level discourse for unsupervised sentence-encoder pretraining, while subsequent SDAE and related methods still generate full sentences at considerable computational cost.These methods include work by Kiros et al. (2015), Hill et al. (2016), Gan et al. (2016), and Ramachandran et al. (2016).Source: §2 · p. 2
• Logeswaran et al. (2016) learn paragraph sentence order and report positive sentence-similarity results, whereas FastSent uses a low-capacity CBOW model for greater speed.The passage says FastSent performs well on semantic textual similarity benchmarks, but its order insensitivity places an upper bound on more intensive extrinsic language-understanding tasks.Source: §2 · p. 2
• Other work models sentence-order recovery, sentence representations as language-model context, or discourse relations as latent variables for language modeling.Li and Hovy (2014) and Li and Jurafsky (2016) focus primarily on intrinsic evaluation rather than transfer; Ji et al. (2016) report language-modeling improvements.Source: §2 · p. 2

3 Discourse Inspired Objectives

• The three objectives capture different aspects of paragraph discourse coherence and can be combined to train a single encoder for broadly useful sentence representations.The objectives are designed for paragraphs extracted from unlabeled text.Source: §3 · p. 2
• ORDER swaps adjacent sentence pairs with probability 0.5 and trains a model to classify whether each pair was switched.The task uses the directional nature of many coherence relations to motivate recovery of original sentence order.Source: §3 · p. 2
• Some coherence relations are unordered, so discourse coherence alone cannot always recover the original sentence order.The passage gives the flowers/birds example as a case where the assumed relation does not establish direction.Source: §3 · p. 3
• NEXT selects the sentence immediately following the first three of a paragraph from five later candidates, relying on adjacent sentences being more coherent than distant ones.The passage notes that multiple candidates can be coherent, with an elaboration taking precedence over a progression in its example.Source: §3 · p. 3
• CONJUNCTION removes a listed conjunction phrase from a sentence pair and trains the model to recover its category.The phrase categories are used when the second sentence starts with a conjunction phrase that signals a coherence relation.Source: §3 · p. 3

Experimental Setup

• Training uses BookCorpus, Gutenberg, and Wikipedia; BookCorpus alone lacks sufficiently rich discourse structure, and combining sources precludes a strict apples-to-apples comparison.The extracted examples number 40M for ORDER, 1.4M for CONJUNCTION, and 4.1M for NEXT.Source: §4 · p. 3
• The experiments compare 1024D CBOW, 1024D GRU, and 512D BiGRU encoders, training each model for eight hours.All use FastText pretrained embeddings with a Highway transformation; the BiGRU sees BookCorpus examples at least once in that time.Source: §4 · pp. 3–4

Intrinsic and Qualitative Evaluation

• Intrinsic results indicate that BiGRU captures the needed sentence properties, CBOW beats the simple GRU, and ORDER and NEXT benefit from joint training with CONJUNCTION.The authors attribute CBOW's result likely to its faster speed and greater number of training epochs.Source: Table 4 · p. 4, §4 · p. 4
• Nearest neighbors for random seed sentences, selected from 400k held-out sentences, suggest that the trained BiGRU is especially sensitive to high-level syntactic structure.Source: Table 5 · p. 4, §4 · p. 4

Extrinsic Evaluation

• DiscSent is comparable to SDAE and Skip Thought with drastically shorter training, matches Logeswaran et al. (2016) on MSRP in one-sixth the time, and joint BiGRU substantially outperforms single models.The text classification results include training time.Source: Table 6 · p. 4, §4 · p. 4

• The paper introduces three unsupervised sentence-representation objectives inspired by discourse coherence.Source: §5 · p. 4
• The system trains from six to over 40 times faster than comparable methods while obtaining comparable results on external evaluation tasks.Source: §5 · p. 4
• The authors hope these tasks prompt research into neural discourse understanding and unsupervised learning for a broader range of language-understanding models.Source: §5 · p. 4