PaperlayerView exact revision on arXiv

Source-linked AI summary

Skip-Thought Vectors

Ryan Kiros, Yukun Zhu, Ruslan Salakhutdinov, Richard S. Zemel, Antonio Torralba, Raquel Urtasun, Sanja Fidler

arXiv:1506.06726v1cs.CLcs.LG

TL;DR

The paper addresses the need for generic sentence representations that are not tuned to a particular supervised task. It trains an encoder-decoder to predict surrounding sentences, adds vocabulary expansion for unseen words, and evaluates frozen vectors across eight tasks. Skip-thought vectors perform robustly across the tasks considered and capture sentence semantics and syntax, while the authors identify unexplored model and context variations.

  • Problem

    Existing sentence-composition methods learn representations through task labels and become tuned to their respective supervised tasks, motivating generic unsupervised sentence representations.

    Source: §1 · p. 1

  • Method

    Skip-thoughts encode a sentence to predict its surrounding sentences, use a corpus of contiguous books, and map external word vectors into the encoder vocabulary space for unseen words.

    Source: §1 · p. 1, §2.2 · p. 4

  • Results

    Skip-thought vectors used as frozen features with linear models perform robustly across all eight tasks considered and learn representations that capture sentence semantics and syntax.

    Source: §1 · p. 2, §4 · p. 9

  • Takeaways & Limitations

    The resulting encoder is an off-the-shelf sentence representation evaluated across diverse tasks without additional fine-tuning.

    Source: §1 · p. 2, §4 · p. 9

    The authors note that variations such as deeper encoders and decoders, larger context windows, paragraph encoding, and other encoders remain unexplored.

    Source: §4 · p. 9

Abstract

from arXiv · show

We describe an approach for unsupervised learning of a generic, distributed sentence encoder. Using the continuity of text from books, we train an encoder-decoder model that tries to reconstruct the surrounding sentences of an encoded passage. Sentences that share semantic and syntactic properties are thus mapped to similar vector representations. We next introduce a simple vocabulary expansion method to encode words that were not seen as part of training, allowing us to expand our vocabulary to a million words. After training our model, we extract and evaluate our vectors with linear models on 8 tasks: semantic relatedness, paraphrase detection, image-sentence ranking, question-type classification and 4 benchmark sentiment and subjectivity datasets. The end result is an off-the-shelf encoder that can produce highly generic sentence representations that are robust and perform well in practice. We will make our encoder publicly available.

1 Introduction

The paper proposes skip-thoughts, an unsupervised sentence encoder trained to predict surrounding sentences, and evaluates its vectors as generic features across multiple tasks. A vocabulary expansion method extends encoding to words absent from training, while experiments show robust performance and semantic and syntactic structure.

Motivation

  • Supervised composition models are tuned to particular task labels, whereas skip-thoughts learns sentence representations without a particular supervised task in mind.The paper frames this as an alternative to task-specific composition learning and paragraph vectors requiring test-time inference.Source: §1 · p. 1

Model

  • Skip-thoughts encodes a sentence and predicts the contiguous previous and next sentences, adapting the skip-gram objective from words to sentences.The approach can use any composition operator as the sentence encoder while modifying only the objective function.Source: §1 · p. 1

Training data

  • The model is trained on BookCorpus, a collection of novels spanning 16 genres and containing narrative, dialogue, emotion, and character interaction.The authors use this contiguous-text corpus to train the skip-thought models.Source: §1 · p. 1

Evaluation

  • Skip-thought vectors are evaluated with linear models as frozen generic features on eight tasks, including semantic relatedness, paraphrase detection, image-sentence ranking, and classification benchmarks.The encoder is used without additional fine-tuning, and the representations perform robustly across the tasks considered.Source: §1 · p. 2

Vocabulary expansion

  • A linear mapping from pre-trained word2vec space into the encoder vocabulary space enables encoding words absent from the training vocabulary.The mapping is learned with un-regularized L2 linear regression, allowing words from the larger word2vec vocabulary to be mapped into the RNN embedding space.Source: §1 · p. 2

2 Approach

Skip-thoughts frame sentence encoding as an encoder-decoder task: an encoder maps a sentence to a vector, and decoders generate its surrounding sentences. The approach uses gated recurrent computations, separate forward and backward decoders, and vocabulary expansion for unseen words.

Encoder-decoder framework

  • The model treats skip-thoughts as an encoder-decoder system in which an encoder maps words to a sentence vector and decoders generate surrounding sentences.Source: §2.1 · pp. 2–3

Encoder

  • The encoder iteratively produces hidden states representing progressively longer prefixes, with update and reset gates controlling recurrent state computation.Source: §2.1 · p. 3

Decoder

  • Separate neural language-model decoders generate the next and previous sentences, using sentence-vector biases in their gate and hidden-state computations.Source: §2.1 · p. 3

Objective

  • Training uses sentence tuples (s_i−1, s_i, s_i+1) and maximizes forward and backward sentence log-probabilities conditioned on the encoder representation.Source: §2.1 · pp. 2–4

Vocabulary expansion

  • Vocabulary expansion learns an unregularized L2 linear mapping from a larger word2vec space into the RNN encoder’s vocabulary space.Source: §2.2 · p. 4
  • The paper notes alternatives to vocabulary expansion, including initializing embeddings with pretrained vectors or training at the character level.Source: §2.2 · p. 4

3 Experiments

The experiments test frozen skip-thought encoders with linear models across semantic, paraphrase, image-sentence, and classification tasks, including comparisons with task-specific and bag-of-words methods. Results show strong generic representations, while task-specific tuning and a skip-thoughts–NB combination provide important advantages.

Experimental setup

  • The evaluation freezes skip-thought encoders, extracts sentence or pairwise features, and trains linear models without additional fine-tuning.Linear classifiers isolate representation quality and simplify reproducibility.Source: §3 · p. 4

Semantic relatedness

  • Skip-thought vectors outperform previous SemEval systems on SICK semantic relatedness and approach LSTM representations trained directly for the task.A dependency tree-LSTM performs better, while some model combinations reach parity with it.Source: §3.2 · pp. 5–6

Paraphrase detection

  • Skip-thoughts alone outperform recursive nets with dynamic pooling without hand-crafted features, while pairwise statistics make them competitive with more engineered state-of-the-art systems.When other features are included, recursive nets with dynamic pooling perform better.Source: §3.3 · pp. 6–7

Image-sentence ranking

  • Skip-thought vectors perform on par with strong image-sentence retrieval methods except for R@1 on image annotation, where other methods perform much better.The result suggests the frozen representations capture image descriptions without learning sentence representations from scratch.Source: §3.4 · p. 7

Classification benchmarks

  • On most classification tasks, skip-thoughts perform about as well as bag-of-words baselines but do not improve over task-specific sentence representations.The authors infer that tuning representations on small datasets can outperform generic unsupervised vectors trained on much larger data.Source: §3.5 · p. 8
  • The skip-thoughts-NB combination is particularly effective on MR and yields a strong text-classification baseline by combining skip-thoughts with bag-of-words features.The encoder also produces t-SNE embeddings in which similar SICK sentence pairs appear near one another without relatedness labels.Source: §3.5 · p. 8, §3.6 · p. 8

4 Conclusion

Skip-thought vectors perform robustly across all eight evaluated tasks as off-the-shelf representations, while the authors view the current objective as only an initial exploration of possible designs.

  • Skip-thought vectors perform well across all eight tasks when evaluated with linear classifiers as off-the-shelf sentence representations.The evaluation uses frozen representations without additional fine-tuning.Source: §4 · p. 9
  • The proposed model explores only part of the possible objective space for learning sentence representations.Suggested extensions include deeper encoders and decoders, larger context windows, paragraph-level encoding and decoding, and convolutional encoders.Source: §4 · p. 9
  • The authors suggest that exploring these objective variations could produce higher-quality sentence representations.Source: §4 · p. 9

Related papers

Loading 1506.06726v1…