Source-linked AI summary
Skip-Thought Vectors
Ryan Kiros, Yukun Zhu, Ruslan Salakhutdinov, Richard S. Zemel, Antonio Torralba, Raquel Urtasun, Sanja Fidler
TL;DR
The paper addresses the need for generic sentence representations that are not tuned to a particular supervised task. It trains an encoder-decoder to predict surrounding sentences, adds vocabulary expansion for unseen words, and evaluates frozen vectors across eight tasks. Skip-thought vectors perform robustly across the tasks considered and capture sentence semantics and syntax, while the authors identify unexplored model and context variations.
Problem
Existing sentence-composition methods learn representations through task labels and become tuned to their respective supervised tasks, motivating generic unsupervised sentence representations.
Method
Skip-thoughts encode a sentence to predict its surrounding sentences, use a corpus of contiguous books, and map external word vectors into the encoder vocabulary space for unseen words.
Results
Skip-thought vectors used as frozen features with linear models perform robustly across all eight tasks considered and learn representations that capture sentence semantics and syntax.
Takeaways & Limitations
The resulting encoder is an off-the-shelf sentence representation evaluated across diverse tasks without additional fine-tuning.
Takeaways & Limitations
The authors note that variations such as deeper encoders and decoders, larger context windows, paragraph encoding, and other encoders remain unexplored.
Abstract
from arXiv · showhide
We describe an approach for unsupervised learning of a generic, distributed sentence encoder. Using the continuity of text from books, we train an encoder-decoder model that tries to reconstruct the surrounding sentences of an encoded passage. Sentences that share semantic and syntactic properties are thus mapped to similar vector representations. We next introduce a simple vocabulary expansion method to encode words that were not seen as part of training, allowing us to expand our vocabulary to a million words. After training our model, we extract and evaluate our vectors with linear models on 8 tasks: semantic relatedness, paraphrase detection, image-sentence ranking, question-type classification and 4 benchmark sentiment and subjectivity datasets. The end result is an off-the-shelf encoder that can produce highly generic sentence representations that are robust and perform well in practice. We will make our encoder publicly available.
1 Introduction
The paper proposes skip-thoughts, an unsupervised sentence encoder trained to predict surrounding sentences, and evaluates its vectors as generic features across multiple tasks. A vocabulary expansion method extends encoding to words absent from training, while experiments show robust performance and semantic and syntactic structure.
- Motivation: Supervised composition models are tuned to particular task labels, whereas skip-thoughts learns sentence representations without a particular supervised task in mind.The paper frames this as an alternative to task-specific composition learning and paragraph vectors requiring test-time inference.
- Model: Skip-thoughts encodes a sentence and predicts the contiguous previous and next sentences, adapting the skip-gram objective from words to sentences.The approach can use any composition operator as the sentence encoder while modifying only the objective function.
- Training data: The model is trained on BookCorpus, a collection of novels spanning 16 genres and containing narrative, dialogue, emotion, and character interaction.The authors use this contiguous-text corpus to train the skip-thought models.
- Evaluation: Skip-thought vectors are evaluated with linear models as frozen generic features on eight tasks, including semantic relatedness, paraphrase detection, image-sentence ranking, and classification benchmarks.The encoder is used without additional fine-tuning, and the representations perform robustly across the tasks considered.
- Vocabulary expansion: A linear mapping from pre-trained word2vec space into the encoder vocabulary space enables encoding words absent from the training vocabulary.The mapping is learned with un-regularized L2 linear regression, allowing words from the larger word2vec vocabulary to be mapped into the RNN embedding space.
2 Approach
Skip-thoughts frame sentence encoding as an encoder-decoder task: an encoder maps a sentence to a vector, and decoders generate its surrounding sentences. The approach uses gated recurrent computations, separate forward and backward decoders, and vocabulary expansion for unseen words.
- Encoder-decoder framework: The model treats skip-thoughts as an encoder-decoder system in which an encoder maps words to a sentence vector and decoders generate surrounding sentences.
- Encoder: The encoder iteratively produces hidden states representing progressively longer prefixes, with update and reset gates controlling recurrent state computation.
- Decoder: Separate neural language-model decoders generate the next and previous sentences, using sentence-vector biases in their gate and hidden-state computations.
- Objective: Training uses sentence tuples (s_i−1, s_i, s_i+1) and maximizes forward and backward sentence log-probabilities conditioned on the encoder representation.
- Vocabulary expansion: Vocabulary expansion learns an unregularized L2 linear mapping from a larger word2vec space into the RNN encoder’s vocabulary space.
- Vocabulary expansion: The paper notes alternatives to vocabulary expansion, including initializing embeddings with pretrained vectors or training at the character level.
3 Experiments
The experiments test frozen skip-thought encoders with linear models across semantic, paraphrase, image-sentence, and classification tasks, including comparisons with task-specific and bag-of-words methods. Results show strong generic representations, while task-specific tuning and a skip-thoughts–NB combination provide important advantages.
- Experimental setup: The evaluation freezes skip-thought encoders, extracts sentence or pairwise features, and trains linear models without additional fine-tuning.Linear classifiers isolate representation quality and simplify reproducibility.
- Semantic relatedness: Skip-thought vectors outperform previous SemEval systems on SICK semantic relatedness and approach LSTM representations trained directly for the task.A dependency tree-LSTM performs better, while some model combinations reach parity with it.
- Paraphrase detection: Skip-thoughts alone outperform recursive nets with dynamic pooling without hand-crafted features, while pairwise statistics make them competitive with more engineered state-of-the-art systems.When other features are included, recursive nets with dynamic pooling perform better.
- Image-sentence ranking: Skip-thought vectors perform on par with strong image-sentence retrieval methods except for R@1 on image annotation, where other methods perform much better.The result suggests the frozen representations capture image descriptions without learning sentence representations from scratch.
- Classification benchmarks: On most classification tasks, skip-thoughts perform about as well as bag-of-words baselines but do not improve over task-specific sentence representations.The authors infer that tuning representations on small datasets can outperform generic unsupervised vectors trained on much larger data.
- Classification benchmarks: The skip-thoughts-NB combination is particularly effective on MR and yields a strong text-classification baseline by combining skip-thoughts with bag-of-words features.The encoder also produces t-SNE embeddings in which similar SICK sentence pairs appear near one another without relatedness labels.
4 Conclusion
Skip-thought vectors perform robustly across all eight evaluated tasks as off-the-shelf representations, while the authors view the current objective as only an initial exploration of possible designs.
- Skip-thought vectors perform well across all eight tasks when evaluated with linear classifiers as off-the-shelf sentence representations.The evaluation uses frozen representations without additional fine-tuning.
- The proposed model explores only part of the possible objective space for learning sentence representations.Suggested extensions include deeper encoders and decoders, larger context windows, paragraph-level encoding and decoding, and convolutional encoders.
- The authors suggest that exploring these objective variations could produce higher-quality sentence representations.