Source-linked AI summary
Distributed Representations of Sentences and Documents
Quoc V. Le, Tomas Mikolov
TL;DR
Bag-of-words representations lose word order and word semantics, motivating fixed-length representations for variable-length texts. Paragraph Vector learns dense paragraph representations by predicting words, and experiments report competitive or state-of-the-art performance across text tasks. A stated scope boundary is that some earlier sentence-level methods do not extend clearly to multi-sentence paragraphs or documents.
Problem
Bag-of-words representations lose word order and semantic relationships, while fixed-length text features are needed for machine-learning tasks.
Method
Paragraph Vector is an unsupervised method that learns representations for variable-length texts by training paragraph vectors to predict surrounding words.
Results
Paragraph Vector achieved competitive or state-of-the-art performance across text classification and sentiment-analysis tasks, outperforming bag-of-words models in reported experiments.
Takeaways & Limitations
Paragraph vectors can capture paragraph semantics and have the potential to overcome key weaknesses of bag-of-words models.
Takeaways & Limitations
Some earlier techniques are restricted to sentences because combining their representations across multiple sentences is unclear.
Abstract
from arXiv · showhide
Many machine learning algorithms require the input to be represented as a fixed-length feature vector. When it comes to texts, one of the most common fixed-length features is bag-of-words. Despite their popularity, bag-of-words features have two major weaknesses: they lose the ordering of the words and they also ignore semantics of the words. For example, "powerful," "strong" and "Paris" are equally distant. In this paper, we propose Paragraph Vector, an unsupervised algorithm that learns fixed-length feature representations from variable-length pieces of texts, such as sentences, paragraphs, and documents. Our algorithm represents each document by a dense vector which is trained to predict words in the document. Its construction gives our algorithm the potential to overcome the weaknesses of bag-of-words models. Empirical results show that Paragraph Vectors outperform bag-of-words models as well as other techniques for text representations. Finally, we achieve new state-of-the-art results on several text classification and sentiment analysis tasks.
1. Introduction
Bag-of-words representations are fixed-length and useful, but lose word order and semantic relationships. Paragraph Vector addresses these limitations by learning dense representations for variable-length texts through word prediction.
- Bag-of-words and bag-of-n-grams provide simple, efficient fixed-length text representations for classification and clustering.
- Bag-of-words loses word order, while bag-of-n-grams suffers from data sparsity and high dimensionality.
- Bag-of-words representations do not capture semantic distances: “powerful,” “strong,” and “Paris” are treated as equally distant.
- Paragraph Vector learns continuous distributed representations for variable-length texts ranging from phrases and sentences to documents.
- The paragraph representation is trained with word vectors to predict the next word from a paragraph context.
- Paragraph Vector outperformed bag-of-words and other text representations, including over 16% relative error-rate improvement in sentiment analysis and about 30% in text classification.
2. Algorithms
The paper develops word- and paragraph-vector frameworks that learn representations by predicting words from context. Paragraph vectors add paragraph-specific information, support memory and bag-of-words variants, and address semantic and ordering weaknesses of traditional representations.
- Learning word vectors: Word-vector models map vocabulary words to columns of W and use combined context vectors to predict the next word.
- Distributed Memory: In the paragraph-vector framework, each paragraph has a unique vector in D combined with word vectors to predict the next contextual word.
- Distributed Memory: The paragraph token supplies information missing from the local context and can function as a memory of the paragraph topic.
- Distributed Memory: PV-DM samples fixed-length sliding-window contexts, sharing word vectors across paragraphs while keeping paragraph vectors paragraph-specific.
- Distributed Bag of Words: PV-DBOW ignores context words and trains paragraph vectors to predict randomly sampled words from a window.
- Applications: Paragraph vectors can serve as features for logistic regression, support vector machines, or K-means, including when labeled data are limited.
- Advantages: Paragraph vectors address bag-of-words weaknesses by capturing word semantics and local word order without the very high dimensionality of bag-of-n-grams.
- Model variants: Combining PV-DM and PV-DBOW produced more consistent performance across the authors’ tasks than PV-DM alone.
3. Experiments
The experiments evaluate Paragraph Vector on sentiment analysis and information retrieval, including sentence- and document-length inputs. Across these tasks, the method outperforms several baselines while supporting representations for long documents without parsing.
- Experimental setup: Paragraph Vector is benchmarked on sentiment analysis and information retrieval, using datasets containing both single sentences and multi-sentence documents.The sentiment experiments use the Stanford Sentiment Treebank and IMDB datasets, while information retrieval tests whether documents match queries.
- Sentiment analysis: The Stanford Sentiment Treebank experiments label full sentences under coarse- and fine-grained classification settings, using learned representations with logistic regression.Training includes sentence and subphrase representations, while test-sentence vectors are inferred with word vectors fixed.
- Sentiment analysis: 2.4% absolute lower error, or 16% relative improvement, is achieved over the cited baselines on coarse-grained sentiment classification.The method outperforms recursive networks despite not requiring parsing.
- Beyond one sentence: Paragraph Vector addresses multi-sentence inputs because it does not require parsing and can represent long documents.This contrasts with techniques restricted to sentences because combining parsed representations across multiple sentences is unclear.
- Information retrieval: Paragraph Vector yields a 32% relative improvement in information-retrieval error rate over bag-of-words and bigram methods.The retrieval study compares bag-of-words, bag-of-bigrams, averaged word vectors, and Paragraph Vector using paragraph triplets derived from query results.
4. Related Work
Prior work developed distributed representations for words, phrases, and sentences, but existing sentence methods often required parsing, supervision, or did not clearly extend beyond single sentences. Paragraph Vector addresses these boundaries with mostly unsupervised representations applicable to variable-length texts.
- Distributed word representations became a successful paradigm in statistical language modeling and NLP applications.
- Phrase- and sentence-level approaches included weighted word averages, parse-tree compositions, and autoencoder-style paragraph models.
- Existing sentence methods typically required parsing, were demonstrated at sentence level, and were supervised.
- Paragraph Vector is mostly unsupervised and can therefore work well with less labeled data.
- Paragraph Vector computes paragraph representations through gradient descent, resembling the gradient-vector construction of Fisher kernels.
5. Discussion
Paragraph Vector learns representations for variable-length text by predicting surrounding words from sampled paragraph contexts. Experiments find it competitive with state-of-the-art methods and suggest applicability beyond text to sequential data.
- Paragraph Vector learns vector representations for variable-length texts, including sentences and documents, from surrounding words in sampled contexts.
- The method trains representations to predict surrounding words in contexts sampled from each paragraph.
- Experiments on Stanford Treebank and IMDB sentiment analysis datasets show performance competitive with state-of-the-art methods.
- The reported performance demonstrates merits in capturing paragraph semantics and potential to overcome weaknesses of bag-of-words models.
- The method can also learn representations for sequential data, including non-text domains where parsing is unavailable.