Source-linked AI summary
Document Embedding with Paragraph Vectors
Andrew M. Dai, Christopher Olah, Quoc V. Le
TL;DR
The paper addresses limited evidence about Paragraph Vectors beyond sentiment analysis, including their comparison with LDA and sensitivity to embedding dimensionality. It benchmarks them on Wikipedia and arXiv document-similarity tasks, finding strong performance and useful semantic vector operations, while showing that joint word-vector training improves embedding quality.
Problem
The paper asks how Paragraph Vectors perform beyond sentiment analysis compared with Bag-of-Words and LDA, and how sensitive they are to hyperparameters.
Method
The paper benchmarks Paragraph Vectors and other document-embedding methods on Wikipedia and arXiv document-similarity tasks while varying embedding dimensions.
Results
Paragraph Vectors outperform LDA on Wikipedia, perform on par with LDA’s best topic setting on arXiv, and support useful semantic vector operations.
Takeaways & Limitations
Paragraph Vectors can support local and non-local corpus navigation, dataset exploration, book recommendation, and reviewer allocation.
Abstract
from arXiv · showhide
Paragraph Vectors has been recently proposed as an unsupervised method for learning distributed representations for pieces of texts. In their work, the authors showed that the method can learn an embedding of movie review texts which can be leveraged for sentiment analysis. That proof of concept, while encouraging, was rather narrow. Here we consider tasks other than sentiment analysis, provide a more thorough comparison of Paragraph Vectors to other document modelling algorithms such as Latent Dirichlet Allocation, and evaluate performance of the method as we vary the dimensionality of the learned representation. We benchmarked the models on two document similarity data sets, one from Wikipedia, one from arXiv. We observe that the Paragraph Vector method performs significantly better than other methods, and propose a simple improvement to enhance embedding quality. Somewhat surprisingly, we also show that much like word embeddings, vector operations on Paragraph Vectors can perform useful semantic results.
1 Introduction
The paper examines whether Paragraph Vectors generalize beyond sentiment analysis and how they compare with established document representations across unsupervised applications. It evaluates document similarity on Wikipedia and arXiv, studies representation dimensionality, and introduces joint word-embedding training as an improvement.
- 1 Introduction: The paper asks how Paragraph Vectors compare with Bag-of-Words and LDA in unsupervised applications beyond sentiment analysis.It also examines sensitivity to changes in hyperparameters.
- 1 Introduction: Two benchmarks evaluate document similarity for Wikipedia browsing and related-article finding on arXiv.The Wikipedia task identifies articles to browse next, while the arXiv task finds related papers.
- 1 Introduction: Paragraph Vectors support local and non-local browsing through simple vector operations, including finding the Japanese equivalence of “Lady Gaga.”The paper reports this as an example of semantic manipulation in document embeddings.
- 1 Introduction: Positive Wikipedia and arXiv results support using Paragraph Vectors for local and non-local browsing of large corpora.The paper frames these results as evidence that good text representations can support language understanding.
- 1 Introduction: Jointly training word embeddings, as in skip gram, improves the quality of paragraph vectors.The paper presents this as a simple improvement to Paragraph Vector.
2 Model
Paragraph Vector inserts a document-level memory vector into language modeling, using it with local context or without local context to predict words. The experiments focus on the more efficient Distributed Bag of Words implementation, tuning paragraph vectors at inference time.
- 2 Model: Distributed Memory adds a memory vector to a standard language model to capture document topics.The model is the original Paragraph Vector formulation described in the paper.
- 2 Model: The paragraph vector is concatenated or averaged with local context word vectors to predict the next word.The prediction task updates both the word vectors and the paragraph vector.
- 2 Model: Distributed Bag of Words removes local context from the prediction task.This simplification yields the more efficient implementation used in the experiments.
- 2 Model: At inference time, backpropagation tunes paragraph vectors without needing the classifier parameters or word vectors.The learned paragraph representation is therefore adapted for the inferred document.
3 Experiments
The experiments evaluate Paragraph Vectors and baselines on Wikipedia and arXiv document-similarity tasks, varying representation dimensionality and examining qualitative and quantitative retrieval. Paragraph Vectors generally outperform LDA on Wikipedia, match its best performance on arXiv, support semantic vector operations, and improve with jointly trained word embeddings.
- Wikipedia experiments: 4,490,000 Wikipedia articles were used to train paragraph vectors, whose t-SNE visualization grouped articles with the same category.Sports articles were less concentrated because Wikipedia contains a wide range of sport descriptions.
- Wikipedia experiments: Paragraph vectors produce better nearest neighbours than LDA for Wikipedia articles, including the qualitative Machine learning comparison.The methods use Hellinger distance for LDA and cosine distance for Paragraph Vectors.
- Wikipedia experiments: Vector operations support local and non-local Wikipedia browsing, including finding Japanese equivalents through pv("Lady Gaga") - wv("American") + wv("Japanese").The reported results indicate analogy-like behavior for paragraph vectors, paralleling word-vector operations.
- Experimental setup: The study evaluates paragraph vectors, LDA, bag of words, and averaged word embeddings on Wikipedia and arXiv document-similarity tasks while varying embedding dimensions.The experiments include qualitative nearest-neighbour and vector-operation analyses alongside triplet-based quantitative evaluation.
- Wikipedia experiments: Paragraph vectors perform better than LDA on Wikipedia, peak at 10,000 dimensions, and improve when word vectors are jointly trained.Averaged word embeddings also outperform LDA, while TF-IDF weighting does not affect LDA performance.
- arXiv experiments: Paragraph vectors perform on par with the best LDA topic count on arXiv, are less sensitive to embedding size, and peak at 100 dimensions.Both models outperform the vector space model; the arXiv evaluation uses 20,000 triplets from papers with shared versus unrelated subjects.
4 Discussion
Paragraph Vectors effectively measure semantic similarity between long texts, outperforming or matching LDA depending on the corpus and supporting vector-based semantic operations.
- Paragraph Vectors are superior to LDA for measuring semantic similarity on Wikipedia articles across all paragraph-vector sizes.
- On arXiv papers, Paragraph Vectors perform on par with LDA’s best-performing number of topics and remain consistent across embedding sizes.
- Vector operations on Paragraph Vectors can support local and nonlocal corpus navigation, dataset exploration, book recommendation, and reviewer allocation.