Source-linked AI summary
An Empirical Evaluation of doc2vec with Practical Insights into Document Embedding Generation
Jey Han Lau, Timothy Baldwin
TL;DR
Doc2vec’s effectiveness and the relative merits of its dbow and dmpv variants remained uncertain because prior results were difficult to reproduce. This paper rigorously evaluates doc2vec against baselines and competing document-embedding methods across two tasks, finding robust performance with large external corpora and gains from pre-trained word embeddings.
Problem
Prior reports disagree about doc2vec’s effectiveness and dbow versus dmpv, while some results have been difficult to reproduce.
Method
The paper conducts a rigorous two-task evaluation comparing doc2vec with word-vector averaging, an n-gram baseline, and two competing document-embedding methods.
Results
Doc2vec performs robustly when trained on large external corpora, benefits from pre-trained word embeddings, and performs particularly strongly on longer documents.
Takeaways & Limitations
Dbow is better than dmpv, and the paper provides general-purpose hyper-parameter recommendations plus code and pre-trained models for replication and use.
Takeaways & Limitations
The evaluation trains doc2vec on development and test documents because the training is unsupervised, then relaxes this setup when using large external corpora.
Abstract
from arXiv · showhide
Recently, Le and Mikolov (2014) proposed doc2vec as an extension to word2vec (Mikolov et al., 2013a) to learn document-level embeddings. Despite promising results in the original paper, others have struggled to reproduce those results. This paper presents a rigorous empirical evaluation of doc2vec over two tasks. We compare doc2vec to two baselines and two state-of-the-art document embedding methodologies. We found that doc2vec performs robustly when using models trained on large external corpora, and can be further improved by using pre-trained word embeddings. We also provide recommendations on hyper-parameter settings for general purpose applications, and release source code to induce document embeddings using our trained doc2vec models.
1 Introduction
Doc2vec extends word2vec to learn embeddings for word sequences, but conflicting reports on its effectiveness and dbow versus dmpv motivate a rigorous evaluation. The paper tests its task performance, tunability, off-the-shelf use, and comparisons with baselines and competing methods.
- Background: Doc2vec extends word2vec by learning embeddings for word sequences at granularities ranging from n-grams to documents.The paper uses “document embedding” for a word-sequence embedding regardless of granularity.
- Motivation: Conflicting findings concern doc2vec’s effectiveness, the relative performance of dbow and dmpv, and its reproducibility on sentiment analysis.Reports also describe sub-par performance against vector averaging methods.
- Research questions: The evaluation asks whether doc2vec works across task settings, whether dbow or dmpv is better, and whether tuning or pre-trained embeddings improve it.It also examines whether doc2vec can function as an off-the-shelf model like word2vec.
- Contributions: Dbow is reported as superior to dmpv, while tuned doc2vec trained on large external corpora performs strongly against averaging, n-gram, and state-of-the-art document-embedding methods.The paper also reports particularly strong performance on longer documents.
- Contributions: The authors release source code for replicating the experiments and inducing document embeddings with their trained models.The release is intended to support practical use and replication.
2 Related Work
Related work establishes word2vec’s negative-sampling objective and its skip-gram and cbow variants, then describes doc2vec’s dbow and dmpv extensions. Other document-embedding approaches use recurrent prediction or paraphrase-based similarity objectives.
- word2vec: Word2vec maximizes the probability of a context word given an input word, with negative sampling rewarding positive dot products and penalizing sampled negative words.The formulation defines σ as the sigmoid, k as the number of negative samples, and Pn(w) as the noise distribution.
- word2vec: Skip-gram predicts context words from one input word, whereas cbow combines multiple input-word vectors to predict a context word.The number of context words is controlled by the window-size hyper-parameter.
- doc2vec: Doc2vec extends word2vec through dbow and dmpv approaches for learning document embeddings.Dbow replaces the input word with a document token, while dmpv adds a document token to multiple target words.
- doc2vec: Dbow ignores word order, while dmpv concatenates the document token with target-word vectors to predict a context word.This distinguishes dbow’s distributed-bag-of-words architecture from dmpv’s concatenated input.
- Other document embeddings: Skip-thought learns sentence vectors with an encoder-decoder recurrent network that predicts words in an adjacent sentence.Its encoder and decoder both use gated recurrent neural-network language models.
- Other document embeddings: Paraphrase-based embeddings optimize word representations and composition so paraphrase pairs receive similar sentence embeddings, with simple averaging performing best.The training data comes from large-scale paraphrase pairs in PPDB.
3 Evaluation Tasks
The paper evaluates doc2vec on duplicate question detection and semantic textual similarity, comparing it with word2vec, ngram, and other embedding methods. Results examine model variants, document length, supervised benchmarks, and hyper-parameter choices.
- Evaluation Tasks: Document embeddings are compared with word2vec mean vectors, an ngram distributional baseline, and other document-embedding methods.The experiments use dbow and dmpv for doc2vec, skip-gram and cbow for word2vec, and ngram features built from unigrams, bigrams, and trigrams.
- Evaluation Tasks: The evaluation covers duplicate question detection across 12 StackExchange subforums and semantic textual similarity across five domains.The question-duplication task uses classification data, while STS predicts similarity for sentence pairs.
- Forum Question Duplication: Doc2vec outperforms word2vec in all subforums except gis, while dbow performs better than or as well as dmpv in 9 of 12 subforums.Ngram has comparable performance in english and gis, and dmpv is substantially better than dbow in english.
- Semantic Textual Similarity: On STS, doc2vec performs better than word2vec, but the gap is smaller for shorter documents and dbow is only slightly stronger than dmpv.The unsupervised methods do not exceed the supervised DLS system overall, although doc2vec outperforms DLS on the belief domain.
- Optimal Hyper-parameter Settings: Sub-sampling high-frequency words is especially important, and dmpv generally requires about one order of magnitude more epochs than dbow to converge.The reported optimal settings are fairly consistent across tasks, with dbow favoring longer context windows than dmpv.
4 Training with Large External Corpora
Training on large external corpora makes doc2vec robust across tasks, while dbow generally outperforms competing baselines and can support off-the-shelf inference.
- Large-corpus training: The study trains doc2vec on large external corpora to test whether it scales beyond small in-domain collections.The external corpora are WIKI and AP-NEWS, containing approximately 35M and 25M documents, respectively.
- Baseline comparison: For external-corpus models, ngram outperforms skip-gram on Q-Dup, whereas skip-gram performs better on STS.
- Baseline comparison: doc2vec outperforms word2vec and ngram across almost all tasks, with a larger advantage on longer Q-Dup documents.On shorter STS documents, the gap is smaller, and word2vec matches doc2vec in some domains.
- External embeddings: GL-NEWS word2vec embeddings perform worse than WIKI and AP-NEWS embeddings despite being trained on a much larger corpus.
- Practical use: WIKI and AP-NEWS dbow models show similar performance, supporting their robustness for general-purpose applications.The paper releases code and trained models for inducing document embeddings using these corpora.
- Competitor comparison: dbow outperforms paragram-phrase on most Q-Dup subforums, while paragram-phrase performs better on STS.The results support a document-length distinction: vector averaging works better for shorter documents, whereas dbow handles longer documents better.
5 Improving doc2vec with Pre-trained Word Embeddings
The paper finds that dbow depends on meaningful word embeddings and improves when initialized with pre-trained vectors, consistently increasing performance without harm.
- Initialization: dbow does not learn word embeddings in its default configuration, instead using randomly initialized and fixed word vectors.
- Initialization: Performance degrades severely when dbow uses randomized word embeddings because optimizing document vectors toward important content words becomes more difficult.
- Mechanism: With learned word vectors, doc2vec positions document embeddings closer to content-word clusters and farther from function-word clusters.The paper illustrates this behavior using a two-dimensional t-SNE projection.
- Pre-trained embeddings: Initializing dbow with pre-trained WIKI and AP-NEWS skip-gram embeddings improves training on smaller in-domain collections.The experiment repeats the earlier in-domain evaluations with external word-vector initialization.
- Results: Across all tasks, pre-trained word embeddings increase performance and never harm dbow results.The number of epochs needed for optimal performance is also lower than before.
- Results: Pre-trained cbow and skip-gram initializations produce similar observations, indicating limited sensitivity to the word-embedding implementation.
6 Discussion
The qualitative comparison shows that doc2vec and word2vec represent documents differently: doc2vec emphasizes content words rather than averaging all word vectors equally.
- Qualitative comparison: The paper uses t-SNE to compare document and word embeddings induced by doc2vec’s dbow and word2vec’s skip-gram.The analysis uses a randomly selected STS document and reports the same general trend across a larger sentence set.
- Word2vec: word2vec represents a document as the centroid of its constituent word embeddings.
- Doc2vec: doc2vec’s document embedding is biased toward content words such as tech, costliest, and bangalore rather than function words.The paper attributes this to negative sampling, which frequently selects high-frequency function words as negatives.
7 Conclusion
The paper rigorously evaluates doc2vec on two tasks, compares it with baselines and competing methods, and derives practical recommendations for robust document-embedding use.
- Evaluation: Across two tasks, doc2vec performs well against word2vec averaging, ngram, and two competing document-embedding methodologies.
- Findings: dbow is generally better than dmpv, while doc2vec benefits from hyper-parameter recommendations, large external corpora, and pre-trained word embeddings.
- Resources: The authors release code and pre-trained models to support replication and off-the-shelf document-embedding induction.