Source-linked AI summary
Efficient Vector Representation for Documents through Corruption
Minmin Chen
TL;DR
Document representations must capture text semantics, while existing approaches such as Paragraph Vectors can require corpus-sized parameters and expensive inference for unseen documents. Doc2VecC trains word embeddings so their average represents a document, with corruption favoring informative words. It matches or outperforms state-of-the-art document representations on sentiment analysis, document classification, and semantic relatedness tasks.
Problem
Bag-of-Words treats words as discrete symbols, limiting its ability to capture similarity and producing sparse, high-dimensional representations; Paragraph Vectors also face corpus-sized models and costly unseen-document inference.
Source: §1 · p. 1Method
Doc2VecC trains word embeddings so a document’s representation is their average, while corruption introduces data-dependent regularization that favors rare or informative words.
Source: §1 · pp. 1–2Results
Doc2VecC matches or outperforms state-of-the-art document representations on sentiment analysis, document classification, and semantic relatedness tasks.
Source: §1 · p. 2, §5 · p. 10Takeaways & Limitations
Documents, including unseen ones, can be represented efficiently by averaging learned word embeddings, and the simple architecture supports training on billions of words per hour on one machine.
Source: §5 · p. 10Paragraph Vectors’ model complexity grows with training-corpus size, and representing unseen documents requires costly inference by adding document vectors and optimizing them.
Source: §2 · p. 3
Abstract
from arXiv · showhide
We present an efficient document representation learning framework, Document Vector through Corruption (Doc2VecC). Doc2VecC represents each document as a simple average of word embeddings. It ensures a representation generated as such captures the semantic meanings of the document during learning. A corruption model is included, which introduces a data-dependent regularization that favors informative or rare words while forcing the embeddings of common and non-discriminative ones to be close to zero. Doc2VecC produces significantly better word embeddings than Word2Vec. We compare Doc2VecC with several state-of-the-art document representation learning algorithms. The simple model architecture introduced by Doc2VecC matches or out-performs the state-of-the-art in generating high-quality document representations for sentiment analysis, document classification as well as semantic relatedness tasks. The simplicity of the model enables training on billions of words per hour on a single machine. At the same time, the model is very efficient in generating representations of unseen documents at test time.
1 INTRODUCTION
Document representations must capture text semantics, but Bag-of-Words is sparse and cannot represent similarity between discrete word or phrase symbols. Doc2VecC instead trains word embeddings so their simple average forms a meaningful document representation, using corruption to favor informative words.
1 INTRODUCTION
- Bag-of-Words is widely used and can work well, but its discrete symbols fail to capture word or phrase similarity and yield sparse, high-dimensional representations.Source: §1 · p. 1
- Paragraph Vectors outperform BoW and LDA on text-understanding tasks, but require corpus-sized parameters and expensive inference for unseen documents.Source: §1 · p. 1
- Doc2VecC represents documents by averaging word embeddings and trains them so this average captures document semantics.Unlike approaches that post-process learned embeddings, the model includes this representation during learning.Source: §1 · pp. 1–2
- A corruption model randomly removes document words during learning, a mechanism described as critical to Doc2VecC’s performance and learning speed.Source: §1 · p. 2
- The corruption-based framework favors rare or informative words and suppresses common, non-discriminative words through data-dependent regularization.Source: §1 · p. 2
- Doc2VecC’s model complexity depends on vocabulary size rather than training-corpus size, and documents can be represented by averaging learned embeddings.Its architecture resembles Word2Vec and is described as efficient to train and test.Source: §1 · p. 2
2 RELATED WORKS AND NOTATIONS
The section surveys document-representation approaches and focuses on Word2Vec and Paragraph Vectors as the methods most similar to Doc2VecC. It contrasts their architectures and motivates averaging word embeddings, while noting Paragraph Vectors’ corpus-dependent complexity and costly inference for unseen documents.
2 RELATED WORKS AND NOTATIONS
- Text representations span BoW and term-frequency variants, language and topic models, denoising autoencoders, distributed vectors, and task-specific CNN or LSTM approaches.Source: §2 · p. 2
- The section focuses on Word2Vec and Paragraph Vectors, using CBoW; extending the discussion to Skipgram is described as straightforward.Source: §2 · p. 2
- The notation defines a training corpus and vocabulary, document and local-context BoW vectors, and input and output projection matrices.The input matrix maps from input space to a hidden space; the output matrix maps from hidden space to output.Source: §2 · p. 2
- Word2Vec learns to predict a target word from local context using input and output projection matrices, while word embeddings can be averaged to form document vectors.Doc2VecC similarly averages embeddings, but encodes compositionality during learning rather than relying on heuristic test-time weighting.Source: §2 · pp. 2–3
- Paragraph Vectors combines word embeddings with a unique document vector that acts as a memory of the document’s global topic.Source: §2 · p. 3
- Paragraph Vectors’ complexity grows with both vocabulary and corpus size, and representing unseen documents requires adding document-vector parameters and optimizing them.Source: §2 · p. 3
3 METHOD
Doc2VecC predicts target words using local context and a global document vector formed by averaging word embeddings. Random corruption of the document during learning both reduces updates and induces data-dependent regularization.
3 METHOD
- Randomly removing a significant portion of document words reduces backpropagation updates and introduces data-dependent regularization.At test time, the document representation is the average of the learned embeddings of its words.Source: §3 · pp. 3–4
- Each update creates an unbiased mask-out/drop-out global context by zeroing each document dimension with probability q and scaling surviving dimensions by 1/(1 − q).This is the corruption process used to generate the global context.Source: §3 · p. 4
- Doc2VecC approximates the target-word probability with negative sampling because computing it exactly is impractical.The probability conditions on local context and the corrupted global context.Source: §3 · p. 4
3.1 CORRUPTION AS DATA-DEPENDENT REGULARIZATION
- Averaged word embeddings form a document vector that captures global semantics and fills information missed by the local context.The model trains target-word prediction using both the local context and this global vector.Source: §3 · p. 3, §3.1 · p. 5
- The regularizer penalizes common-word embeddings more, while reducing the penalty when an active word is critical to a confident prediction.Its strength is modulated by σ_w,c,x(1 − σ_w,c,x).Source: §3.1 · p. 5
4 EXPERIMENTS
Doc2VecC is evaluated against document-representation baselines on sentiment analysis, word analogy, Wikipedia classification, and semantic relatedness. Across these tasks, it performs strongly, with outcomes varying by task and sequence length.
4 EXPERIMENTS
- The experiments assess Doc2VecC on sentiment analysis, document classification, and semantic relatedness alongside other representation-learning algorithms.The sentiment setup includes both representation learning with all available data and learning with training and unlabeled data only.Source: §4 · p. 5, §4.2 · p. 6
4.1 BASELINES
- The experiments compare document-representation methods including BoW, denoising autoencoders, Word2Vec+IDF, Doc2Vec, skip-thought vectors, and RNNLM.LSTM-based methods are additionally compared on semantic relatedness.Source: §4.1 · p. 5
4.2 SENTIMENT ANALYSIS
- Doc2VecC achieves similar or better sentiment-classification performance than Paragraph Vectors and beats BoW by 15%.Including test data during representation learning benefits all representation-learning algorithms; the reported comparison uses a linear SVM.Source: §4.2 · p. 6, Table 1 · p. 6
- Doc2VecC’s learning and representation generation are faster than Paragraph Vectors’, which requires inference for unseen documents.Paragraph Vectors takes 4 minutes 17 seconds to infer representations for 25,000 test documents, versus 7 seconds for the other methods.Source: §4.2 · pp. 6–7, Table 2 · p. 7
4.3 WORD ANALOGY
- Doc2VecC significantly outperforms Word2Vec on word embeddings, with close to 20% uplift when trained on the full corpus.On the 14 subtasks with 100-dimensional embeddings trained on the full dataset, Doc2VecC significantly outperforms Word2Vec across almost all subtasks.Source: §4.3 · p. 8
4.4 DOCUMENT CLASSIFICATION
- Across representation sizes, Doc2VecC significantly reduces Wikipedia classification error and can match or outperform existing algorithms with smaller vectors.The comparison uses a linear SVM; most algorithms are not sensitive to representation size, while Doc2Vec benefits most from increasing it.Source: §4.4 · p. 9, Table 5 · p. 9
- In the t-SNE visualization, Doc2VecC groups documents by coarse category; musicians appear closer to albums than to athletes or species.The projection covers seven manually grouped categories, and the athletes group is less concentrated because it includes varied sports descriptions.Source: §4.4 · pp. 9–10, Figure 4 · p. 9
4.5 SEMANTIC RELATEDNESS
- Doc2VecC slightly underperforms LSTM-based methods and skip-thought vectors on SICK, while outperforming competition winners and several baselines.The authors report that Doc2VecC is much faster to train and test than skip-thought vectors; training on the book corpus took under two hours versus two weeks on GPU.Source: §4.5 · p. 10
5 CONCLUSION
Doc2VecC produces document vectors by averaging word embeddings and uses corruption-based regularization to favor informative words. The paper reports greater testing efficiency and more expressive representations than several existing document-representation algorithms.
5 CONCLUSION
- Doc2VecC outperforms several existing document-representation algorithms in testing efficiency and the expressiveness of generated representations.Table 6 reports test-set results for the SICK semantic-relatedness task, including SemEval submissions, baseline methods, LSTM-based methods, and skip-thought vectors.Source: §5 · p. 10, Table 6 · p. 11
- Doc2VecC represents each document as an average of learned word embeddings, with semantic capture enforced during learning.Source: §5 · p. 10
- Its data-dependent regularization favors informative or rare words while dampening embeddings of common, non-discriminative words.Documents can then be represented efficiently by averaging the learned word embeddings.Source: §5 · p. 10