Source-linked AI summary
Advances in Pre-Training Distributed Word Representations
Tomas Mikolov, Edouard Grave, Piotr Bojanowski, Christian Puhrsch, Armand Joulin
TL;DR
Pre-trained word representations are important for NLP and machine learning, but training them requires extracting useful statistics from large corpora. The paper combines cbow with position weighting, phrase preprocessing, and subword features, producing publicly available models that achieve strong results across analogy, rare-word, question-answering, and classification evaluations.
Problem
Pre-trained representations are widely used, but their training must capture statistical information efficiently from large unlabeled corpora.
Method
The paper enriches cbow with position-dependent weighting, phrase representations, and subword information, evaluated across several benchmarks.
Results
87% accuracy on word analogy tasks improved to 88.5% with subword features, while the models also strongly outperformed GloVe on rare words and improved SQuAD accuracy.
Takeaways & Limitations
Training established algorithms on very large, deduplicated corpora and combining these techniques can provide further quality gains in freely available word and phrase vectors.
Abstract
from arXiv · showhide
Many Natural Language Processing applications nowadays rely on pre-trained word representations estimated from large text corpora such as news collections, Wikipedia and Web Crawl. In this paper, we show how to train high-quality word vector representations by using a combination of known tricks that are however rarely used together. The main result of our work is the new set of publicly available pre-trained models that outperform the current state of the art by a large margin on a number of tasks.
1. Introduction
Pre-trained word representations provide distributional information that supports NLP and machine-learning applications, especially when task data are limited. This work combines several underused training and preprocessing improvements and evaluates them across analogy, rare-word, and question-answering benchmarks.
- Pre-trained word representations are widely used as building blocks for NLP and machine-learning applications.They derive distributional information from statistics in large unlabeled text corpora.
- Their training must efficiently capture statistical information from large and varied text sources.
- The paper combines position-dependent features, phrase representations, and subword information that are rarely used together.
- Quality is measured on syntactic, semantic, and phrase-based analogies, the Rare Words dataset, and SQuAD question answering.
2. Model Description
The model description extends cbow by replacing impractical vocabulary-wide softmax with negative sampling and enriching context representations with positions, phrases, and subword information.
- Standard cbow model: Cbow learns word representations by predicting each word from the surrounding words in a symmetric context window.The objective maximizes the log-likelihood of words given their contexts.
- Standard cbow model: Negative sampling replaces the impractical vocabulary-wide softmax with binary classification against sampled negative words.The correct word is learned in contrast with a set of negative candidates.
- Standard cbow model: The standard context vector averages context-word vectors and uses a dot product to score the predicted word against that context.Words and predicted targets use different parameterizations.
- Position-dependent Weighting: Position-dependent weighting reweights context-word vectors by learned representations for their relative positions.This adds richer positional context without the computational cost of explicitly representing every word-position combination.
- Phrase representations: Phrase preprocessing selects n-grams with mutual information and merges them into single tokens through repeated passes.The authors repeat the process 5-6 times and retain some unigram occurrences through probabilistic merging.
- Subword information: Subword enrichment represents words with character n-gram vectors to capture internal structure useful for rare or misspelled words and morphologically rich languages.The implementation restricts n-grams to 3-6 characters and uses hashing to reduce memory demands.
3. Training Data
The models are trained on several publicly available, very large text sources, including Wikipedia, news, UMBC, Gigaword, and Common Crawl. Common Crawl processing uses language filtering and low-quality-data removal without complex normalization.
- Training uses English Wikipedia, Statmt News, UMBC, Gigaword, and Common Crawl text sources.The Wikipedia corpus uses the June 2017 English archive and contains more than 9 billion words.
- Common Crawl documents are filtered with a unigram language model to retain English text and discard low-quality data.
- The authors avoid complex data normalization and preprocessing so published vectors can be readily used, leaving normalization for later post-processing.
- Table 1 reports corpus sizes in billions of words after tokenization and sentence de-duplication.
4. Results
The fastText models were evaluated across analogy, rare-word, question-answering, and text-classification tasks, generally outperforming comparable GloVe models. Performance gains were associated with large corpora, sentence de-duplication, position-dependent weighting, and phrases.
- Ablations and training data: Sentence-level de-duplication, position-dependent weighting, and phrases significantly improved model quality on semantic and syntactic analogy datasets.These results used Common Crawl containing 630B words.
- Word and phrase analogies: 87% accuracy on word analogy tasks was reported as the best published result by a large margin, improving to 88.5% with sub-word features.The comparison was against existing GloVe models trained on comparable corpora.
- Rare-word and question-answering evaluation: The models reported very strong Rare Words performance, again outperforming GloVe models by a large margin.The Rare Words dataset was one of the standard benchmarks used for evaluation.
- Rare-word and question-answering evaluation: Replacing GloVe pre-trained vectors with fastText vectors in a SQuAD question-answering system produced a significant improvement in accuracy.The setup followed Chen et al. (2017).
- Word and phrase analogies: 88% accuracy on the phrase-based analogy dataset was achieved with the model trained on Common Crawl.The authors described this as a new state-of-the-art result.
- Text classification: The new fastText word vectors produced superior overall performance across MRPC, MR CR, SUBJ, MPQA, SST, and TREC text-classification tasks.The vectors initialized classifiers trained with the supervised fastText toolkit.
5. Discussion
The paper focuses on high-quality pre-trained word and phrase vectors, showing that large datasets and selected training improvements can raise quality. The resulting models are freely available to researchers and engineers.
- Findings: Training known algorithms on very large text datasets and combining selected tricks produced higher-quality pre-trained representations.The authors emphasize sentence de-duplication, phrase construction, position-dependent weights, and subword features.
- Training choices: Sentence de-duplication was found to be very important before training on large corpora such as Common Crawl.Phrase construction was performed as a preprocessing step.
- Training choices: Position-dependent weights and subword features added the final boost of accuracy to the CBOW architecture.The paper also identifies phrase construction as part of the improvement pipeline.
- Availability: The described pre-trained models are freely available through the fastText webpage for use in projects involving textual data.The release includes word and phrase vector representations.