Source-linked AI summary
Learning Word Vectors for 157 Languages
Edouard Grave, Piotr Bojanowski, Prakhar Gupta, Armand Joulin, Tomas Mikolov
TL;DR
The paper addresses the limited availability of high-quality pretrained word vectors across languages with small or insufficient corpora. It trains subword-aware models from Wikipedia and Common Crawl, introduces analogy datasets for French, Hindi, and Polish, and reports strong multilingual performance, while noting weaker quality for low-resource languages.
Problem
High-quality pretrained word-vector models are mostly available for English, while many languages lack sufficiently large corpora for broad coverage.
Method
The authors train subword-aware skipgram and position-weighted CBOW models using Wikipedia and Common Crawl data, and evaluate them with multilingual analogy datasets.
Results
The models achieve high quality across evaluated languages; Common Crawl improves models for languages with small Wikipedia, while performance is lower for low-resource languages such as Hindi.
Takeaways & Limitations
The released resources provide multilingual word vectors and evaluation datasets, with Common Crawl offering larger coverage and benefits for languages poorly represented in Wikipedia.
Abstract
from arXiv · showhide
Distributed word representations, or word vectors, have recently been applied to many tasks in natural language processing, leading to state-of-the-art performance. A key ingredient to the successful application of these representations is to train them on very large corpora, and use these pre-trained models in downstream tasks. In this paper, we describe how we trained such high quality word representations for 157 languages. We used two sources of data to train these models: the free online encyclopedia Wikipedia and data from the common crawl project. We also introduce three new word analogy datasets to evaluate these word vectors, for French, Hindi and Polish. Finally, we evaluate our pre-trained word vectors on 10 languages for which evaluation datasets exists, showing very strong performance compared to previous models.
1. Introduction
Word vectors are widely useful in NLP, but high-quality pretrained models remain disproportionately available for English because many languages lack sufficiently large training corpora.
- Pretrained models are standard tools for NLP applications, yet they are mostly available for English.
- Although Wikipedia offers high-quality comparable data across languages, many language editions are too small for broad-coverage word vectors.
- Prior releases include English models trained on 100B Google News tokens, GloVe models trained on 840B tokens, and Wikipedia models for 100 languages.
2. Training Data
The training pipeline combines curated Wikipedia with large-scale Common Crawl text, using language identification and deduplication to construct multilingual corpora.
- The authors train word vectors on mixtures of Wikipedia and Common Crawl data.
- Wikipedia provides curated, high-quality text in more than 200 languages, but many language editions are relatively small, including Hindi with 39 million tokens.
- A fastText language detector recognizes 176 languages and retains lines longer than 100 characters with confidence at least 0.8.
- The pipeline removes duplicate lines by hashing, an important cleanup step because crawl data contains substantial boilerplate.
3. Models
The paper compares two subword-aware word-vector models: fastText skipgram and a position-weighted CBOW extension.
- The training comparison includes two models: skipgram with subword information and CBOW with position weights and subword information.
- Skipgram: The fastText skipgram model represents each word using the sum of vectors for its character ngrams, while retaining a full-word vector.
- CBOW: The CBOW model predicts a target word from surrounding context words and represents that context using position-dependent vector weighting.
- CBOW: In position-weighted CBOW, each context word vector is element-wise multiplied by a position vector before the vectors are summed.
- CBOW: The word vectors used by CBOW are themselves sums over character ngram vectors.
4. Evaluations
The evaluation introduces analogy datasets for French, Hindi, and Polish, compares word-vector variants against a fastText baseline, and studies training-data effects across ten languages.
- 4.1. Evaluation Datasets: Word analogy datasets for French, Hindi, and Polish extend evaluation beyond the previously available language resources.The French, Hindi, and Polish datasets adapt English analogies with language-specific translations and category changes.
- 4.1. Evaluation Datasets: Models are evaluated on word analogies using the 200,000 most frequent training-data words, so Wikipedia and Wikipedia+Crawl models have different vocabularies.The evaluation covers ten languages, while vocabulary restriction affects exact model coverage.
- 4.2. Model Variants: All tested modifications improve average analogy accuracy over the default fastText skipgram baseline, with CBOW using position weights producing the largest overall gain.Character n-grams of length 5 preserve accuracy except for Czech, while using fewer n-grams can speed training.
- 4.3. Results: +23.5 for Finnish, +9.7 for Polish, +16.0 for Hindi, and +17.8 for Chinese are the reported improvements from crawl data for languages with small Wikipedia resources.For high-resource languages such as German, Spanish, and French, crawl data does not increase or slightly decreases analogy accuracy, although it provides larger coverage.
5. Conclusion
The paper releases multilingual word vectors, analogy datasets, and a language identifier, and reports that Common Crawl improves coverage and performance for languages with small Wikipedia resources. It also identifies lower quality for low-resource languages such as Hindi as an area for future improvement.
- 5. Conclusion: The work contributes word vectors trained on Wikipedia and Common Crawl, three analogy datasets, and a language identifier recognizing 176 languages.These resources support multilingual training and evaluation.
- 5. Conclusion: Common Crawl provides larger coverage and better models for languages with small Wikipedia resources, despite being noisy.The conclusion presents this as a central benefit of adding crawl data.
- 5. Conclusion: Word-vector quality is much lower for low-resource languages such as Hindi than for other languages.The paper proposes exploring additional techniques for these languages in future work.