Source-linked AI summary
Polyglot: Distributed Word Representations for Multilingual NLP
Rami Al-Rfou, Bryan Perozzi, Steven Skiena
TL;DR
Multilingual NLP systems face limited portability and heavy dependence on English-specific, hand-crafted features. The paper trains and releases word embeddings from Wikipedias in more than 100 languages, evaluates them for part-of-speech tagging, and examines their semantic groupings. The resulting resource reaches near state-of-the-art performance in the reported languages and captures interesting semantic relations, while leaving combined-feature evaluation and out-of-vocabulary handling for future work.
Problem
Multilingual NLP is difficult because systems rely heavily on language-specific features and have mainly been built and tested for English.
Method
The authors train word embeddings from Wikipedias in 117 languages using a neural architecture that learns from original and corrupted word sequences.
Results
The embeddings provide near state-of-the-art performance on a fundamental NLP task in the reported evaluations and express interesting semantic relations through word groupings.
Takeaways & Limitations
The publicly released multilingual embeddings give researchers a resource for developing tools and conducting comparative experiments across languages.
Takeaways & Limitations
The study evaluates embeddings as features alone, while combining them with traditional NLP features may improve task results.
Abstract
from arXiv · showhide
Distributed word representations (word embeddings) have recently contributed to competitive performance in language modeling and several NLP tasks. In this work, we train word embeddings for more than 100 languages using their corresponding Wikipedias. We quantitatively demonstrate the utility of our word embeddings by using them as the sole features for training a part of speech tagger for a subset of these languages. We find their performance to be competitive with near state-of-art methods in English, Danish and Swedish. Moreover, we investigate the semantic features captured by these embeddings through the proximity of word groupings. We will release these embeddings publicly to help researchers in the development and enhancement of multilingual applications.
1 Introduction
The paper addresses the difficulty of building portable multilingual NLP systems by releasing minimally normalized word embeddings for 117 languages and evaluating them across languages.
- Motivation: Multilingual NLP systems remain difficult to build because preprocessing and intermediate representations vary across languages, while research and tools focus largely on English and other rich-resource languages.Language-specific hand-engineered features also limit portability, enhancement, and maintenance.
- Motivation: Automatically learned word embeddings offer task-independent features derived from large amounts of plain text, avoiding reliance on expert-designed features.Prior work reported strong performance in language modeling and sequence tagging.
- Contributions: The authors generate word embeddings for 117 languages with more than 10,000 Wikipedia articles, using vocabularies of up to 100,000 words.The embeddings are intended to reduce computational and language-coverage barriers.
- Contributions: The released embeddings use minimal normalization, including preserving capitalization in European languages to retain underlying linguistic features.For example, capitalization distinguishes “Apple” from “apple” in the English model.
- Contributions: The work evaluates embeddings as features for part-of-speech tagging and examines their syntactic and semantic properties across many languages.The authors also provide efficient implementation improvements enabling training under different settings or on different corpora.
2 Related Work
Related work establishes distributed word representations as useful alternatives to conventional NLP features, while this paper extends the approach beyond English and preserves more linguistic variation.
- Prior representation learning: Semi-supervised and unsupervised feature-learning methods have improved language modeling, parsing, and other supervised NLP tasks using large unannotated corpora.Word clustering also supports semantic classes and cross-language knowledge transfer.
- Prior representation learning: Distributed word representations encode information about words and their usage but traditionally require substantial computational resources to train.Prior work accelerated training through architectural changes and error sampling.
- SENNA: SENNA showed that embeddings can nearly replace common NLP features across several tasks while achieving competitive performance and faster execution than comparable pipelines.Its embeddings were learned by distinguishing original phrases from corrupted versions rather than normalizing vocabulary probabilities.
- Novelty: This paper adopts a similar network architecture because of SENNA’s training speed and prior English-task performance, but trains embeddings for 117 languages.It also avoids excessive normalization, enabling evaluation without manufactured features, and releases the embeddings and generation resources.
- Open challenge: Embedding compositionality remains challenging for semantic tasks, motivating continued work on combining word representations into phrase meaning.Prior approaches addressed paraphrase detection and sentiment analysis.
3 Distributed Word Representation
The model learns word embeddings by ranking original phrases above corrupted alternatives: word vectors are concatenated, transformed by a hidden layer, and converted into phrase scores.
- Representation learning: Word embeddings map dictionary word indices to high-dimensional feature vectors whose dimensions jointly encode multiple concepts.The embeddings are learned by back-propagating task error from automatically generated examples in unlabeled data.
- Training objective: For each observed word sequence, the training procedure creates a corrupted sequence by randomly replacing its middle word and penalizes the model unless the original scores higher by a margin.The objective uses hinge loss over the corpus.
- Network architecture: The network retrieves vectors for a 2n + 1 word sequence from a shared matrix C and concatenates them into projection layer P.P then serves as input to a hidden layer with |H| units.
- Network architecture: Hidden activations are computed from the projection layer, and a linear combination of those activations produces the phrase score.The score is used to rank original and corrupted phrases.
- Model parameters: The learned parameters are W1, W2, b1, b2, and C, with total size approximately M ∗ (nH + |V |).The parameterization includes the embedding matrix and the hidden-layer weights and biases.
4 Corpus Preparation
The paper uses Wikipedia as a multilingual source, processing and normalizing its text before restricting vocabularies to frequent tokens. Corpus size and linguistic variation produce substantial differences in coverage across languages.
- Wikipedia provides large, varied, well-studied, openly accessible, and growing corpora for multilingual embedding training.
- 117 languages had more than 10,000 Wikipedia articles, while only five had more than one million.
- The pipeline extracts Wikipedia text, tokenizes it, and normalizes digits, internal hyphens, and brackets.
- The vocabulary retains the 100,000 most frequent tokens, replacing out-of-vocabulary words with ⟨UNK⟩.
- Significant Wikipedias generally exceed 92% word coverage, except German, Russian, Arabic, and Czech.
- Chinese achieves the highest word coverage, consistent with its relatively limited vocabulary size.
5 Training
The models use fixed context windows and compact neural architectures trained with minibatch stochastic gradient descent. Training and development errors decline over time without observed French-model overfitting.
- The model uses a context window of 5, vocabulary size 100,000, embedding size 64, and hidden-layer size 32.
- Training examples sweep windows over sentences, replacing unknown words and padding sentence boundaries with special tokens.
- French training and test errors show no observed overfitting after 23 days, while larger corpora produce smoother error curves.
- Minibatches contain 16 examples, with gradients estimated by stochastic gradient descent because exact calculation is prohibitive at corpus scale.
- Each language uses a learning rate of 0.1 and a 90%/5%/5% train-development-test split.
- Training and development errors decrease as the number of observed examples increases.
6 Qualitative Analysis
Nearest-neighbor analysis shows that multilingual embeddings capture syntactic and semantic structure. Examples include lexical groupings, historical name relations, morphology, compositionality, and culturally specific concepts.
- French and Spanish embeddings group expected colors and professions together.
- English neighbors capture historical renamings of Indian cities while preserving shared conceptual proximity.
- Arabic embeddings group diacritic variants, conjunction-attached forms, and a distinct morphological form for enumerating couples.
- German examples preserve compositional semantics for multi-unit words, while Russian examples group Soviet-related figures.
- Chinese neighbors include solar terms and traditional holidays sharing calendar dates, and Italian neighbors relate upper- and lowercase forms.
7 Sequence Tagging
The paper evaluates embeddings as the sole features for part-of-speech tagging across multiple languages and compares performance with established systems. It also examines vocabulary coverage, including effects from domain differences between Wikipedia and tagging data.
- The quantitative evaluation uses embeddings as the sole features for a well-studied part-of-speech tagging task.
- Table 4 reports known-word, out-of-vocabulary, and total test accuracy, comparing the model with TnT results.
- The tagging datasets come from a domain different from Wikipedia, which is reflected in the results.
- Embeddings benefited every considered language, with the greatest gains occurring for languages with fewer training examples.
- Vocabulary coverage is measured separately as raw token coverage and unique-word coverage after normalization.
8 Conclusion
Word embeddings are presented as a valuable resource, especially for resource-scarce languages, with near state-of-the-art performance on part-of-speech tagging and examples of semantic relations. The work evaluates embeddings as features and identifies broader multilingual uses and future improvements.
- Word embeddings are particularly valuable for resource-scarce languages.
- The embeddings provide an off-the-shelf route to near state-of-the-art performance on a fundamental NLP task.
- The embeddings express semantic relations that may support applications such as semantic compositionality.
- Using embeddings with traditional NLP features is identified as a way to significantly improve NLP task results.
- The evaluation compares accuracy from randomly initialized taggers with the authors’ results, finding embeddings generally helpful, especially with few training examples.Scores are the best found for each language; better-resourced languages could be trained longer before overfitting.
- Future work targets larger context windows, domain adaptivity, cross-lingual embedding mappings, and improved handling of out-of-vocabulary words for robust real-world performance.