Source-linked AI summary
Massively Multilingual Word Embeddings
Waleed Ammar, George Mulcaire, Yulia Tsvetkov, Guillaume Lample, Chris Dyer, Noah A. Smith
TL;DR
Multilingual word embeddings need estimation and evaluation methods that work across many languages without depending on parallel data. The paper introduces dictionary-based multiCluster and multiCCA, proposes multiQVEC-CCA, and reports better correlations with downstream tasks than prior intrinsic metrics, while also providing an evaluation portal and method releases.
Problem
Multilingual NLP needs shared word embeddings and estimation methods covering many languages, while useful resources may consist of monolingual data and dictionaries rather than parallel corpora.
Method
The paper develops dictionary-based multiCluster and multiCCA estimators, adapts QVEC into multiQVEC, proposes multiQVEC-CCA, and provides a web portal for intrinsic and extrinsic evaluation.
Results
Both multiQVEC and multiQVEC-CCA obtain better correlations with downstream tasks than intrinsic methods previously used in the literature, while the estimators train embeddings for 59 languages.
Takeaways & Limitations
The methods, benchmarks, portal, and open-source releases provide resources for estimating and evaluating massively multilingual word embeddings.
Takeaways & Limitations
multiQVEC and multiQVEC-CCA depend on multilingual supersense annotations that may be inconsistent across languages or unavailable publicly.
Abstract
from arXiv · showhide
We introduce new methods for estimating and evaluating embeddings of words in more than fifty languages in a single shared embedding space. Our estimation methods, multiCluster and multiCCA, use dictionaries and monolingual data; they do not require parallel data. Our new evaluation method, multiQVEC-CCA, is shown to correlate better than previous ones with two downstream tasks (text categorization and parsing). We also describe a web portal for evaluation that will facilitate further research in this area, along with open-source releases of all our methods.
1 Introduction
The paper targets massively multilingual word embeddings to support multilingual NLP, introducing dictionary-based estimation methods, improved evaluation metrics, and an evaluation portal.
- Motivation: Massively multilingual word embeddings are proposed as a basis for future multilingual NLP, including cross-language transfer and handling words unseen in parallel data.Shared representations can support machine translation through vector-space neighbors and transfer models across languages.
- Contributions: multiCluster and multiCCA estimate multilingual embeddings from monolingual data and pairwise parallel dictionaries, training embeddings for 59 languages without requiring parallel corpora.Parallel corpora can still be used when available.
- Contributions: multiQVEC and multiQVEC-CCA evaluate multilingual embeddings, with both achieving better correlations with extrinsic tasks than previously used intrinsic metrics.multiQVEC-CCA addresses a theoretical shortcoming of multiQVEC.
- Contributions: An easy-to-use web portal evaluates arbitrary multilingual embeddings with intrinsic and extrinsic metrics, alongside released benchmarks and methods.The authors intend these resources to facilitate future research.
2 Estimating Multilingual Embeddings
The paper estimates multilingual embeddings by combining monolingual distributional data with bilingual dictionaries or parallel corpora, using multiCluster, multiCCA, and comparison baselines.
- Problem setup: The goal is a partial embedding function that places same-language semantic neighbors and cross-language translations nearby while covering many word types.Monolingual corpora model within-language similarity, while dictionaries or parallel corpora model cross-language similarity.
- Problem setup: Most prior multilingual-embedding work considered two languages, whereas this paper develops dictionary-based methods for more than two languages.The methods are multiCluster and multiCCA, evaluated alongside multiSkip and a translation-invariance matrix-factorization baseline.
- MultiCluster: multiCluster builds connected components of translation-linked language–surface-form pairs, replaces tokens with cluster IDs, concatenates monolingual corpora, and trains embeddings over the resulting sequences.Words in one multilingual cluster share an embedding, creating cross-language anchor points.
- Baselines: The multiSkip baseline learns bilingual embeddings from parallel corpora by combining monolingual context objectives with aligned-word contexts, and can extend across languages by summing bilingual objectives.Unlike the proposed dictionary-based methods, multiSkip is trained only on parallel corpora.
- MultiCCA: multiCCA first trains separate monolingual embeddings, then projects each non-English space into the English embedding space using dictionary-based canonical correlation analysis.English is selected as the shared space because it typically has the largest corpora and broad dictionary availability.
- Baselines: The translation-invariance baseline factorizes monolingual and transformed bilingual statistics jointly, taking the rows of its low-rank matrix U as multilingual embeddings.Its objective approximates the monolingual matrix and several transformations induced by bilingual alignments.
3 Evaluating Multilingual Embeddings
The paper streamlines multilingual-embedding evaluation with intrinsic metrics designed to capture linguistic quality, cross-lingual alignment, downstream usefulness, and computational efficiency. It extends QVEC to multiple languages and introduces multiQVEC-CCA to address QVEC’s basis dependence and dimensionality bias.
- Evaluation goals: A good multilingual-embedding evaluation should assess monolingual similarity, cross-lingual equivalence, downstream-task correlation, and computational efficiency.Coverage is assessed separately by counting words represented by an embedding function.
- QVEC: QVEC aligns embedding dimensions with linguistic-property dimensions to maximize cumulative Pearson correlation against a manually annotated resource.The alignment matrix restricts each distributional dimension to at most one linguistic dimension.
- QVEC limitations: QVEC is not invariant to linear basis transformations and produces higher unnormalized scores for embeddings with more dimensions.These weaknesses prevent reliable comparison across arbitrary bases and different embedding dimensionalities.
- QVEC-CCA: QVEC-CCA uses canonical correlation analysis to maximize correlations between projections of the embedding and linguistic matrices.This replaces cumulative dimension-wise correlation with correlations between learned basis projections.
- Extrinsic evaluation: Downstream evaluation uses embeddings as the sole dense word representation, with multilingual document classification and dependency parsing as task settings.The classifier covers seven languages, while parsing removes part-of-speech and morphology features and freezes the provided embeddings.
4 Evaluation Portal
The evaluation portal supports reproducible research on multilingual word embeddings by combining shared data resources with evaluation of uploaded embeddings. Researchers can select compatible tasks and obtain scores, while new datasets and metrics can be registered through the mirrored repository.
- Resources: The portal provides the monolingual and bilingual data used to estimate the paper’s multilingual embeddings.It also offers standard development and test datasets for the evaluation metrics.
- Evaluation workflow: Researchers can upload arbitrary multilingual embeddings, inspect language coverage, choose compatible evaluation tasks, and receive scores.Task availability depends on which languages the uploaded embeddings cover.
- Extensibility: New evaluation datasets and metrics can be registered through the GitHub repository that mirrors the portal backend.This extends the portal beyond the paper’s initial evaluation suite.
5 Experiments
The experiments test whether intrinsic metrics track downstream performance and compare multilingual embedding estimation methods across 59- and 12-language settings. The proposed metrics correlate better with downstream tasks, while method rankings vary by evaluation metric and language coverage.
- 5.1 Correlations between intrinsic vs. extrinsic evaluation metrics: Intrinsic metrics previously used in the literature correlate poorly with downstream tasks, whereas multiQVEC and multiQVEC-CCA correlate better with both tasks.The comparison uses Pearson correlations across 17 embeddings and six evaluation metrics.
- 5.2 Evaluating multilingual estimation methods: Intrinsic comparisons can be confounded by differing vocabulary coverage, because scores may use different test instances for different embeddings.The paper reports coverage and repeats evaluations on intersecting vocabularies when coverage differences are large.
- 5.2 Evaluating multilingual estimation methods: multiCCA consistently outperforms multiCluster across all evaluation metrics when multilingual embeddings are trained for 59 languages.Most coverage differences between the two methods are relatively small.
- 5.2 Evaluating multilingual estimation methods: In five evaluations, including both extrinsic tasks, a dictionary-based method proposed in this paper performs best.In the remaining four intrinsic evaluations, translation-invariance performs best.
- 5.2 Evaluating multilingual estimation methods: multiSkip ranks last in five evaluations and never ranks first, while its implementation does not use monolingual data.It learns only from monolingual contexts observed in parallel corpora.
6 Related Work
Prior multilingual-embedding research spans bilingual applications including translation, dependency parsing, and document classification. Earlier work also trained embeddings independently for more than 100 languages, without placing them in one shared vector space.
- Bilingual embeddings have been studied for machine translation, cross-lingual dependency parsing, and cross-lingual document classification.
- Al-Rfou’ et al. trained embeddings for more than 100 languages, but each language used an independent vector space.
7 Conclusion
The paper presents dictionary-based multilingual embedding methods, an improved evaluation metric, and an evaluation portal intended to support future research. Its released software context includes a prior library that supports bilingual rather than multilingual embeddings and portal evaluation across nine metrics.
- The proposed multiCCA and multiCluster methods were used to train multilingual embeddings for 59 languages.
- Both multiQVEC and multiQVEC-CCA correlate better with downstream tasks than previously used intrinsic methods.
- The bicvm software library had been extended to more than two languages, but its released library supported only bilingual embeddings.
- The evaluation portal evaluates uploaded multilingual embeddings on nine metrics with development and test modes.