Source-linked AI summary
Unsupervised Multilingual Word Embeddings
Xilun Chen, Claire Cardie
TL;DR
Prior unsupervised multilingual embedding methods rely on independent bilingual models and do not fully exploit relations among languages. This paper combines multilingual adversarial training with multilingual pseudo-supervised refinement, and reports better results than unsupervised and supervised approaches on multilingual translation and cross-lingual similarity.
Problem
Method
MAT+MPSR uses monolingual embeddings and multilingual objectives to map languages into a shared space while exploiting relations between language pairs.
Source: §1 · p. 2, §3.1 · p. 4, §3.2 · p. 5Results
Takeaways & Limitations
The framework exploits relations among language pairs at the computational cost of learning O(N) rather than O(N^2) bilingual models.
Source: §3 · p. 3The encoders are assumed to be orthogonal linear mappings, based on prior findings about cross-lingual vector spaces.
Source: §3 · p. 3
Abstract
from arXiv · showhide
Multilingual Word Embeddings (MWEs) represent words from multiple languages in a single distributional vector space. Unsupervised MWE (UMWE) methods acquire multilingual embeddings without cross-lingual supervision, which is a significant advantage over traditional supervised approaches and opens many new possibilities for low-resource languages. Prior art for learning UMWEs, however, merely relies on a number of independently trained Unsupervised Bilingual Word Embeddings (UBWEs) to obtain multilingual embeddings. These methods fail to leverage the interdependencies that exist among many languages. To address this shortcoming, we propose a fully unsupervised framework for learning MWEs that directly exploits the relations between all language pairs. Our model substantially outperforms previous approaches in the experiments on multilingual word translation and cross-lingual word similarity. In addition, our model even beats supervised approaches trained with cross-lingual resources.
1 Introduction
The paper presents an unsupervised approach to multilingual embeddings that uses relations among languages rather than mapping them independently to one target language. It reports stronger performance than unsupervised and supervised alternatives while remaining as efficient as the pivot approach.
1 Introduction
- Unsupervised multilingual embeddings map words from multiple languages into one shared space, where words with similar meanings are close together.Such representations have been found beneficial for cross-lingual and monolingual NLP tasks.Source: §1 · p. 1
- Prior unsupervised multilingual methods independently map each language into a chosen target language, leaving interactions between source languages unused.A fixed pivot can degrade quality when languages such as French and Italian are each mapped through a less similar language such as English.Source: §1 · pp. 1–2
- Training direct bilingual models for every pair costs O(N^2) models rather than BWE-Pivot’s O(N), and each model still uses only its own language pair.Information from other related languages, such as Spanish when aligning French and Italian, may be useful.Source: §1 · p. 2
- The proposed method is as efficient as BWE-Pivot and outperforms BWE-Pivot, BWE-Direct, and state-of-the-art supervised methods in the reported experiments.The experiments cover multilingual word translation and cross-lingual word similarity.Source: §1 · p. 2
- The proposed fully unsupervised algorithm uses monolingual corpora and exploits interdependencies between language pairs to learn multilingual embeddings.Its two stages are Multilingual Adversarial Training and Multilingual Pseudo-Supervised Refinement.Source: §1 · p. 2
2 Related Work
Prior cross-lingual representation methods often require parallel corpora or bilingual lexica, while efforts to reduce supervision largely address individual language pairs. For unsupervised multilingual embeddings, prior methods build on independent bilingual models, whereas this work extends adversarial training and refinement to multiple languages.
2 Related Work
- Traditional bilingual embedding training uses parallel corpora or bilingual lexica, and reduced-supervision approaches still focus on one language pair.Fully unsupervised bilingual embedding methods are also described in prior work.Source: §2 · p. 2
- Fully unsupervised multilingual methods have relied on independent bilingual models to map languages into a chosen target space, rather than exploiting relations among languages.The cited examples include BWE-Pivot and BWE-Direct.Source: §2 · p. 2
- Prior adversarial methods address tasks including unsupervised bilingual embeddings, but consider only two languages; MAT generalizes this approach to multiple languages.The cited applications also include cross-lingual text classification and unsupervised machine translation.Source: §2 · p. 2
- Bilingual refinement can use iterative Procrustes with pseudo-supervision, but multilingual mappings lack a closed-form solution, motivating gradient-based MPSR.The passage describes the progression from weak supervision to fully unsupervised bilingual refinement.Source: §2 · p. 2
3 Model
The model maps monolingual embeddings from multiple languages into a shared space using orthogonal linear transformations. It combines multilingual adversarial training with pseudo-supervised refinement, while scaling linearly with the number of languages.
3 Model
- For N languages, orthogonal linear encoders map monolingual spaces into shared target T, while decoders return vectors to their source spaces; the framework exploits arbitrary language pairs at O(N) mapping cost.Choosing an existing language as the target yields an equivalent model with N−1 mappings.Source: §3 · pp. 2–3
3.1 Multilingual Adversarial Training
- One discriminator per language judges genuine versus converted vectors, while mappings try to fool the destination discriminator, explicitly using relations between all language pairs.MAT also permits encoding a language to the target space and back to itself; its training updates each discriminator or mapping at least once per step and scales linearly with the number of languages.Source: §3.1 · p. 4
3.2 Multilingual Pseudo-Supervised Refinement
- MPSR refines MAT mappings with pseudo-supervised lexica for every language pair, built from mutual nearest neighbors among each language’s 15,000 most frequent words.CSLS replaces cosine similarity when finding neighbors to address hubness; the method uses n = 10 nearest neighbors.Source: §3.2 · p. 5
3.3 Orthogonalization
- After each training step, an orthogonalization update keeps the mappings approximately orthogonal.The update uses β = 0.001.Source: §3.3 · p. 5
3.4 Unsupervised Multilingual Validation
- For unsupervised model selection, the validation criterion macro-averages mean CSLS over language pairs, with equal pair weights in this work.The criterion is adapted from a bilingual measure based on mean CSLS similarities between frequent words and their nearest-neighbor translations; different pair weights can be used.Source: §3.4 · pp. 5–6
4 Experiments
The experiments evaluate MAT+MPSR on multilingual word translation and cross-lingual word similarity against unsupervised and supervised approaches. It performs strongly across both tasks, with translation gains over supervised methods and similarity results competitive with leading supervised systems.
4.1 Multilingual Word Translation
- 1.3% improvement over Sup-BWE-Direct, despite no cross-lingual supervision; MAT+MPSR also achieves the highest performance on all but three language pairs.The translation table reports precision@1 in percentage.Source: §4.1 · p. 6, Table 1 · p. 7
- On Romance language pairs, MAT+MPSR achieves the best performance through explicit multilingual learning without increasing computational cost, overcoming BWE-Pivot’s disadvantage.BWE-Pivot performs notably worse than BWE-Direct on these pairs when English is a less similar pivot language.Source: §4.1 · pp. 6–7
- BWE-Pivot sometimes outperforms BWE-Direct on de-it, it-de, and pt-de, potentially because German is similar to the English pivot.The authors suggest that a different pivot could benefit other language pairs.Source: §4.1 · p. 7
4.2 Cross-Lingual Word Similarity
- On cross-lingual word similarity, MAT+MPSR leads the unsupervised baselines on all language pairs and performs similarly overall to Luminoso while outperforming NASARI.The table reports Spearman’s ρ; the performance margin is less significant than in word translation, which the authors relate to the task’s greater leniency.Source: Table 2 · p. 8, §4.2 · p. 8
- For pairs involving Farsi, all unsupervised methods outperform the supervised SemEval-2017 systems, and MAT+MPSR has a higher margin over unsupervised baselines on low-resource pairs.Farsi is not covered by the Europarl bitext available to supervised methods on European languages.Source: §4.2 · p. 8
5 Conclusion
The conclusion presents MAT+MPSR as a fully unsupervised multilingual embedding method that uses relations across language pairs and outperforms prior unsupervised and supervised systems on two evaluation tasks. It identifies instability in the underlying BWE framework on some languages as a direction for future adaptation.
5 Conclusion
- MAT+MPSR learns multilingual word embeddings from monolingual corpora by explicitly exploiting relations between all language pairs without increasing computational cost.Prior unsupervised multilingual approaches relied on separate unsupervised bilingual models and did not fully use interdependencies among languages.Source: §5 · p. 8
- On multilingual word translation and SemEval-2017 cross-lingual word similarity, MAT+MPSR outperforms existing unsupervised and supervised models and achieves new state-of-the-art performance.Source: §5 · p. 8
- The method’s reliance on Lample et al.’s BWE framework leaves instability on some languages, including Finnish, as a limitation motivating future adaptation to more robust methods.The authors propose exploring other BWE frameworks.Source: §5 · p. 9