Source-linked AI summary
A robust self-learning method for fully unsupervised cross-lingual mappings of word embeddings
Mikel Artetxe, Gorka Labaka, Eneko Agirre
TL;DR
Prior unsupervised mapping methods often fail in realistic settings involving distant languages or non-comparable corpora. This paper combines a structure-based unsupervised initialization with robust self-learning, and reports success across all tested scenarios, including results surpassing prior supervised mappings.
Problem
Previous unsupervised mapping methods were mainly evaluated under favorable conditions and often fail in realistic settings involving distant languages or non-comparable corpora.
Method
The method builds a seed-dictionary-free initialization from embedding structural similarity and combines it with robust iterative self-learning.
Results
The method succeeds in all tested cases and achieves the best results across prior unsupervised and supervised mappings.
Takeaways & Limitations
Fully unsupervised cross-lingual mapping can work in realistic settings and surpass previous supervised systems on the reported benchmarks.
Takeaways & Limitations
Self-learning does not work from a completely random initialization, and one prior system was excluded because of comparison and computational constraints.
Abstract
from arXiv · showhide
Recent work has managed to learn cross-lingual word embeddings without parallel data by mapping monolingual embeddings to a shared space through adversarial training. However, their evaluation has focused on favorable conditions, using comparable corpora or closely-related languages, and we show that they often fail in more realistic scenarios. This work proposes an alternative approach based on a fully unsupervised initialization that explicitly exploits the structural similarity of the embeddings, and a robust self-learning algorithm that iteratively improves this solution. Our method succeeds in all tested scenarios and obtains the best published results in standard datasets, even surpassing previous supervised systems. Our implementation is released as an open source project at https://github.com/artetxem/vecmap
1 Introduction
Existing unsupervised mapping methods were evaluated mainly in favorable settings and often fail with distant languages or non-comparable corpora. The paper proposes structural initialization plus robust self-learning to address these conditions.
- Cross-lingual mapping independently trains monolingual embeddings and aligns them in a shared space through a linear transformation.
- Adversarial methods often fail in realistic scenarios; on English-Finnish, all existing methods obtain below 2% translation accuracy.
- Self-learning can bootstrap mappings from as few as 25 word pairs but gets stuck in poor local optima when initialization is inadequate.
- The proposed method matches similarity distributions across languages to induce an initial dictionary, then robustly improves it through self-learning.Equivalent translations such as “two” and “due” have more similar distributions than unrelated words such as “two” and “cane.”
- The combined fully unsupervised method is reported to work in all tested realistic scenarios and surpass previous supervised systems.
2 Related work
Cross-lingual mapping research progressed from supervised dictionary-based methods to semi-supervised self-learning and fully unsupervised adversarial approaches. These lines differ in how they obtain supervision and optimize the shared representation.
- Standard mapping methods independently train embeddings and use linear transformations to place two languages in a shared space.
- Most earlier systems learn the mapping from bilingual dictionaries containing a few thousand entries, using regression, canonical, or orthogonal objectives.
- Semi-supervised self-learning alternates mapping and dictionary induction, achieving supervised-comparable results from only 25 word pairs.
- Seed dictionaries can be reduced using document-aligned corpora, shared words, cognates, or shared numerals.
- Adversarial approaches seek fully unsupervised mappings, with later systems adding noise, orthogonality, or iterative refinement to improve results.
3 Proposed method
The method normalizes embeddings, constructs an unsupervised structural initialization, improves it through robust self-learning, and applies final refinement. Its initialization exploits similarity-matrix structure rather than a seed dictionary.
- 3 Proposed method: The pipeline performs normalization, unsupervised initialization, robust self-learning, and final symmetric re-weighting.
- 3.1 Pre-processing: Length normalization and mean centering make dot products equivalent to cosine similarity and directly related to Euclidean distance.
- 3.2 Unsupervised initialization: Similarity matrices align words across both axes, so sorting each row and comparing rows can induce cross-lingual correspondences under isometry.The method uses normalized square roots of the similarity matrices to form alternative representations for initialization.
- 3.2 Unsupervised initialization: The initialization is weak but above chance, reaching 0.112 average cosine similarity and 0.52% dictionary accuracy in English-Italian.A random solution has 0.009 average cosine similarity, while the optimal supervised solution has 0.582.
- 3.3 Robust self-learning: Self-learning alternates orthogonal mapping optimization and dictionary induction until convergence.The learned mappings maximize current dictionary similarities, then retrieve nearest-neighbor dictionary entries in the mapped spaces.
- 3.3 Robust self-learning: Self-learning converges to a local optimum but fails from a completely random initialization, motivating the structural initialization.
- 3.3 Robust self-learning: Robustness comes from CSLS retrieval, bidirectional dictionary induction, stochastic exploration, and a 20,000-word frequency cutoff.CSLS uses k = 10 nearest neighbors to correct similarity scores for hubness; stochasticity helps escape poor local optima.
4 Experimental settings
The evaluation measures bilingual lexicon extraction across easier and more challenging language and corpus conditions, using repeated runs, accuracy, success counts, and runtime. It also documents dataset and comparison constraints.
- Bilingual lexicon extraction evaluates induced-dictionary accuracy against a gold standard.
- The main benchmark covers English-Italian, English-German, English-Finnish, and English-Spanish using 300-dimensional embeddings from several monolingual crawling corpora.
- Each method is run 10 times, reporting best and average accuracy, successful runs above 5%, and average runtime.
- The comparison benchmark uses 50-dimensional Wikipedia embeddings and is easier because its corpora are comparable.
- The experiments test public implementations, alternative hyperparameters, and both default and paper-reported settings for competing methods.
- Some language pairs are excluded because test dictionaries were unavailable or licensing restrictions applied.
- One prior system is omitted because it lacks one-to-one dictionaries, requires proprietary software, and has much higher computational cost.
5 Results and discussion
The proposed method is evaluated on challenging unsupervised mapping scenarios, state-of-the-art comparisons, and component ablations. It is robust across runs, competitive or superior to prior systems, and benefits from its initialization and self-learning components.
- Experimental setup: The experiments cover main results, comparisons with previous systems, and ablations of initialization, self-learning, and symmetric re-weighting.The ablation evaluates best and average accuracies, successful runs, and runtime across 10 runs.
- Main results: In the Zhang et al. dataset, the method performs at par with Conneau et al. for Spanish-English and Italian-English, while substantially outperforming it for Turkish-English.Zhang et al.'s method obtains the worst results by a large margin despite reproduced reported results.
- Main results: In the more challenging Dinu et al. dataset and extensions, the method obtains the best results in all metrics for all four tested language pairs.Zhang et al. fails in this scenario, while Conneau et al. reaches only 1.62% in its best English-Finnish run.
- Robustness and efficiency: The method converges to a good solution in every run and is the only system obtaining positive results for English-Finnish.It surpasses previous methods by at least 1-3 points in all but the easiest pairs and is not sensitive to difficult-to-tune hyperparameters.
- Robustness and efficiency: Its runtime adapts to task difficulty through a dynamic convergence criterion, taking 1.7 versus 0.6 minutes for Spanish-English and Turkish-English in one dataset.Reported times are also 12.9 versus 7.3 minutes for English-Finnish and English-German in the other dataset.
- Ablation test: The unsupervised initialization avoids the failure of self-learning with random initialization, while stochastic dictionary induction, CSLS, bidirectional induction, and symmetric re-weighting improve performance or robustness.Symmetric re-weighting improves accuracy by around 1-2 points without increasing execution time.
6 Conclusions
The method combines a weak unsupervised initialization with robust self-learning and refinement, succeeding across realistic scenarios and outperforming prior unsupervised and supervised mappings. The authors also identify extensions beyond bilingual word-level embeddings and provide an open-source implementation.
- The method combines an unsupervised initial solution exploiting embedding-space structure with robust self-learning and final symmetric re-weighting.The full pipeline includes preprocessing, initialization, iterative improvement, and refinement.
- The method succeeds in all tested cases and achieves the best results against previous unsupervised and supervised mapping methods.
- The implementation is released as an open-source project at https://github. com/artetxem/vecmap.
- The authors plan to extend the method from bilingual to multilingual settings and from words to longer phrases.