Source-linked AI summary
Word Translation Without Parallel Data
Alexis Conneau, Guillaume Lample, Marc'Aurelio Ranzato, Ludovic Denoyer, Hervé Jégou
TL;DR
Cross-lingual embedding methods traditionally required bilingual dictionaries or parallel corpora, while character-based alternatives remained limited in performance and language coverage. This paper aligns monolingual embedding spaces without cross-lingual supervision through adversarial mapping, synthetic-dictionary refinement, and hubness-aware similarity scaling. It reaches or surpasses supervised approaches in several tasks and language pairs, including distant languages and low-resource English-Esperanto.
Problem
Existing methods either require parallel data or underperform supervised approaches, and character-based methods are limited to languages sharing an alphabet.
Method
The method adversarially learns a linear mapping between monolingual spaces, refines it with synthetic anchors and Procrustes optimization, and uses cross-domain similarity scaling.
Results
The approach reaches or outperforms supervised methods across three cross-lingual tasks and several language pairs, including English-Russian, English-Chinese, and English-Esperanto.
Takeaways & Limitations
Cross-lingual word embeddings and bilingual dictionaries can be learned from unaligned monolingual data, including for distant and low-resource language pairs.
Takeaways & Limitations
The method's cross-domain similarity alternatives have limitations: existing approaches update source and target similarities differently, and ISF requires noisy unsupervised hyper-parameter cross-validation.
Abstract
from arXiv · showhide
State-of-the-art methods for learning cross-lingual word embeddings have relied on bilingual dictionaries or parallel corpora. Recent studies showed that the need for parallel data supervision can be alleviated with character-level information. While these methods showed encouraging results, they are not on par with their supervised counterparts and are limited to pairs of languages sharing a common alphabet. In this work, we show that we can build a bilingual dictionary between two languages without using any parallel corpora, by aligning monolingual word embedding spaces in an unsupervised way. Without using any character information, our model even outperforms existing supervised methods on cross-lingual tasks for some language pairs. Our experiments demonstrate that our method works very well also for distant language pairs, like English-Russian or English-Chinese. We finally describe experiments on the English-Esperanto low-resource language pair, on which there only exists a limited amount of parallel data, to show the potential impact of our method in fully unsupervised machine translation. Our code, embeddings and dictionaries are publicly available.
1 INTRODUCTION
Existing unsupervised or weakly supervised cross-lingual embedding methods either underperform supervised systems or depend on parallel signals and shared alphabets. This paper aligns monolingual spaces without cross-lingual annotated data, using adversarial training, synthetic dictionary refinement, and unsupervised model selection.
- Prior methods either fail to reach competitive performance or require aligned corpora or a seed parallel lexicon.
- The proposed model uses only two large monolingual corpora and learns a source-to-target linear mapping through adversarial training.A discriminator distinguishes mapped source embeddings from target embeddings while the mapping is trained to fool it.
- The method extracts a synthetic dictionary and fine-tunes the mapping with closed-form Procrustes optimization.
- 66.2% accuracy on English-Italian exceeded the best supervised approach at 63.7% on a standard word-translation benchmark.The comparison used 200k-word vocabularies.
- The approach reaches or outperforms supervised methods across word translation, sentence translation retrieval, and cross-lingual word similarity.
- Cross-domain similarity adaptation mitigates hubness and improves word-translation performance in both supervised and unsupervised settings.The method adapts self-tuning similarity scaling to a bipartite graph spanning two language domains.
2 MODEL
The model learns a cross-lingual mapping without parallel supervision by adversarially aligning independently trained embedding spaces, refining the mapping with Procrustes, and using CSLS for translation. Its design addresses difficulties from rare words and hubness through frequency-based anchors and a bi-partite neighborhood adjustment.
- 2 MODEL: The final translation selects the target word with highest cosine similarity to the mapped source embedding.The mapping and distance metric are applied after adversarial alignment and refinement.
- 2.1 DOMAIN-ADVERSARIAL SETTING: The approach learns a linear mapping between independently trained source and target embeddings without cross-lingual supervision.Adversarial training makes mapped source embeddings and target embeddings as similar as possible.
- 2.1 DOMAIN-ADVERSARIAL SETTING: Adversarial training aligns embedding distributions by training a discriminator against a mapping that prevents accurate language-origin predictions.The discriminator distinguishes mapped source embeddings from target embeddings, while the mapping is trained to fool it.
- 2.2 REFINEMENT: The initial adversarial mapping is refined with Procrustes using frequent mutual nearest neighbors as a synthetic parallel vocabulary.Rare words are harder to align because their embeddings receive fewer updates and may reflect different corpus contexts.
- 2.3 CROSS-DOMAIN SIMILARITY LOCAL SCALING (CSLS): CSLS adjusts cross-lingual similarity using neighborhoods from a bi-partite graph, increasing similarity for isolated vectors and decreasing it in dense regions.This targets hubness, where some vectors become nearest neighbors of many points.
3 TRAINING AND ARCHITECTURAL CHOICES
Training uses independently trained 300-dimensional fastText embeddings, an adversarial discriminator, and orthogonal mapping updates. Model selection relies on an unsupervised criterion based on CSLS-generated translations rather than parallel-data validation.
- Embedding preparation: 300-dimensional fastText embeddings are lower-cased, filtered by frequency, and restricted to the 200k most frequent words.The embeddings are trained on Wikipedia corpora, and words appearing fewer than five times are discarded.
- Adversarial training: The discriminator is a two-hidden-layer multilayer perceptron, while both discriminator and mapping optimization use stochastic gradient descent.The discriminator has hidden layers of size 2048, dropout noise at rate 0.1, smoothing coefficient s = 0.2, batch size 32, and learning rate 0.1.
- Adversarial training: Rare words are excluded from discriminator inputs because they slightly degrade performance, so training uses the 50,000 most frequent words sampled uniformly.Sampling by word frequency did not noticeably change the results.
- Mapping constraint: An orthogonal mapping preserves monolingual embedding geometry and made training more stable in the authors’ experiments.Orthogonal transformations preserve dot products and Euclidean distances; the implementation keeps the matrix near the orthogonal manifold after each update.
- Unsupervised model selection: Because parallel-data validation is unavailable, model selection uses average cosine similarity between CSLS-generated translations of the 10k most frequent source words.The criterion is reported to correlate better with evaluation performance than Wasserstein distance and can select the model at epoch 10 in the illustrated experiment.
4 EXPERIMENTS
The experiments evaluate unsupervised cross-lingual embeddings across word translation, sentence retrieval, word similarity, and low-resource English–Esperanto translation. Results show strong gains from CSLS, adversarial alignment, refinement, and Wikipedia-trained embeddings, including competitiveness with supervised methods.
- Evaluation tasks: The evaluation covers word translation, sentence translation retrieval, cross-lingual word similarity, and English–Esperanto word-by-word translation.Word translation uses precision@k; sentence retrieval also reports precision@k, while word similarity reports Pearson correlation.
- Similarity metrics: CSLS provides robust gains across all language pairs, reaching up to 7.2% over nearest-neighbor similarity in en-eo.It is also computationally faster than ISF and does not require hyper-parameter tuning.
- Word translation: 44.9% accuracy on English-Italian word translation makes Procrustes-CSLS outperform all previous approaches.The comparison uses 1.5k source queries and 200k target words.
- Monolingual embeddings: Wikipedia-trained fastText embeddings produce an 18.8% word-translation gain over prior CBOW settings, with only 2% attributable to fastText over same-corpus CBOW.The authors associate most of the gain with changing the training corpora to Wikipedia.
- Unsupervised alignment: Adversarial alignment works without parallel data, while refinement closes the supervised gap and can reach 83% source-word translation accuracy.The refined unsupervised method outperforms the supervised baseline on en-it and en-es, and sentence retrieval gains reach up to 15%.
- Low-resource translation: On English–Esperanto, unsupervised P@1 is 28.2% versus 29.3% supervised in English–Esperanto, and 25.6% versus 24.3% in the reverse direction.With CSLS, naive word-by-word translation obtains 11.1 and 14.3 BLEU respectively.
5 RELATED WORK
Prior work aligned monolingual embedding spaces or vocabularies using distributional similarities, structural constraints, and limited supervision. Unsupervised adversarial approaches remained weaker than supervised methods and faced model-selection limitations.
- Earlier bilingual lexicon methods aligned discrete word representations using statistical similarities between languages.
- Cross-lingual embeddings extended this goal from vocabulary alignment to applications including bilingual lexicon extraction, sentence retrieval, and document classification.
- Orthogonality constraints improved linear mappings and yielded a closed-form Procrustes solution.
- Hubness corrections included nearest-neighbor reciprocity and inverted-softmax reweighting of similarities involving frequently retrieved words.
- Character-string and digit-based seed dictionaries reduced supervision but remained applicable mainly when languages shared alphabets or symbols.
- Prior unsupervised adversarial methods had weak supervised comparisons, unreliable model-selection criteria, and limited vocabulary sizes.One approach selected models using discriminator-accuracy drops that did not correlate with overall performance or permit hyperparameter tuning.
6 CONCLUSION
The paper aligns embedding spaces using only unaligned monolingual data, then refines the resulting synthetic dictionary with supervised-style optimization. It reports competitive or superior performance across language pairs, including Spanish-English and English-Esperanto.
- The method aligns word embedding spaces without cross-lingual supervision, using only unaligned datasets from each language.
- Adversarial training initializes a linear mapping and produces a synthetic parallel dictionary for subsequent Procrustean optimization.
- An unsupervised validation criterion and CSLS similarity measure are identified as key ingredients, with CSLS mitigating hubness and improving word translation accuracy.
- 83.3% Spanish-English word translation accuracy is reported, on par with supervised approaches.
- The method also works on English-Esperanto, indicating applicability to a low-resource language pair.
7 APPENDIX
The appendix examines how corpus and embedding-method differences affect English-to-English alignment. It also presents fully unsupervised Esperanto-English translations whose meanings are preserved but could be improved with a language model.
- The experiment merges English embedding spaces trained with Wikipedia or Gigaword and with skip-gram, CBOW, or fastText.
- Alignment is perfect across different random seeds on the same corpus, while differing corpora make retrieval harder, especially for rare words.
- Using both different embedding models and different corpora further deteriorates alignment performance.
- The Esperanto-English examples provide fully unsupervised word-by-word translations that reflect source-sentence meaning.
- A simple language model could potentially improve the Esperanto-English translations.