Source-linked AI summary
On the Limitations of Unsupervised Bilingual Dictionary Induction
Anders Søgaard, Sebastian Ruder, Ivan Vulić
TL;DR
The paper examines why unsupervised bilingual dictionary induction, a key component of unsupervised machine translation, is unreliable under common cross-lingual embedding assumptions. It evaluates these assumptions and methods across linguistic and embedding conditions, finding strong sensitivity to language pair, corpus comparability, and embedding parameters, while identical-word seeds improve robustness and graph similarity predicts performance.
Problem
Unsupervised machine translation relies on bilingual dictionary induction without dictionaries, translations, or comparable corpora, but the validity and robustness of its embedding-alignment assumptions are limited.
Method
The paper examines Conneau et al. (2018)'s adversarial embedding alignment, tests it across languages and embedding conditions, and introduces a Laplacian-eigenvalue graph-similarity metric.
Results
Unsupervised BDI is strongly affected by language pair, corpus comparability, and embedding parameters; eigenvector similarity predicts performance, while identical-word seeds make induction more robust.
Takeaways & Limitations
Unsupervised BDI should be assessed against embedding-space similarity and tested beyond closely matched languages, corpora, and embedding settings.
Takeaways & Limitations
Adversarial training is inherently unstable: Estonian, Finnish, and Greek sometimes perform on par with Turkish in approximately 1 out of 10 runs.
Abstract
from arXiv · showhide
Unsupervised machine translation---i.e., not assuming any cross-lingual supervision signal, whether a dictionary, translations, or comparable corpora---seems impossible, but nevertheless, Lample et al. (2018) recently proposed a fully unsupervised machine translation (MT) model. The model relies heavily on an adversarial, unsupervised alignment of word embedding spaces for bilingual dictionary induction (Conneau et al., 2018), which we examine here. Our results identify the limitations of current unsupervised MT: unsupervised bilingual dictionary induction performs much worse on morphologically rich languages that are not dependent marking, when monolingual corpora from different domains or different embedding algorithms are used. We show that a simple trick, exploiting a weak supervision signal from identical words, enables more robust induction, and establish a near-perfect correlation between unsupervised bilingual dictionary induction performance and a previously unexplored graph similarity metric.
1 Introduction
Unsupervised cross-lingual embeddings aim to induce bilingual lexicons and machine translation without dictionaries, translations, or other cross-lingual supervision. This paper argues that the assumptions behind such methods are not generally valid and identifies factors limiting unsupervised bilingual dictionary induction.
- Unsupervised cross-lingual embeddings could enable bilingual lexicon induction and machine translation for low-resource languages without dictionaries or translations.
- These approaches assume that monolingual word-embedding graphs are approximately isomorphic across languages.
- The authors find that this graph-isomorphism assumption is not generally true, creating important limitations for methods based on it.
- The paper examines Conneau et al. (2018), introducing a Laplacian-eigenvalue metric to quantify embedding-space similarity.
- Unsupervised BDI performance depends heavily on language pair, monolingual-corpus comparability, and word-embedding algorithm parameters.
2 How similar are embeddings across languages?
The paper tests whether nearest-neighbor graphs in different languages are approximately isomorphic and finds substantial structural differences, even for frequent translated words. It therefore introduces a Laplacian-spectrum metric to quantify graph similarity and relate it to unsupervised BDI performance.
- Isomorphism: Nearest-neighbor graphs are far from isomorphic even for English and German embeddings trained on comparable Wikipedia corpora with identical hyperparameters.
- Isomorphism: English and German graphs remain non-isomorphic when comparing frequent English words with their German translations, partly because German case creates distinct forms.
- Isomorphism: Nearest-neighbor graphs of frequent translated nouns are also non-isomorphic, and only 1/10 Spanish noun samples had isomorphic corresponding graphs.
- Eigenvector similarity: The paper introduces a spectral metric based on Laplacian eigenvalues to quantify varying degrees of similarity between non-isomorphic embedding graphs.
- Eigenvector similarity: The metric uses the sum of squared differences between the largest k Laplacian eigenvalues, with k selected from a 90% cumulative-eigenvalue criterion.
- Eigenvector similarity: A value of Δ=0 indicates isospectral graphs, while larger Δ values indicate less similar Laplacian spectra.
3 Unsupervised cross-lingual learning
The paper evaluates fully unsupervised cross-lingual embedding alignment across languages, corpora, embedding settings, and query types. Conneau et al.'s method combines adversarial mapping, iterative Procrustes refinement, and CSLS, while identical-word seeds provide a weakly supervised alternative.
- Experimental settings: The evaluation extends beyond Conneau et al.'s languages to Estonian, Finnish, Greek, Hungarian, Polish, and Turkish, including morphologically varied settings.
- Experimental settings: The experiments vary corpus domains, using combinations of Wikipedia, EuroParl, and the EMEA medical corpus to test robustness under non-comparable data.
- Method: The method learns source and target embeddings, trains an adversarial translation matrix, refines it with bidirectional seed pairs and Orthogonal Procrustes analysis, then applies CSLS for nearest-neighbor retrieval.
- Method: Iterative Procrustes refinement creates new seed translation pairs and improves performance across the board in the reported experiments.
- Weak supervision: Identically spelled words shared across vocabularies can serve as bilingual seeds without additional transliteration, lemmatization, or normalization.
4 Experiments
Experiments show that unsupervised bilingual dictionary induction is highly sensitive to language pair, corpus comparability, embedding-training choices, and evaluation conditions. Identical-word supervision is more robust in many settings, while graph similarity closely tracks unsupervised performance.
- 4.2 Impact of language similarity: Unsupervised BDI fails almost completely for English–Estonian and English–Finnish, with fewer than 1/1000 correct nearest-neighbor translations.Performance is more reasonable for languages with dependent marking and for the similar Estonian–Finnish pair.
- 4.3 Impact of domain differences: Unsupervised BDI performs well for Spanish when corpora share a domain but approaches zero across different domains; Finnish and Hungarian remain poor even with similar domains.The identical-word baseline achieves decent performance in many of these setups.
- 4.3 Impact of domain differences: P@1 for English–Spanish drops from 81.19% to 46.52% with smaller Wikipedia corpora.The result underscores the importance of large monolingual corpora from similar domains.
- 4.4 Impact of hyper-parameters: Unsupervised BDI fails for English–Spanish when the monolingual spaces use different underlying embedding algorithms.Small hyper-parameter changes within the same algorithm cause only slight score drops, and Spanish cbow is not explained by lower monolingual quality.
- 4.6 Impact of evaluation procedure: Verb queries have the lowest performance across the evaluated part-of-speech classes.Frequency sensitivity is strongest for Hungarian, weaker for Spanish, and P@1 is zero across all frequency ranks for English–Finnish.
- 4.7 Graph similarity: The graph similarity metric correlates strongly with unsupervised BDI performance, with ρ ∼0.89.This metric is proposed as a diagnostic for identifying embedding spaces where the approach is likely to work.
5 Related work
Prior work developed unsupervised cross-lingual embeddings and machine translation methods intended to operate without aligned linguistic data. These approaches critically depend on accurate unsupervised bilingual dictionary induction and back-translation.
- Unsupervised cross-lingual embedding methods aim to learn bilingual representations without aligned words, sentences, or documents.
- Recent unsupervised bilingual dictionary induction methods use adversarial learning with a discriminator and a generator that learns a translation matrix.
- Unsupervised neural machine translation systems reconstruct corrupted source sentences and translate them back after translation.
- These unsupervised neural machine translation methods critically rely on accurate unsupervised bilingual dictionary induction and back-translation.
6 Conclusion
The paper examines when unsupervised bilingual dictionary induction succeeds and finds that morphology, corpus-domain differences, and embedding algorithms can challenge it. It also identifies eigenvector similarity of sampled nearest-neighbor subgraphs as predictive of performance.
- Differences in morphology, corpus domains, or word embedding algorithms may challenge unsupervised bilingual dictionary induction.
- Eigenvector similarity of sampled nearest-neighbor subgraphs predicts unsupervised bilingual dictionary induction performance.
- The authors intend these findings to guide further development of unsupervised bilingual dictionary induction and related methods.