Source-linked AI summary
How to (Properly) Evaluate Cross-Lingual Word Embeddings: On Strong Baselines, Comparative Analyses, and Some Misconceptions
Goran Glavas, Robert Litschko, Sebastian Ruder, Ivan Vulic
TL;DR
Recent projection-based CLE models were evaluated inconsistently and often only on BLI, limiting understanding of their downstream utility. The paper introduces a unified multi-task evaluation of supervised and unsupervised models across many language pairs, finding that performance depends on the task and that BLI-oriented optimization can harm downstream results. It identifies robust baselines and challenges claims that unsupervised models generally match or surpass supervised ones.
Problem
Projection-based CLE models are often evaluated only on BLI under incompatible protocols, leaving their downstream capacity and cross-model comparisons unclear.
Method
The paper systematically evaluates supervised and unsupervised projection-based CLE models on BLI and three downstream tasks across 28 language pairs using unified protocols.
Results
Performance depends strongly on the task: RCSLS is strongest on BLI, while BLI-oriented optimization can deteriorate downstream performance; VECMAP is the most robust unsupervised model but is significantly outperformed by supervised models.
Takeaways & Limitations
CLE evaluation should span multiple downstream tasks and reassess simple baselines rather than treating BLI performance as sufficient evidence of general utility.
Takeaways & Limitations
The task-correlation analysis uses coarse-grained model-level aggregation, while selecting the strongest baselines requires finer-grained tuning.
Abstract
from arXiv · showhide
Cross-lingual word embeddings (CLEs) enable multilingual modeling of meaning and facilitate cross-lingual transfer of NLP models. Despite their ubiquitous usage in downstream tasks, recent increasingly popular projection-based CLE models are almost exclusively evaluated on a single task only: bilingual lexicon induction (BLI). Even BLI evaluations vary greatly, hindering our ability to correctly interpret performance and properties of different CLE models. In this work, we make the first step towards a comprehensive evaluation of cross-lingual word embeddings. We thoroughly evaluate both supervised and unsupervised CLE models on a large number of language pairs in the BLI task and three downstream tasks, providing new insights concerning the ability of cutting-edge CLE models to support cross-lingual NLP. We empirically demonstrate that the performance of CLE models largely depends on the task at hand and that optimizing CLE models for BLI can result in deteriorated downstream performance. We indicate the most robust supervised and unsupervised CLE models and emphasize the need to reassess existing baselines, which still display competitive performance across the board. We hope that our work will catalyze further work on CLE evaluation and model analysis.
1 Introduction and Motivation
Projection-based CLE research lacked fair, comprehensive evaluation across tasks and language pairs. This paper compares supervised and unsupervised models under unified protocols to test whether BLI reflects downstream performance and whether unsupervised models outperform supervised ones.
- Motivation: Recent projection-based CLE models are often evaluated only on bilingual lexicon induction, although CLEs are intended to support downstream multilingual NLP.BLI performance may correlate poorly with downstream text classification and parsing, leaving the true cross-lingual capacity of CLE models unclear.
- Motivation: Published BLI results are difficult to compare because studies use different language pairs, training dictionaries, and evaluation dictionaries.These mismatched protocols hinder comparative analysis and can produce partial or inconsistent conclusions.
- Motivation: Supervised projection-based CLEs commonly require only 1K–5K translation pairs, while unsupervised models require no bilingual supervision.The paper therefore treats unsupervised models as practically justified only if they can outperform supervised alternatives.
- Contributions: The study evaluates state-of-the-art supervised and unsupervised projection-based CLE models on BLI, document classification, information retrieval, and natural language inference.The unified benchmark covers 28 language pairs spanning diverse language types, including pairs without English.
- Contributions: The authors report that optimizing CLE models for BLI may severely hurt downstream performance, while simple evaluated methods can outperform more intricate competitors.This motivates reassessing existing baselines and evaluating CLE models across tasks rather than relying on BLI alone.
2 Projection-Based Cross-Lingual Word Embeddings: Methodology
Projection-based CLEs align independently trained monolingual spaces by constructing a seed lexicon, forming aligned subspaces, and learning projections into a shared space. The evaluation includes supervised and unsupervised methods, including CCA, Procrustes-based baselines, bootstrapping, adversarial alignment, heuristics, and ranking-based optimization.
- Projection-Based Framework: Projection-based CLE induction learns a mapping between independently trained monolingual vector spaces using a supervised or automatically induced seed bilingual lexicon.Unsupervised dictionary induction commonly assumes approximate isomorphism between monolingual spaces.
- Projection-Based Framework: The framework retrieves source and target monolingual vectors for dictionary pairs, creates aligned subspaces, and learns projections into a shared cross-lingual space.Most models directly project source vectors into the target space, while the general case learns two projection matrices.
- Unsupervised Models: The unsupervised methods use diverse seed-dictionary strategies, including adversarial learning, similarity heuristics, PCA, and optimal transport.MUSE uses adversarial alignment, whereas VECMAP relies on similarity-distribution heuristics and subsequent Procrustes refinement.
- Supervised Models: CCA is evaluated as a simple supervised baseline that maps both input embedding spaces into a shared space.The method had been largely neglected in recent BLI evaluations.
- Supervised Models: The primary supervised baseline is the linear Procrustes solution, extended by PROC-B to bootstrap additional dictionary pairs when supervision is small.PROC-B iteratively learns bidirectional projections and augments the lexicon with mutual nearest neighbours.
- Supervised Models: RCSLS learns the projection by maximizing relaxed CSLS rather than minimizing Euclidean distances between aligned vectors.Its objective is explicitly designed for strong BLI performance.
3 Bilingual Lexicon Induction
The paper evaluates projection-based CLE models on BLI across 28 diverse language pairs using common dictionaries, metrics, and significance testing. Results show that simple supervised baselines are highly competitive, while performance varies with language proximity and supervision size.
- Evaluation Setup: BLI evaluation covers 28 pairs among eight languages spanning Germanic, Romance, Slavic, Uralic, and Turkic families.The setup uses 300-dimensional fastText embeddings, 200K-word vocabularies, and dictionaries with 1K, 3K, or 5K training pairs plus 2K test pairs.
- Evaluation Measures: MAP is advocated over P@k because it distinguishes models by the precise rank of the correct translation, while significance is tested with Bonferroni-corrected two-tailed t-tests.In the one-correct-translation setting, MAP equals mean reciprocal rank.
- Results and Discussion: RCSLS achieves the strongest BLI performance among the evaluated models, consistent with its objective being tailored specifically to BLI.CCA, PROC, and DLV trained on identical dictionaries are statistically indistinguishable, even at α = 0.1.
- Results and Discussion: PROC-B significantly improves over PROC with a 1K-pair dictionary and significantly outperforms RCSLS.Training with 3K or 5K pairs significantly beats 1K, but 5K does not significantly beat 3K, supporting limited gains from larger linear-map dictionaries.
- Results and Discussion: VECMAP is the most robust unsupervised model, yet supervised models trained with 1K to 5K pairs significantly outperform it.VECMAP is the only unsupervised model producing successful runs consistently for all language pairs.
- Results and Discussion: Language proximity strongly affects BLI, with same-family pairs performing better and supervised–unsupervised gaps widening for distant pairs.Reported MAP gaps include 19 points for EN–RU, 14 for DE–RU and EN–FI, and 10 for EN–TR and EN–HR.
4 Downstream Evaluation
The downstream evaluation shows that CLE performance is strongly task-dependent: models that excel on BLI do not necessarily transfer well to language understanding, classification, or retrieval. Across these tasks, bootstrapped PROC-B and unsupervised VECMAP emerge as robust reference models, while non-orthogonal RCSLS exposes a particularly strong BLI–downstream mismatch.
- Evaluation setup: The evaluation moves beyond BLI to XNLI, CLDC, and CLIR, covering diverse cross-lingual transfer and retrieval settings.XNLI tests sentence-level understanding, CLDC shallow topical meaning, and CLIR coarse semantic matching.
- Natural Language Inference: RCSLS is the best BLI model but performs suboptimally on XNLI, showing that BLI gains do not reliably predict cross-lingual language understanding.The paper reports a Spearman correlation of only 0.13 between corresponding BLI and XNLI scores across language pairs and models.
- Document Classification: RCSLS achieves peak CLDC scores, whereas PROC, PROC-B, and DLV remain weak despite strong performance on BLI and XNLI.RCSLS significantly outperforms VECMAP, which in turn significantly outperforms the other supervised models on CLDC.
- Information Retrieval: PROC-B outperforms all other CLE methods on CLIR, while the best BLI model, RCSLS, performs significantly worse than the simple PROC baseline.The significance result remains after Bonferroni correction at α = 0.05.
- Further discussion: Without RCSLS, BLI correlates almost perfectly with XNLI and CLIR but only weakly with CLDC, indicating that projection properties affect cross-task correspondence.RCSLS relaxes orthogonality to maximize CSLS similarity, allowing distortions of the source embedding space whose downstream impact requires further investigation.
- Further discussion: The task-correlation analysis is coarse-grained, so selecting the strongest baselines still requires finer-grained tuning by language pair and evaluation task.The experiments nevertheless identify PROC-B as a robust supervised baseline and VECMAP as a robust unsupervised baseline.
5 Conclusion
The paper argues that CLE progress has outpaced fair, systematic evaluation. Its comprehensive multi-task comparison shows task-dependent quality and motivates unified evaluation, baseline reassessment, and further model analysis.
- 5 Conclusion: The study addresses inadequate CLE evaluation by systematically testing models on BLI and three downstream tasks across many language pairs.It reports that CLE quality depends substantially on the task and that BLI-focused optimization can deteriorate downstream performance.
- 5 Conclusion: The authors identify robust supervised and unsupervised CLE models and emphasize reassessing existing baselines through unified, comprehensive evaluation.They present these directions as support for future CLE evaluation, analysis, and model development.