Source-linked AI summary
Loss in Translation: Learning Bilingual Word Mapping with a Retrieval Criterion
Armand Joulin, Piotr Bojanowski, Tomas Mikolov, Herve Jegou, Edouard Grave
TL;DR
Bilingual word mapping traditionally trains a least-squares rotation but uses a different retrieval criterion at inference. This paper directly optimizes a retrieval-inspired objective end to end and reports state-of-the-art word translation, especially for distant language pairs.
Problem
Existing bilingual mappings train with square loss but infer with a different retrieval criterion, creating a mismatch and leaving hubness as a limitation.
Method
The paper trains bilingual mappings directly with a CSLS-inspired retrieval objective, using convex relaxations and projected subgradient descent.
Results
The approach outperforms the state of the art on standard word-translation benchmarks, with improvements averaging 3 to 4% on five MUSE language pairs.
Takeaways & Limitations
End-to-end training makes the learning criterion consistent with inference and supports effective non-orthogonal mappings.
Takeaways & Limitations
The mapped models remain subject to hubness, in which some target vectors become nearest neighbors for unusually many words.
Abstract
from arXiv · showhide
Continuous word representations learned separately on distinct languages can be aligned so that their words become comparable in a common space. Existing works typically solve a least-square regression problem to learn a rotation aligning a small bilingual lexicon, and use a retrieval criterion for inference. In this paper, we propose an unified formulation that directly optimizes a retrieval criterion in an end-to-end fashion. Our experiments on standard benchmarks show that our approach outperforms the state of the art on word translation, with the biggest improvements observed for distant language pairs such as English-Chinese.
1 Introduction
Prior bilingual mapping methods learn linear transformations from small lexicons, but inference-time hubness correction creates a mismatch with square-loss training. The paper addresses this by training directly with a retrieval-inspired objective and reports state-of-the-art bilingual mappings.
- Small bilingual lexicons can supervise linear mappings that generalize to unseen words and support cross-language predictive-model transfer.
- Orthogonal Procrustes alignment of normalized word vectors improves accuracy on standard benchmarks.
- Hubness causes some word vectors to become nearest neighbors for abnormally many other words.
- Inference corrections such as ISF and CSLS are inconsistent with the square loss used to train the mapping.
- The proposed objective is inspired by CSLS, uses convex relaxations optimized by projected subgradient descent, and can exploit words outside the training lexicon.
- The approach achieves state-of-the-art bilingual mappings for more than 25 language pairs and shows benefits from alternative losses, unsupervised information, and non-orthogonal mappings.
2 Preliminaries on bilingual mappings
Bilingual mapping learns a transformation from source to target word vectors using a small seed lexicon, then retrieves translations for unseen source words. Traditional training uses square-loss least squares, often with an orthogonality constraint, while inference uses nearest-neighbor criteria that can suffer from hubness.
- The setup uses monolingually learned source and target vectors, with the first n bilingual pairs serving as seeds.
- The mapping W is learned from seed pairs by minimizing a loss between transformed source vectors and target vectors.
- With square ℓ2 loss, training becomes a least-squares problem solved in closed form.
- Orthogonality constrains W to preserve distances and similarities, and previous work found that it improves inferred lexicons.
- After learning W, translations for non-seed words are inferred through nearest-neighbor search over target vectors.
- Hubness affects bilingual nearest-neighbor inference, while ISF and CSLS mitigate it at inference time despite training with square loss.
3 Word translation as a retrieval task
The paper makes bilingual mapping consistent with retrieval by directly incorporating CSLS into training, then relaxes the optimization constraints to obtain efficient convex formulations.
- Retrieval criterion: The mapping objective directly includes CSLS so that the training and inference criteria are consistent.The model assumes ℓ2-normalized word vectors and initially constrains W to orthogonal matrices.
- Retrieval criterion: CSLS is introduced as a similarity measure using cosine similarity and nearest-neighbor sets in the target and source vocabularies.The second term makes the loss symmetric without changing the neighbors of x.
- Convex relaxation: Replacing cosine nearest-neighbor search with dot products yields an equivalent formulation that becomes convex when orthogonality is relaxed.The k nearest neighbors are the vectors with the largest dot products with Wxi.
- Convex relaxation: The non-smooth objective is optimized over the convex hull of orthogonal matrices, or without alignment constraints, using projected subgradient descent.The convex hull is the unit ball of the spectral norm, while projection uses SVD and singular-value thresholding.
- Convex relaxation: The objective is convex and piecewise linear because its neighborhood terms are maxima of linear functions of W.The maximization ranges over subsets of size k.
- Unsupervised information: Unsupervised information is incorporated by extending neighborhood searches from annotated target words to the whole dictionary.This enables the method to use words that do not occur in the seed lexicon.
4 Experiments
Experiments show that RCSLS consistently improves bilingual word-mapping performance by training with a retrieval-oriented criterion, while also benefiting from unlabeled vocabulary and unconstrained mappings. The method remains robust across benchmark settings and does not negatively impact analogy or similarity performance.
- The MUSE benchmark: RCSLS outperforms the state of the art by 3 to 4% in accuracy on five MUSE language pairs in both directions.The comparison uses 5k-word bilingual lexicons and Wikipedia fastText vectors; all methods use CSLS for retrieval.
- The MUSE benchmark: RCSLS performs 0.8% better without constraints for all evaluated language pairs.This contrasts with the +5.3% benefit of orthogonal constraints under the square loss.
- The MUSE benchmark: Extending normalization with unannotated words significantly improves RCSLS performance on all language pairs.The extended setting uses the full vocabulary rather than only training-lexicon pairs.
- The WaCky dataset: On noisy lexicons and relatively poor word vectors, RCSLS is on par with the state of the art and remains robust.These experiments use the WaCky setting of Dinu et al. and CSLS retrieval.
- Comparison with existing aligned vectors: With additional supervision, RCSLS improves more than Procrustes, including gains of +2.9% with CSLS and +6.1% with NN.Using the original aligned vectors gives a +3.5% improvement over MUSE vectors; the full lexicon adds further gains.
- Impact on word vectors: The learned mappings do not negatively affect word analogies or substantially alter word similarity, while reverse mapping improves Spanish analogy accuracy by 1%.The Spanish improvement is from 85% to 86% for Common Crawl vectors and from 87% to 88% for Wikipedia + News vectors.
5 Conclusion
The paper concludes that convexly relaxed CSLS training improves bilingual word-vector alignment while making training consistent with inference. It also finds that removing orthogonality does not degrade aligned-vector quality.
- Convex relaxation of the CSLS loss significantly improves bilingual word-vector alignment.
- The CSLS reformulation generalizes beyond dot products and supports end-to-end training consistent with inference.
- Removing the orthogonality constraint does not degrade the quality of aligned vectors.
Appendix A Ablation study
The ablation study examines training-lexicon size, nearest-neighbor count, alternative CSLS functions, and exact string matches. Results favor consistent retrieval-oriented training and show that local criterion properties matter more than the specific loss form.
- Size of training lexicon: RCSLS gains over Procrustes are marginal with small training sets but increase as the training lexicon grows.Figure 1 measures accuracy on the en-de pair as training-set size changes on a log scale.
- Impact of the number of nearest neighbors: 10 nearest neighbors is optimal for CSLS and RCSLS, while performance decreases significantly with many neighbors.The parameter affects Procrustes through retrieval and RCSLS through both its loss and retrieval criterion.
- Comparison of alternative criterions: Replacing the linear CSLS function with logSumExp does not significantly change performance.The logSumExp alternative is equivalent to logistic regression with hard mining.
- Comparison with alternative methods: The gap between RCSLS and other methods is higher with a nearest-neighbor criterion, suggesting that RCSLS transfers properties of CSLS to aligned-vector dot products.
Appendix B Alignment and word vectors
The appendix evaluates how embedding quality and non-orthogonal mappings affect alignment and downstream vector geometry. Non-orthogonal mappings do not negatively affect English analogies and achieve similar cross-lingual similarity results to orthogonal mappings.
- Quality of the embedding model: Word-vector quality affects the performance of word translation when vectors are trained on the same Wikipedia data with or without subword information.
- Impact on English word vectors: Non-orthogonal mappings do not negatively affect English word analogy performance across target languages.Table 11 compares raw English vectors with their mappings to four languages.
- Cross-lingual similarity: Non-orthogonal aligned vectors obtain similar cross-lingual word-similarity results to vectors aligned with an orthogonal matrix.The evaluation reports Pearson correlation on NASARI datasets.