Source-linked AI summary
Offline bilingual word vectors, orthogonal transformations and the inverted softmax
Samuel L. Smith, David H. P. Turban, Steven Hamblin, Nils Y. Hammerla
TL;DR
The paper addresses offline bilingual word-vector alignment and translation without requiring extensive expert bilingual supervision. It proves that the alignment should be orthogonal, introduces inverted softmax retrieval, and achieves strong word- and sentence-translation precision across dictionary and dictionary-free settings.
Problem
Offline bilingual word vectors require aligning independently trained monolingual embeddings, traditionally using expert-compiled bilingual dictionaries.
Method
The paper uses an orthogonal transformation obtained by SVD, combines it with inverted softmax retrieval, and constructs dictionary-free alignments from identical character strings or sentence vectors.
Results
The method improves precision @1 from 34% to 43% for English-to-Italian word translation, achieves 40% using a pseudo-dictionary, and retrieves English sentence translations with 68% precision from 200k Italian candidates.
Takeaways & Limitations
Orthogonal alignment supports offline bilingual vectors with expert dictionaries, identical character strings, or sentence-level bilingual signals, including translation of previously unseen words and sentences.
Takeaways & Limitations
The SVD alignment uses dictionary matrices rather than full vocabularies, and the authors question the need for CCA’s additional representation-learning stage.
Abstract
from arXiv · showhide
Usually bilingual word vectors are trained "online". Mikolov et al. showed they can also be found "offline", whereby two pre-trained embeddings are aligned with a linear transformation, using dictionaries compiled from expert knowledge. In this work, we prove that the linear transformation between two spaces should be orthogonal. This transformation can be obtained using the singular value decomposition. We introduce a novel "inverted softmax" for identifying translation pairs, with which we improve the precision @1 of Mikolov's original mapping from 34% to 43%, when translating a test set composed of both common and rare English words into Italian. Orthogonal transformations are more robust to noise, enabling us to learn the transformation without expert bilingual signal by constructing a "pseudo-dictionary" from the identical character strings which appear in both languages, achieving 40% precision on the same test set. Finally, we extend our method to retrieve the true translations of English sentences from a corpus of 200k Italian sentences with a precision @1 of 68%.
1 INTRODUCTION
Bilingual word vectors align words across languages by meaning, either through joint training or by mapping independently trained spaces. This work argues that offline alignment is best understood as an orthogonal rotation found with SVD and enhanced by inverted softmax retrieval.
- Motivation: Bilingual vectors align words by meaning across languages, supporting machine translation and language-agnostic text classification.Monolingual models require independent training for each language, and text in one language cannot improve another language’s classifier without translation.
- Online and offline approaches: Online methods learn both languages together using monolingual text plus a smaller bilingual corpus of aligned sentences.The bilingual signal acts as a regularisation term that encourages similar words in both languages to be nearby.
- Online and offline approaches: Offline methods independently train two vector spaces and then align them with a linear transformation using bilingual word correspondences.The map analogy describes rotating one independently drawn space until corresponding landmarks align.
- Paper contribution: The paper proves that a self-consistent transformation should be orthogonal, interpretable as a rotation, and obtainable through singular value decomposition.Figure 1 illustrates the resulting shared English-Italian semantic space using a 5000-pair training dictionary and held-out examples.
- Paper contribution: The inverted softmax addresses hubness, where some target words become nearest neighbours for many source words and reduce translation performance.It normalises probabilities over source words rather than target words.
2 OFFLINE BILINGUAL LANGUAGE VECTORS
Offline bilingual vectors align separately trained monolingual spaces with a linear map, which this work constrains to an orthogonal transformation learned by SVD. The method also addresses hubness with inverted softmax and extends alignment from dictionaries to sentences.
- 2.1 PREVIOUS WORK: Offline bilingual vectors map separately trained source and target embeddings using a learned linear transformation.Mikolov et al. trained the map by minimizing squared reconstruction error, then retrieved translations by cosine similarity and nearest neighbours.
- 2.2 THE SIMILARITY MATRIX AND THE ORTHOGONAL TRANSFORM: Self-consistency requires the mapping to be orthogonal, satisfying O^T O = I and preserving vector norms.Mapping a source vector to the target and back should recover the original vector; with normalized embeddings, similarity becomes cosine similarity.
- 2.3 THE SVD AND CCA: The orthogonal map is obtained by computing M = Y_D^T X_D = UΣV^T and setting O = UV^T.The matrices are formed from paired dictionary vectors, and the SVD gives a numerically efficient solution because M has the embedding dimensionality.
- 2.3 THE SVD AND CCA: The CCA variant adds mean subtraction and SVD-based representation changes before applying orthogonal alignment, but the authors consider its first stage potentially redundant.The concern is that high-quality word vectors already exist and the additional transformations are learned only from dictionary matrices.
- 2.4 THE INVERTED SOFTMAX: Inverted softmax normalizes over source words, assessing how likely a candidate target translates back to the source to mitigate hubness.Nearest-neighbour retrieval can over-select hub target words; the method instead chooses the target maximizing this reverse translation probability.
- 2.5.2 ALIGNED SENTENCES: Aligned sentence pairs provide weak word-level alignment information, allowing the orthogonal map to retrieve translations of both words and sentences.Sentence vectors are normalized sums of word vectors, so words and sentences share a vector space; many aligned pairs supply information for alignment.
3 EXPERIMENTS
Experiments evaluate word and sentence translation between English and Italian using expert, pseudo, and phrase dictionaries. The proposed SVD-based method with inverted softmax achieves strong precision, including without expert bilingual knowledge and for sentence retrieval.
- Experimental setup: Experiments reuse Dinu et al.’s word vectors, dictionaries, and test sets, comparing the proposed procedure with Mikolov’s method, Dinu’s method, and CCA.The vectors were trained with word2vec from English and Italian corpora; nearest-neighbour retrieval was used for the comparison methods.
- Expert training dictionary: 43% precision @1 is achieved from English to Italian, and 38% from Italian to English, when combining SVD, inverted softmax, and dimensionality reduction.These experiments use the expert training dictionary; all methods perform better in the English-to-Italian direction.
- Expert training dictionary: 69% precision @1 is achieved on test cases drawn from the 5k most common English words, with lower precision for less common words.The result uses the expert training dictionary and evaluates English-to-Italian translation.
- Identical character strings: 40% mean precision @1 is achieved from English to Italian using a pseudo-dictionary of identical character strings and no expert bilingual knowledge.The pseudo-dictionary contains 47,074 shared strings; Mikolov’s and Dinu’s methods achieve 1% and 6%, respectively.
- Europarl sentence retrieval: 67.8% precision @1 is achieved when retrieving an English sentence’s Italian translation from 200k candidates using only sentence vectors.The sentence vectors sum and normalise word vectors, omitting word order and sentence length information.
- Europarl sentence retrieval: Nearest-neighbour retrieval raises Italian-to-English sentence retrieval from 48.6% to 65.6% on the same orthogonal transformation.The result followed divergence of the inverted-softmax inverse temperature β and suggests retrieval should tune between conventional and inverted softmax.
4 SUMMARY
The paper combines orthogonal SVD-based alignment with inverted softmax retrieval to improve offline bilingual word translation and extend it to dictionary-free settings and sentence retrieval.
- 43% precision @1 improves on Mikolov’s original 34% when translating common and rare English words into Italian.The method uses a 5k-word English–Italian training dictionary and combines SVD, inverted softmax, and dimensionality reduction.
- 40% precision is achieved using a pseudo-dictionary of identical word strings, without an expert bilingual signal.Mikolov’s method achieves 1% precision in this setting.
- The optimal transformation is orthogonal and is obtained with a single SVD applied to a dictionary of translation pairs.
- 68% precision @1 retrieves true English-sentence translations from 200k Italian candidate sentences.The sentence-retrieval experiment uses sentence vectors and is described as worthy of further investigation.
A THE ORTHOGONAL PROCRUSTES PROBLEM
The appendix relates the paper’s cost function to the orthogonal Procrustes problem, whose analytic solution applies under normalization and orthogonality assumptions.
- The orthogonal Procrustes problem minimizes squared reconstruction error subject to an orthogonal constraint.This related problem provides the analytic solution used for the paper’s cost function.
- Normalized X and Y, together with norm-preserving O, support the equivalence between the relevant cost formulations.
- A is constant, so the cost functions in equations 6 and 13 are equivalent.
B ADDITIONAL EXPERIMENTS
Additional experiments report precision @5 and @10 for sentence-translation retrieval, comparing word- and phrase-dictionary setups and translation directions.
- The inverted softmax performs well for English-to-Italian sentence retrieval but is less effective in the reverse direction.
- Dinu’s method improves more rapidly from precision @1 to @5 to @10 than the other methods.
- Dinu’s method performs better with the word dictionary at precision @1 but prefers the phrase dictionary at precisions @5 and @10.
- Precision @5 and @10 evaluate retrieving true translations from 200k Italian sentences, or vice versa, across 5k samples.The experiments use bilingual vectors from either Dinu’s word dictionary or a Europarl phrase dictionary, with ns = 12800 in inverted softmax.