Non-Adversarial Unsupervised Word Translation

1 Introduction

• The motivation is that languages describing similar human experiences may share statistical properties, including overlap among their most common words.The paper also cites prior evidence that co-occurrence word-vector spaces can be related by an affine transformation.Source: §1 · p. 1
• GAN-based translation treats the target-language distribution as real and the mapped source-language distribution as fake.These approaches had recently shown effectiveness with modern word-vector representations.Source: §1 · p. 1
• The paper asks whether GANs are necessary for the success of unsupervised word translation and proposes directly mapping words across languages.The proposal relies on a simple learned mapping and the similarity of the concepts described by the two languages.Source: §1 · p. 1
• Matching means and covariances initializes the mapping, after which closest-word matches can iteratively refine the affine transformation.Some matches may be correct even from a random initialization and can help improve the transformation.Source: §1 · p. 2
• The method is reported to outperform state-of-the-art GAN-based methods while using few hyper-parameters, parallelizing easily, and remaining interpretable.The authors say each step has a clear goal and a success metric evaluable without a ground-truth bilingual lexicon.Source: §1 · p. 2

2 Related Work

• Earlier methods found matches among small word sets using co-occurrence or context variability, while later work expanded coverage with seed matches and heuristics.Examples include spelling and frequency heuristics combined linearly and greedy matching, as well as latent cross-lingual links learned with iterative EM.Source: §2 · p. 2
• Artetxe et al. (2017) also used iterative word-vector matching, but initialized it with seed matches, unlike this paper’s method.Their approach used word2vec embeddings.Source: §2 · p. 2
• Circularity objectives encourage translations in both directions to return samples to their starting points and are related to orthogonality for linear transformations.Unlike this paper’s use of circularity in the optimization energy, Conneau et al. (2017) used it to validate solutions and select hyperparameters.Source: §2 · p. 2
• Recent fully unsupervised solutions by Zhang et al. (2017a,b) and Conneau et al. (2017) all used GANs, differing in training, model selection, and matching details.The cited discussion notes that discriminator-based methods can be hard to interpret because their weight-dependent metrics evolve during training.Source: §2 · p. 2
• Alternatives included fitting random noise vectors iteratively and measuring image distinguishability with pretrained-network activations, called perceptual loss.The passage identifies Bojanowski et al. (2017) and Chen and Koltun (2017), respectively.Source: §2 · pp. 2–3

3 Non-Adversarial Word Translation

• The unsupervised objective is to predict a corresponding target-language word for each source word without training examples of the correspondence function.If multiple target words are possible, the objective is to predict one member of that set.Source: §3 · p. 3

3.1 Approximate Alignment with PCA

• PCA initialization projects centered word vectors onto principal components, relying on the assumption that language pairs share some principal axes of variation.The paper notes that word distributions differ from 3D point clouds and PCA projection does not align them in the general case.Source: §3.1 · p. 3, §3.2 · p. 3

3.2 Mini-Batch Cycle Iterative Closest Point

• Mini-Batch Cycle ICP learns transformations in both directions, with cycle constraints requiring words mapped across languages and back to remain unchanged.It computes nearest-neighbor matches at each epoch and optimizes with mini-batch SGD; the pipeline initializes on projected data, then refines on the original 300D vectors using reciprocal pairs.Source: §3.2 · pp. 3–4

3.3 Fine-tuning

• Optional Procrustes fine-tuning iteratively estimates transformations and matches over the 200k-word vocabulary, initialized from MBC-ICP.The paper reports typical European-language accuracy gains of 1–2% for its method, compared with 10–15% for previous work.Source: §3.3 · p. 4

3.4 Matching Metrics

• CSLS improves final retrieval performance over nearest-neighbor matching by adjusting cosine similarity using each word’s average similarity to its 10 nearest neighbors in the other domain.The paper evaluates final retrieval using CSLS.Source: §3.4 · p. 4

• Multiple stochastic solutions help initialize MBC-ICP, but require either choosing the best solution within a fixed run budget or finding a stopping criterion for serial runs.Stochasticity comes from randomized PCA and the order of mini-batches.Source: §4 · pp. 4–5
• Values significantly below the median reconstruction cost almost always correspond to a good solution, although isotropic distributions have many degenerate solutions.Source: §4 · p. 5
• Reconstruction loss clearly distinguishes converged from non-converging runs, and the paper shows final-cost distributions for En-Fr and En-Ar.Figure 1 plots loss by epoch for successful and unsuccessful runs and shows final reconstruction-loss distributions for both language pairs.Source: §4 · p. 5, Figure 1 · p. 5

Experimental setup

• The evaluation measures word-translation accuracy and runtime, comparing MBC-ICP with the MUSE-repository GAN baseline and a supervised Procrustes baseline.For reported baseline results, the authors describe using prior results as-is except for specified language and direction cases, where they report the best baseline result per direction.Source: §5 · p. 5

Word translation accuracy

• Across non-European pairs, the unsupervised methods have complementary failures: the GAN fails on Farsi, Hindi, Bengali, Vietnamese, and one direction each of Japanese and Indonesian; MBC-ICP fails on Chinese, Japanese, and Vietnamese.Where both methods succeed, MBC-ICP tends to do much better.Source: §5 · p. 6

Runtime

• Each MBC-ICP solution runs in 47 seconds on CPU and can be parallelized across stochastic initializations; skipping its optional refinement costs typically 1–2% accuracy.The paper reports the GAN baseline takes around 3,000 seconds on GPU and requires finetuning for competitive performance.Source: §5 · pp. 6–7

Word translation accuracy

• MBC-ICP outperforms the GAN baseline on all evaluated European pairs before finetuning and remains ahead after finetuning, while matching the supervised baseline on European pairs except En-Ru.Finetuning typically adds only 1–2% for MBC-ICP on European pairs; the GAN benefits more from finetuning on 200k words.Source: Table 1 · p. 6, §5 · p. 6, Table 2 · p. 7

Ablation analyses

• On En-Es, ablations show much lower convergence without PCA or with PCA but no dimensionality reduction, and the best run without PCA fails to obtain a good translation.The authors conclude that both PCA and dimensionality reduction are essential for the method's success.Source: §5 · p. 7, Table 3 · p. 7

Stochasticity

• Using both PCA and batch-order stochasticity usually yields a higher fraction of converging runs than using either source alone, including on En-Es and En-Ar.Stochastic PCA is beneficial because aligned principal components across languages usually do not occur.Source: §5 · p. 7, Table 4 · p. 7

Convergence statistics

• MBC-ICP converges on more runs than P-ICP for En-Es, En-Fr, and En-Ru, while P-ICP has no converged runs for En-Ar or En-It across 500 runs.The authors attribute the broader convergence range to mini-batch updates and batch-order stochasticity; convergence rates generally track language-pair distance, with exceptions.Source: §5 · pp. 7–8, Figure 2 · p. 8

6 Conclusions

• The method outperforms current state-of-the-art GAN methods in pre- and post-finetuning word translation accuracy and runs much faster with CPU parallelization.The authors also describe it as interpretable, with each stage having an intuitive loss function and understandable objective.Source: §6 · p. 8
• Our method performs better on West and South Asian languages, while GANs perform better on Chinese.Performance is typically better for more related language pairs, though Indonesian-to-English outperforms Russian-to-English.Source: §6 · p. 8
• The work highlights the potential benefits of considering alternatives to adversarial methods in unsupervised learning.Source: §6 · p. 8