Source-linked AI summary
Exploiting Similarities among Languages for Machine Translation
Tomas Mikolov, Quoc V. Le, Ilya Sutskever
TL;DR
Existing machine translation dictionaries and phrase tables are costly to expand, motivating a method for inferring missing entries. The paper learns distributed representations from large monolingual corpora and a linear mapping from a small bilingual dictionary, achieving strong translation performance while supporting related and distant language pairs.
Problem
Dictionaries and phrase tables require substantial effort to generate and remain central resources for statistical machine translation.
Method
The method trains monolingual distributed representations and learns a linear projection between language spaces from a small bilingual dictionary to translate missing words and phrases.
Results
The approach successfully learns meaningful translations for individual words and short phrases, including between language pairs that are not closely related.
Takeaways & Limitations
The method can enrich and improve existing dictionaries and phrase tables for machine translation.
Takeaways & Limitations
A word-count baseline is computationally expensive at translation time and ignores words absent from the known dictionary when forming count vectors.
Abstract
from arXiv · showhide
Dictionaries and phrase tables are the basis of modern statistical machine translation systems. This paper develops a method that can automate the process of generating and extending dictionaries and phrase tables. Our method can translate missing word and phrase entries by learning language structures based on large monolingual data and mapping between languages from small bilingual data. It uses distributed representation of words and learns a linear mapping between vector spaces of languages. Despite its simplicity, our method is surprisingly effective: we can achieve almost 90% precision@5 for translation of words between English and Spanish. This method makes little assumption about the languages, so it can be used to extend and refine dictionaries and translation tables for any language pairs.
1 Introduction
The paper proposes using distributed representations and a learned linear projection to automate dictionary and phrase-table expansion for machine translation. The approach complements existing methods and is intended to remain useful across both related and substantially different language pairs.
- Statistical machine translation depends on dictionaries and phrase tables that require substantial effort to generate, while remaining below human translation performance.
- The method learns monolingual word representations from large text collections, then uses a small bilingual dictionary to learn a linear projection between language spaces.At test time, a source vector is projected into the target space and the most similar target word is returned.
- Similar geometric arrangements across language vector spaces provide the basis for learning an accurate linear mapping for translation.The visualization uses English and Spanish numbers and animals projected into two dimensions with PCA and manually rotated.
- The approach complements morphology- and context-based translation methods, while remaining useful for substantially different language pairs such as English–Czech and English–Chinese.
- Translation scores for word pairs can augment phrase tables with candidates or filter errors from existing translation tables and dictionaries.
2 The Skip-gram and Continuous Bag-of-Words Models
Skip-gram and CBOW learn distributed word representations from neighboring-word prediction objectives. Their low computational complexity supports training on very large corpora, while their vector spaces capture semantic similarity and linear conceptual relations.
- Distributed representations place similar words near one another, enabling generalization to novel patterns and more robust model estimation.The paper explains this through words such as school and university appearing in similar contexts.
- CBOW combines surrounding-word representations to predict the middle word, whereas Skip-gram uses an input word representation to predict its context.
- Both models can be trained efficiently on large text datasets because their architectures have low computational complexity.The paper states that they can be trained on billions of words in hours; Skip-gram is preferable with smaller monolingual data, while CBOW is faster on larger datasets.
- Skip-gram training maximizes the average log probability of predicting context words within a window around each center word.The objective sums over context positions and training words in the corpus.
- The full-softmax formulation is expensive because computing its gradient scales with vocabulary size, motivating hierarchical softmax as a more efficient alternative.Hierarchical softmax reduces the complexity of computing the log probability approximately logarithmically in vocabulary size.
- The learned vectors also encode relationships between concepts through linear operations, including analogies such as France–Paris and king–man+woman.
3 Linear Relationships Between Languages
The paper argues that corresponding concepts in different languages often preserve similar geometric arrangements in their vector spaces. A linear transformation learned from a few known translations can therefore recover additional translations.
- English and Spanish vectors for corresponding numbers and animals exhibit similar geometric arrangements.
- The relationship between language vector spaces can be modeled as a linear mapping involving rotation and scaling.
- Knowing translations for only one and four allows the method to learn a transformation matrix that can translate the remaining numbers.
4 Translation Matrix
The Translation Matrix learns a transformation that maps source-language word vectors toward vectors for their translations, then retrieves the closest target-language word. Despite its simplicity, the method worked well experimentally.
- The transformation is learned by minimizing the squared distance between mapped source vectors and target translation vectors.
- The method learns a transformation matrix W that makes each source vector x_i approximate its paired target vector z_i.The paired vectors represent a source word and its translation.
- At prediction time, the method maps a new source vector with z = Wx and selects the closest target-language representation using cosine similarity.
- Despite its simplicity, the linear transformation worked better than nearest neighbor and as well as neural network classifiers.
5 Experiments on WMT11 Datasets
The WMT11 experiments evaluate word and short-phrase translation using monolingual English, Spanish, and Czech corpora, controlled preprocessing, and several baselines. Results show that combining morphology with the Translation Matrix helps related languages, while the matrix approach is more useful for distant pairs.
- 5.1 Setup Description: The experiments use WMT11 monolingual datasets for English, Spanish, and Czech, with tokenization, deduplication, numeric normalization, and special-character removal.
- 5.1 Setup Description: Short phrases such as “ice cream” are represented as single tokens when their co-occurrence exceeds expectations from unigram probabilities.
- 5.1 Setup Description: Named entities are removed to test non-obvious translations, while vocabularies contain words occurring at least five times in the corpus.
- 5.1 Setup Description: Accuracy is measured on the subsequent 1K source-language words after training on the most frequent 5K words, reporting top 1 and top 5 exact-match accuracy.
- 5.2 Baseline Techniques: The baselines compare edit distance based on morphology with word co-occurrence counts, whose vectors are mapped using the dictionary and matched in the target space.
- 5.2 Baseline Techniques: The count-based baseline is computationally expensive because its translation vector size grows linearly with dictionary size and ignores words absent from the known dictionary.
- 5.3 Results with WMT11 Data: The Translation Matrix approach can be combined with Edit Distance, which works well for English–Spanish but is less useful for more distant English–Czech pairs.
- 5.3 Results with WMT11 Data: The best settings used source vectors around 2x–4x larger than target vectors, including 800-dimensional English and 200-dimensional Spanish vectors for English-to-Spanish translation.
6 Large Scale Experiments
Large-scale monolingual data improves translation performance and enables useful translations of infrequent words, while confidence scoring and combined similarity measures support selective high-precision translation.
- Precision improves as the amount of monolingual training data increases for English-to-Spanish translation.The experiments use several-billion-word English and Spanish corpora and evaluate the next 1K words after the 5K dictionary entries.
- Around 60% Precision@5 remains achievable for words ranked 15K–19K, despite declining accuracy on harder, less frequent test sets.The linear transformation is trained on the 5K most frequent words and their translations.
- 25% Precision@5 is obtained for ranks 15K–19K with small WMT11 models, showing that large monolingual datasets are needed for accurate infrequent-word translation.
- 6.1 Using Distances as Confidence Measure: A confidence score based on the closest target-language vector can skip translations whose projected vectors are below a threshold.The score is maxi∈V cos(Wx, zi), and translations are skipped when it falls below the threshold.
- 6.1 Using Distances as Confidence Measure: Combining the translation matrix with edit distance further improves accuracy, especially for Precision@1.For EN→ES words ranked 5K–6K, approximately half can be translated at around 90% Precision@5.
- High-quality translations can be used either to fill missing dictionary entries or to identify existing entries that may be ambiguous or incorrect.
7 Examples
The examples illustrate translation behavior, high-confidence English-to-Spanish outputs, dictionary-error detection, and extension to languages whose word concepts differ from English.
- 7 Examples: Spanish-to-English mistakes are often semantically related to the correct translation.The examples use the translation matrix alone, without edit-distance similarity.
- 7.2 High Confidence Translations: High-confidence English-to-Spanish translations achieve around 75% Precision@1 when edit distance and the translation matrix are combined.The examples use a confidence score above 0.5 and models trained on large corpora.
- 7.3 Detection of Dictionary Errors: Dictionary-error detection compares system outputs with existing entries, but the manually selected demonstration is highly subjective.
- 7.3 Detection of Dictionary Errors: Existing dictionary entries were about as accurate as or more accurate than the system in about 85% of cases, while the system was better in the remaining 15%.
- The method is evaluated on Vietnamese, where the concept of a word differs from English and large-scale phrase data are used.The Vietnamese Skip-gram training data includes about 1.3B phrases related to English words and short phrases.
8 Conclusion
The approach successfully learns meaningful translations for individual words and short phrases, including across languages that are not closely related. It can also enrich and improve existing dictionaries and phrase tables, with low-resource applications left for future research.
- The approach learns meaningful translations for individual words and short phrases using large monolingual data and a small starting dictionary.
- It works well for language pairs that are not closely related, including English–Czech and English–Vietnamese.
- The method can enrich and improve existing dictionaries and phrase tables, potentially improving current state-of-the-art machine translation systems.
- Applying the approach to low-resource domains remains an open direction for future research.