Source-linked AI summary
Learning Crosslingual Word Embeddings without Bilingual Corpora
Long Duong, Hiroshi Kanayama, Tengfei Ma, Steven Bird, Trevor Cohn
TL;DR
Prior crosslingual embedding methods can require expensive bilingual resources, scale poorly with monolingual data, or fail to handle translation ambiguity. This paper learns crosslingual embeddings from monolingual corpora and a high-coverage dictionary using context-based translation selection. It reports state-of-the-art bilingual lexicon induction and competitive results on monolingual word similarity and crosslingual document classification.
Problem
Prior approaches can require costly bilingual corpora, scale poorly to large monolingual data, or inadequately handle translation ambiguity.
Source: §1 · p. 1Method
The method uses a high-coverage bilingual dictionary with monolingual corpora and selects translations by context in an EM-style training algorithm.
Source: §1 · pp. 1–2, §4.1 · p. 3Results
The model achieves state-of-the-art bilingual lexicon induction and competitive results on monolingual word similarity and crosslingual document classification.
Source: §10 · p. 8Takeaways & Limitations
The authors report that their combination techniques, especially regularization, can also improve monolingual word embeddings.
Source: §10 · p. 8Using expectations over translations as in standard EM slightly degraded results.
Source: §4.3 · p. 4
Abstract
from arXiv · showhide
Crosslingual word embeddings represent lexical items from different languages in the same vector space, enabling transfer of NLP tools. However, previous attempts had expensive resource requirements, difficulty incorporating monolingual data or were unable to handle polysemy. We address these drawbacks in our method which takes advantage of a high coverage dictionary in an EM style training algorithm over monolingual corpora in two languages. Our model achieves state-of-the-art performance on bilingual lexicon induction task exceeding models using large bilingual corpora, and competitive results on the monolingual word similarity and cross-lingual document classification task.
1 Introduction
Crosslingual transfer needs lexical matching, but prior approaches can require costly bilingual resources, fail to scale with monolingual data, or mishandle translation ambiguity. This paper uses a high-coverage dictionary and context-based translation selection, reporting strong results across bilingual, monolingual, and transfer evaluations.
1 Introduction
- Crosslingual word embeddings place source- and target-language lexicons in a shared vector space, addressing lexical matching for transfer.Such transfer can apply a model built in a resource-rich language to a resource-poor target language.Source: §1 · p. 1
- Prior methods may need expensive bilingual or comparable corpora, struggle to scale to large monolingual data, or inadequately handle translation ambiguity.The authors evaluate monolingual and crosslingual quality using word similarity and bilingual lexicon induction.Source: §1 · p. 1
- A Panlex bilingual dictionary supplies the bilingual signal, while an EM-style algorithm selects translations using context to handle polysemy.The method requires monolingual data in addition to the dictionary.Source: §1 · pp. 1–2
- The model achieves state-of-the-art bilingual lexicon induction, improves crosslingual word similarity, and is competitive on cross-lingual document classification.The introduction also reports that embedding-combination techniques improve monolingual word embeddings.Source: §1 · p. 2
- The paper evaluates embedding-combination methods for crosslingual and monolingual tasks and reports uniformly excellent results across monolingual, bilingual, and transfer settings.The reported contribution includes combining learned embedding matrices during training.Source: §1 · p. 2
2 Related work
Prior crosslingual embedding work uses bilingual resources, including parallel text, comparable documents, or dictionaries. These approaches differ in resource demands, treatment of translation ambiguity, and training strategy.
2 Related work
- Parallel-text approaches use word alignments to connect tokens, while sentence-level approaches learn representations for sentences and their translations but require large parallel corpora.The paper notes that such corpora are unavailable for many language pairs.Source: §2 · p. 2
- Comparable-text methods use Wikipedia documents or shared entries, but comparable bilingual data is scarce for many languages.One approach forms bilingual pseudo-documents from lemmatized nouns before learning representations.Source: §2 · p. 2
- Dictionary methods either randomly substitute translations without handling polysemy or filter to one-to-one pairs, whereas this paper uses all translations and disambiguates them during training.Filtering avoids ambiguity but discards dictionary information.Source: §2 · p. 2
- Prior work differs in training strategy: some train each language separately and combine embeddings later, while most others train multilingual models jointly.The paper adopts joint training, which prior work suggests performs better than cascade training.Source: §2 · pp. 2–3
3 Word2Vec
The paper builds on CBOW, which predicts a center token from its surrounding context. It uses negative sampling as an approximation to the training objective.
3 Word2Vec
- CBOW models the probability of a token at a position using logistic regression with a factored parameterization.The model learns context and word embedding matrices and maximizes the training corpus’s log-pseudo likelihood.Source: §3 · p. 3
- The context representation sums vectors for words in a window around the center, and the model uses separate context and word embedding matrices.The window has size k and is centered at the token position.Source: §3 · p. 3
- Negative sampling approximates the objective by distinguishing training data from negative examples drawn from a noise distribution.The objective uses p randomly drawn negative examples.Source: §3 · p. 3
4 Our Model
The model extends CBOW to learn crosslingual embeddings from monolingual corpora and a bilingual dictionary, selecting translations using context. It also explores joint prediction and ways to combine the word and context embedding spaces.
4 Our Model
- The model uses two monolingual corpora and a bilingual dictionary, a data condition described as less stringent than parallel or comparable texts.The motivation is that dictionary translations can capture lexical ambiguity, such as the different Italian translations of English “bank.”Source: §4 · p. 3
4.1 Dictionary replacement
- Context in one language predicts a centre word’s translation in the other, while the original context remains fixed.The selected dictionary translation is used with the context vector for a stochastic gradient update of the CBOW log-likelihood.Source: §4.1 · pp. 3–4
- The EM-inspired algorithm selects a dictionary translation using a semantic representation of the word and context, then updates the model with that translation.The algorithm operates over both monolingual corpora.Source: §4.1 · pp. 3–4, §4 · p. 3
4.2 Joint Training
- Joint training predicts both the word and its translation from monolingual context, with α = 0.5 setting the contribution of the two objective terms.Negative examples are drawn from the combined vocabulary unigram distribution.Source: §4.2 · p. 4
4.3 Combining Embeddings
- The model’s U space is argued to capture bilingual signal better and V monolingual regularities; the authors report a small improvement from using both embeddings over context alone.The paper also considers post-training interpolation, concatenation, and training-time regularization to bind the spaces; standard-EM expectations slightly degrade results.Source: §4.3 · p. 4
5 Experiment Setup
The experiments assess crosslingual lexicon induction, monolingual word similarity, and crosslingual document classification. They use Wikipedia-derived monolingual data and PanLex as the bilingual dictionary.
5 Experiment Setup
- The evaluation covers bilingual lexicon induction, word similarity, and crosslingual document classification.Word similarity uses WordSim353 and Rareword datasets.Source: §5 · pp. 4–5
5.1 Monolingual Data
- Each language’s monolingual training data is capped at the top 5 million sentences, about 100 million words.The passage notes that obtaining billions of monolingual words can be challenging for low-resource languages.Source: §5.1 · p. 5
5.2 Dictionary
- PanLex supplies the only bilingual correspondence, covering around 1,300 language varieties with about 12 million expressions, but its translations are often noisy.The dictionary draws on glossaries, dictionaries, and automatic inference from other languages.Source: §5.2 · p. 5
6 Bilingual Lexicon Induction
The section evaluates bilingual lexicon induction and reports that the combined model outperforms prior approaches. Joint prediction and EM-based translation selection improve the model, while Panlex performs better than Wiktionary in these experiments.
6 Bilingual Lexicon Induction
- Bilingual lexicon induction tests whether a source-language word's translation can be predicted in the target language, with a single gold translation per term.The task therefore requires discriminating among candidate translations.Source: §6 · p. 5
- Panlex yields much better model performance than Wiktionary, despite Wiktionary's higher translation quality and lower coverage.For the top 100,000 frequent English words, coverage is 42.1% for Panlex and 16.8% for Wiktionary, with 5.2 versus 1.9 translations on average.Source: §6 · p. 5
- The qualitative examples show the model retrieving the correct English translation and related words in both source and target languages.Table 1 lists the top ten nearest words for Spanish gravedad and Italian tassazione.Source: §6 · p. 5, Table 1 · p. 5
- The combined model outperforms previous approaches by a large margin on bilingual lexicon induction.Table 2 reports BLI from Spanish, Italian, and Dutch to English; Vulić and Moens's Dutch result is poor where their comparable corpus is small.Source: Table 2 · p. 6, §6 · p. 6
- The EM model that selects translations outperforms randomly selecting a dictionary translation.The comparison uses the Panlex dictionary and the authors report a significant margin.Source: §6 · p. 6
- The joint model further improves performance, especially for Dutch, with improvements consistent across three language pairs and both evaluation metrics.The authors also describe the combined model as effective at disambiguating morphology.Source: §6 · p. 6
7 Monolingual Word Similarity
The combined crosslingual embeddings outperform prior crosslingual embedding baselines on monolingual similarity and also surpass the same-setting monolingual CBOW baseline. The authors attribute much of the gain to combining word and context embeddings, with dictionary links also helping connect synonyms.
7 Monolingual Word Similarity
- The evaluation measures correlation with human similarity judgments on English and German WordSim353 and English RareWord.The datasets contain word pairs with annotator-provided similarity scores.Source: §7 · p. 6, Table 3 · p. 7
- The combined model outperforms Luong et al. and Gouws and Søgaard on monolingual similarity evaluation.Table 3 reports Spearman rank correlation on WS-De, WS-En, and RW-En and compares crosslingual embeddings with other baselines.Source: Table 3 · p. 7, §7 · p. 7
- The combined model performs better than the same-setting monolingual CBOW baseline and moves closer to monolingual state-of-the-art results on each dataset.The cited monolingual state-of-the-art methods use massive monolingual data, WordNet, or commercial search-engine outputs.Source: §7 · p. 7
- Most of the improvement over monolingual CBOW comes from combining word and context embeddings; dictionary links can also bring synonyms closer.Plane, airplane, and aircraft share the Italian translation aereo, encouraging proximity in the embedding space.Source: §7 · p. 7
8 Model selection
Model selection balances monolingual similarity against bilingual lexicon induction when combining embeddings. The authors tune interpolation and regularization, then use regularization with U as the output across language pairs.
8 Model selection
- As γ increases, English WordSim353 performance rises while BLI performance falls, so the authors choose γ = 0.5 as a compromise.The endpoints γ = 0 and γ = 1 use U and V alone, respectively.Source: §8 · p. 7
- The authors select δ = 0.01, and the cited passage places this selection alongside evaluation on the same BLI and WS-En settings.Source: §8 · p. 7
- Regularization with U as output is selected as the cross-task compromise and applied across language pairs without further tuning.Table 4 compares combining algorithms on English-Italian BLI and English monolingual WordSim353.Source: Table 4 · p. 8, §8 · p. 8
9 Crosslingual Document Classification
The section evaluates crosslingual document classification by training a classifier in one language and applying it directly in another. The model achieves competitive performance despite relying only on monolingual data and a bilingual dictionary rather than large bitext.
9 Crosslingual Document Classification
- The task trains a document classifier in a source language and applies it directly to classify target-language documents, without target-language document annotations.The setup uses Reuter RCV1/RCV2 data.Source: §9 · p. 8
- Documents are represented as tf.idf-weighted bags of word embeddings; an average perceptron trains on 1,000 source-language documents and tests on 5,000 target-language documents.The crosslingual embeddings place the target-language document representation in the same space as the source language.Source: §9 · p. 8
- The English–German embeddings use combined models, with monolingual data drawn from the RCV1/RCV2 corpus as in prior work.The combined models are described in equation (5).Source: §9 · p. 8
- Our model achieves competitive CLDC performance, while the other Table 5 models use large Europarl bitext.Table 5 reports results for both en→de and de→en; its caption identifies the MT baseline as phrase-based translation.Source: Table 5 · p. 8, §9 · p. 8
10 Conclusion
The paper introduces dictionary-based crosslingual embedding training with EM translation selection over monolingual corpora. It reports state-of-the-art bilingual lexicon induction and competitive results on word similarity and crosslingual document classification.
10 Conclusion
- The method achieves state-of-the-art bilingual lexicon induction and competitive results on monolingual word similarity and crosslingual document classification.The paper also reports that its combination techniques, especially regularization, are highly effective.Source: §10 · p. 8
- EM translation selection uses a large noisy dictionary to model polysemy and learn bilingual correspondences from monolingual corpora.The algorithm is described as efficient for training on massive amounts of monolingual data.Source: §10 · p. 8
- The combination techniques, especially regularization, are reported as highly effective and potentially useful for improving monolingual word embeddings.Source: §10 · p. 8