Source-linked AI summary
BilBOWA: Fast Bilingual Distributed Representations without Word Alignments
Stephan Gouws, Yoshua Bengio, Greg Corrado
TL;DR
Bilingual embedding methods face limits from slow training and dependence on narrow, expensive parallel data. BilBOWA combines monolingual training with a sampled bag-of-words cross-lingual loss from sentence-aligned data, achieving state-of-the-art or improved results across document classification and word translation. Its main trade-off is that it remains slower to use than offline alignment methods.
Problem
Existing bilingual embedding methods are either too slow to train at scale or limited to expensive, domain-biased parallel data.
Method
BilBOWA jointly trains monolingual embeddings and uses a sampled bag-of-words cross-lingual loss on sentence-aligned raw text without word alignments.
Results
BilBOWA achieves state-of-the-art English-German cross-lingual document classification, improves English-Spanish word translation, and obtains up to three orders of magnitude speedup.
Takeaways & Limitations
BilBOWA enables efficient large-scale cross-lingual training over monolingual text while retaining strong performance on document classification and lexical translation.
Takeaways & Limitations
Despite its speedup, BilBOWA remains slower to use than offline translation-matrix and multilingual-CCA methods.
Abstract
from arXiv · showhide
We introduce BilBOWA (Bilingual Bag-of-Words without Alignments), a simple and computationally-efficient model for learning bilingual distributed representations of words which can scale to large monolingual datasets and does not require word-aligned parallel training data. Instead it trains directly on monolingual data and extracts a bilingual signal from a smaller set of raw-text sentence-aligned data. This is achieved using a novel sampled bag-of-words cross-lingual objective, which is used to regularize two noise-contrastive language models for efficient cross-lingual feature learning. We show that bilingual embeddings learned using the proposed model outperform state-of-the-art methods on a cross-lingual document classification task as well as a lexical translation task on WMT11 data.
1. Introduction
The paper motivates bilingual embeddings as a way to learn language-invariant syntactic and semantic features while addressing the scalability, data, and domain limitations of existing methods. BilBOWA uses large monolingual corpora plus limited sentence-aligned data without word alignments.
- Motivation: Unsupervised word representations can learn syntactic and semantic features that generalize across tasks and languages.Such representations are useful for transferring limited label information from high-resource to low-resource languages.
- Limitations of prior methods: Existing bilingual embedding methods are either too slow for large-scale application or restricted to parallel data.Parallel-only training also limits data volume and introduces domain bias because parallel corpora often cover narrow domains.
- BilBOWA: BilBOWA learns bilingual word embeddings from essentially unlimited monolingual raw text and a limited sample of sentence-aligned raw-text data without word-level alignments.The sentence-aligned data supplies the bilingual signal used to align embeddings during monolingual training.
- BilBOWA: The BilBOWA-loss is a computationally efficient sampled cross-lingual objective that aligns monolingual embeddings online using sampled bag-of-words sentence-aligned data.Its design supports large-scale training while avoiding word-alignment estimation.
- Results: BilBOWA outperforms current state-of-the-art methods on document classification and lexical translation, reducing training time from several days to minutes or hours.The paper evaluates the induced embeddings on both cross-lingual document classification and lexical translation.
2. Learning Cross-lingual Word Embeddings
Cross-lingual embeddings combine monolingual distributional learning with a cross-lingual objective so related words across languages receive similar representations. The paper contrasts offline alignment, parallel-only, and jointly trained approaches, motivating BilBOWA's efficient regularization strategy.
- Cross-lingual embedding goals: Monolingual embedding algorithms learn word vectors from raw-text context by bringing frequently co-occurring words closer in the embedded space.These representations capture useful syntactic and semantic information from large datasets.
- Cross-lingual embedding goals: Cross-lingual training additionally encourages similar words across languages to receive similar representations, enabling transfer of classifiers to languages with limited labeled data.The monolingual objectives model each language while the cross-lingual objective links their representations.
- Offline alignment: Offline alignment trains separate monolingual embeddings and then projects translation pairs using a dictionary, but it requires an accurate dictionary and typically one translation per word.The latter restriction ignores polysemy across languages.
- Parallel-only models: Parallel-only models learn from sentence-aligned data alone, making them fast with efficient training but limiting them to expensive, domain-biased parallel corpora.Their training data may not match the style or register of the target application domain.
- Jointly trained models: Jointly trained models optimize monolingual objectives while enforcing a cross-lingual regularizer during training.The regularizer minimizes a similarity-weighted distance between word representations across languages.
- Computational challenge: The prior jointly trained Klementiev model is slow because its neural language-model softmax and cross-lingual objective can scale with vocabulary size and all cross-vocabulary interactions.The worst-case cross-lingual computation scales as the product of the two vocabularies.
3. The BilBOWA Model
BilBOWA jointly trains scalable monolingual embeddings with a sampled cross-lingual loss derived from sentence-aligned raw text, avoiding word alignments and reducing computational cost. Parallel subsampling improves the approximation by reducing frequent-word over-regularization.
- Model overview: BilBOWA combines monolingual objectives with a cross-lingual regularizer to learn bilingual embeddings from large monolingual corpora and limited sentence-aligned data.The model does not require word-level alignments and can extend trivially to multilingual embeddings.
- Monolingual objective: Noise-contrastive training replaces the vocabulary-dependent softmax, allowing monolingual updates to scale independently of vocabulary size.The approach scores observed word-context combinations against randomly sampled unlikely combinations.
- Cross-lingual objective: The sampled BilBOWA-loss reduces cross-lingual computation to words in observed sentence-pair samples instead of the naive O(|V e| · |V f|) interactions.It uses sampled bag-of-words sentence-aligned data and avoids estimating word alignments.
- Cross-lingual objective: The BilBOWA-loss minimizes a sampled L2-loss between mean bag-of-words vectors of parallel sentences, approximating global alignment statistics through local co-occurrences.Used alone, this objective collapses embeddings, so it functions as a regularizer alongside monolingual losses.
- Parallel subsampling: Parallel subsampling discards words according to unigram frequency, flattening the distribution and reducing over-regularization of frequent words.Without subsampling, frequent words cluster near the origin; the effect is largely reduced with subsampling.
4. Implementation and Training Details
The implementation uses asynchronous C-based training with separate monolingual and cross-lingual threads sharing embedding parameters. Training uses sampled updates on Wikipedia and Europarl data, with update clipping improving stability and speed.
- Implementation: The C implementation launches separate skipgram threads for each language and a cross-lingual thread that asynchronously accesses shared embeddings.Training uses online asynchronous stochastic gradient descent.
- Training data: Monolingual training uses freely available pretokenized Wikipedia datasets, while cross-lingual training uses the sentence-aligned Europarl v7 corpus without prior word alignment.The cross-lingual thread samples random parallel sentence pairs.
- Optimization: The learning rate is set to 0.1 with linear decay, and gradients are computed from the BilBOWA square loss and skipgram log-linear objectives.Each thread updates parameters asynchronously.
5. Experiments
The experiments evaluate BilBOWA on cross-lingual document classification and word translation, showing strong accuracy and substantial training-speed advantages over prior methods.
- Evaluation tasks: The evaluation covers cross-lingual document classification for semantic transfer and word translation for fine-grained lexical transfer.The classification setup directly transfers a source-language classifier to target-language documents represented by tf-idf-weighted sums of embedding vectors.
- Cross-lingual document classification: BilBOWA significantly improves upon Klementiev et al.’s classification results while training in 6 minutes instead of 10 days, a 2,400-fold speedup.The comparison uses 40-dimensional embeddings and the same training data.
- Cross-lingual document classification: BilBOWA outperforms BiCVM on both classification directions and reaches 75% on German-to-English, surpassing the BAE result on that task.Reported BAE accuracies are 91.8 for English-to-German and 72.8 for German-to-English; BiCVM reports 83.7 and 71.4.
- WMT11 word translation: The word-translation evaluation uses English and Spanish Wikipedia embeddings with Europarl sentence-aligned data and compares top-1 and top-5 translation accuracy.Table 2 measures accuracy out of 100, with higher values better, using distances in the embedding space to select translation pairs.
- WMT11 word translation: BilBOWA improves absolute translation accuracy by 6% from English to Spanish and 9% from Spanish to English over the prior state of the art.The results indicate that raw-text sentence-aligned data can support fine-grained translation equivalences without word-level alignments or training dictionaries.
6. Discussion
BilBOWA’s sampled L2 bag-of-words loss drives its speedup and scalability, while its remaining runtime cost reflects a trade-off for learning finer-grained relationships from more data.
- Efficiency and accuracy: The sampled L2 bag-of-words cross-lingual loss is the main source of BilBOWA’s speedup over the Klementiev model.It allows training to scale to much larger datasets, which the discussion connects with more accurate features.
- Training design: Asynchronous training significantly speeds up learning without noticeably affecting embedding quality, while parallel subsampling improves feature accuracy.Update clipping was required for asynchronous training, especially at larger embedding dimensionalities.
- Training design: Parallel subsampling is especially important for frequent words and improves accuracy in both monolingual and cross-lingual settings.The discussion attributes this benefit to more accurate training for frequent words.
- Trade-offs: BilBOWA remains much slower than offline translation-matrix and multilingual-CCA methods, despite its speedup.The model is presented as a trade-off that can learn finer-grained cross-lingual relationships and use much larger monolingual datasets than parallel-only methods.
7. Conclusion
BilBOWA learns bilingual representations from monolingual raw text and limited sentence-aligned data without word alignments or dictionaries, efficiently supporting large-scale training and strong cross-lingual performance.
- Conclusion: BilBOWA induces bilingual word representations from monolingual raw text and limited parallel data without requiring word alignments or dictionaries.It combines efficient monolingual embedding training with a sampled cross-lingual objective whose per-step computation scales with sentence length.
- Conclusion: BilBOWA achieves state-of-the-art English-German cross-lingual document classification with up to three orders of magnitude speedup.It also improves upon the previous state of the art in English-Spanish word translation.