Source-linked AI summary
Cross-lingual Models of Word Embeddings: An Empirical Comparison
Shyam Upadhyay, Manaal Faruqui, Chris Dyer, Dan Roth
TL;DR
The paper addresses the lack of systematic evidence comparing cross-lingual embedding approaches with different supervision requirements. It unifies and evaluates four approaches across language pairs and intrinsic and extrinsic tasks, finding that richer supervision generally helps semantic tasks while cheaper supervision can remain competitive for some tasks.
Problem
A systematic comparison of cross-lingual embedding models using different forms of supervision was missing, limiting analysis of which approach suits a particular NLP task.
Method
The paper unifies four cross-lingual embedding approaches and evaluates them across four language pairs on monolingual and cross-lingual similarity, document classification, and dependency parsing.
Results
Richer cross-lingual supervision performs best overall on semantic tasks, whereas cheaper supervision is often competitive on monolingual similarity and syntactic dependency parsing.
Takeaways & Limitations
Semantic tasks benefit more from richer cross-lingual supervision than syntactic tasks within the evaluated settings.
Takeaways & Limitations
The comparison covers bilingual models trained on two languages at a time and does not cover all cross-lingual embedding approaches.
Abstract
from arXiv · showhide
Despite interest in using cross-lingual knowledge to learn word embeddings for various tasks, a systematic comparison of the possible approaches is lacking in the literature. We perform an extensive evaluation of four popular approaches of inducing cross-lingual embeddings, each requiring a different form of supervision, on four typographically different language pairs. Our evaluation setup spans four different tasks, including intrinsic evaluation on mono-lingual and cross-lingual similarity, and extrinsic evaluation on downstream semantic and syntactic applications. We show that models which require expensive cross-lingual knowledge almost always perform better, but cheaply supervised models often prove competitive on certain tasks.
1 Introduction
The paper systematically compares four cross-lingual embedding approaches that use different supervision forms across four language pairs and intrinsic and extrinsic tasks. Expensive supervision performs best on semantic tasks, while cheaper supervision can remain competitive for syntactic tasks.
- Research gap and scope: The study addresses the missing systematic comparison of cross-lingual embedding models requiring different alignment supervision.It evaluates four models trained on four language pairs.
- Evaluation: The evaluation covers monolingual and cross-lingual word similarity, cross-lingual document classification, and cross-lingual dependency parsing.These constitute intrinsic and extrinsic evaluations of semantic and syntactic applications.
- Unified framework: The models can be viewed within a general framework for inducing cross-lingual word embeddings.The framework includes parameters and suitably defined losses for bilingual embedding induction.
- Main findings: Expensive cross-lingual supervision using word or sentence alignments performs best on semantic tasks.The comparison contrasts this supervision with weaker forms such as context-agnostic translation dictionaries.
- Main findings: For cross-lingual dependency parsing, models using weaker supervision can be competitive with models requiring expensive supervision.The result indicates that supervision requirements differ in usefulness across downstream tasks.
2 Bilingual Embeddings
The paper presents a unified comparison of four bilingual embedding models that use different forms of cross-lingual supervision, ranging from word-aligned parallel data to comparable documents. The models induce embeddings through cross-lingual skip-gram links, sentence representation alignment, CCA projection, or pseudo-bilingual documents.
- 2.1 Bilingual Skip-Gram Model (BiSkip): BiSkip expands skip-gram contexts with word-alignment links and predicts context words across languages in both directions.Its cross-lingual objective complements monolingual skip-gram terms with bilingual prediction.
- 2.2 Bilingual Compositional Model (BiCVM): BiCVM learns word vectors by making representations of sentence-aligned sentence pairs similar while using randomly sampled negative pairs to avoid degeneracy.The model minimizes squared ℓ2 distance between aligned sentence representations and applies a noise-contrastive large-margin update.
- 2 Bilingual Embeddings: Four models are compared across distinct supervision requirements, from expensive word-aligned parallel corpora to cheaper comparable documents.BiSkip uses word alignments, BiCVM sentence alignments, BiCCA a bilingual lexicon, and BiVCD comparable documents.
- 2.3 Bilingual Correlation Based Embeddings (BiCCA): BiCCA projects independently trained monolingual vectors with CCA using translation pairs from a bilingual lexicon, producing bilingual-enriched representations.The paired vocabularies are equal-sized and corresponding words are translations of one another.
- 2.4 Bilingual Vectors from Comparable Data (BiVCD): BiVCD merges comparable documents into pseudo-bilingual documents using a length-ratio strategy, then trains skip-gram vectors over the merged corpus.In the experiments, aligned sentences are treated as comparable despite the method being designed for comparable corpora.
3 Data
The experiments train cross-lingual embeddings for four language pairs using parallel corpora from Europarl, WMT-related datasets, UN data, and FBIS. The English–Chinese corpus is additionally preprocessed with the Stanford Chinese Segmenter.
- 3 Data: Four language pairs are evaluated: English–German, English–French, English–Swedish, and English–Chinese.The pairs are abbreviated en-de, en-fr, en-sv, and en-zh.
- 3 Data: The English–German and English–Swedish experiments use the Europarl v7 parallel corpus.The English–French experiments combine Europarl with News Commentary and the UN corpus from WMT 2015.
- 3 Data: The English–Chinese experiments use the FBIS parallel corpus from the news domain and apply the Stanford Chinese Segmenter during preprocessing.Corpus statistics for all languages are reported in Table 1.
4 Evaluation
The evaluation compares cross-lingual embeddings across intrinsic similarity and extrinsic semantic and syntactic transfer tasks. Expensive supervision generally benefits semantic performance, while cheaper supervision can remain competitive for syntax.
- Evaluation setup: The study evaluates embeddings on English similarity, cross-lingual dictionary induction, document classification, and dependency parsing across four language pairs.The models use 200-dimensional embeddings and are compared under a shared evaluation setup.
- Monolingual evaluation: Bilingually trained English vectors outperform monolingual vectors on SimLex in every case except BiCCA on en-sv.SimLex evaluates Spearman correlation between human similarity rankings and cosine-based vector rankings.
- Monolingual evaluation: BiCVM performs best overall on SimLex, although its improvement over BiSkip and BiVCD is often insignificant.BiCVM and BiVCD require sentence- or document-aligned data rather than word-aligned parallel data.
- Monolingual evaluation: QVEC ranks BiSkip highest on average, followed by Mono, BiVCD, and BiCCA, while BiCVM ranks last.The differing rankings show that word-similarity performance alone does not capture all linguistic properties measured by QVEC.
- Extrinsic evaluation: BiSkip performs significantly better than most models for cross-lingual document classification, whereas BiCCA is best on dependency parsing and BiSkip is within one average point.Semantic transfer benefits from sentence- and word-level alignment, but syntactic transfer can favor cheaper supervision.
- Extrinsic evaluation: Compared with monolingual vectors, BiSkip and BiCCA improve dependency parsing on average, while BiCVM and BiVCD consistently perform worse.The authors attribute this pattern to word-alignment information capturing syntax more effectively than sentence-level contexts.
5 Qualitative Analysis
The qualitative analysis shows that supervision type shapes cross-lingual geometry: sentence-level models align translations more closely, whereas dictionary- and comparable-context models separate antonyms better.
- Cross-lingual comparability: BiSkip and BiCVM place English–French translation pairs almost at the same point in vector space.BiCCA and BiVCD produce more distant translated-word representations.
- Cross-lingual comparability: Sentence-level supervision helps explain why BiSkip and BiCVM produce the most comparable cross-lingual vectors.These models use more expensive sentence-level supervision than the other two models.
- Antonym separation: BiCCA and BiVCD separate antonyms such as peace–war and paix–guerre more effectively than BiSkip and BiCVM.In the sentence-trained models, these antonym pairs are represented very closely.
- Antonym separation: Parallel-sentence training can place antonyms close together because co-occurring English words also co-occur in their French translations.Bilingual dictionary and comparable-sentence-context supervision instead helps pull synonyms and antonyms apart.
6 Discussion
The discussion frames the study as a unified, unbiased comparison of bilingual embedding models while acknowledging coverage limits from model availability and the two-language setting.
- Contribution and scope: The paper formulates cross-lingual word-vector learning in a unified framework and compares existing models in an unbiased manner.The selected models can be trained on two languages at a time.
- Limitations: The comparison does not cover multilingual models trained on more than two languages.Ammar et al. (2016) is cited as an example of the excluded setting.
- Limitations: The paper does not cover all cross-lingual embedding approaches because some lack public code and BilBOWA’s available code produced inferior results.The authors report that toolkit issues could not be resolved after contacting the original authors.
- Model selection: The included models are representative of different supervision forms, including categories shared with BilBOWA, cross-lingual auto-encoders, and CCA extensions.This selection was motivated by comparing distinct forms of cross-lingual supervision.
7 Conclusion
The conclusion presents a systematic intrinsic and extrinsic comparison under a unified representation. Results indicate that richer supervision benefits cross-lingual semantic tasks most, while cheaper or word-level supervision remains competitive elsewhere.
- Contribution: The study provides a systematic comparison of cross-lingual embedding methods across several intrinsic and extrinsic downstream NLP tasks.Its methods span diverse forms of cross-lingual supervision.
- Intrinsic evaluation: On monolingual word similarity, cheaper supervision such as BiVCD performs almost on par with models using expensive supervision.This is an intrinsic-task result.
- Semantic tasks: For cross-lingual document classification and dictionary induction, the model with the most informative supervision performs best.These are identified as cross-lingual semantic tasks.
- Syntactic tasks: For dependency parsing, word-alignment-supervised models perform slightly better than the alternatives overall.The conclusion contrasts this syntactic pattern with the stronger benefit of richer supervision for semantic tasks.
- Overall pattern: Semantic tasks benefit more from richer cross-lingual supervision than syntactic tasks.This is the paper’s overall conclusion about supervision and task type.