PaperlayerView exact revision on arXiv

Source-linked AI summary

A Survey Of Cross-lingual Word Embedding Models

Sebastian Ruder, Ivan Vulić, Anders Søgaard

arXiv:1706.04902v4cs.CLcs.LG

TL;DR

Cross-lingual word embeddings support multilingual meaning comparison and transfer, but the literature contains many models with differing data requirements and formulations. This survey develops a comprehensive typology, compares objectives and supervision signals, and shows that seemingly different approaches often optimize similar objectives or are equivalent modulo optimization choices. It also reviews evaluation, multilingual extensions, and open challenges.

  • Problem

    Cross-lingual NLP needs representations that transfer lexical knowledge across languages, especially to low-resource languages, while existing embedding models vary in data requirements and formulation.

  • Method

    The survey organizes models by alignment data, standardizes notation, compares objective functions, and connects word-level approaches through proofs of approximate equivalence.

  • Results

    Many cross-lingual embedding models optimize roughly the same objectives, with differences chiefly associated with data, regularization, hyper-parameters, and optimization strategies.

  • Takeaways & Limitations

    Mapping-based approaches are often preferred for word-aligned parallel data because they are conceptually simple, easy to use, and computationally inexpensive.

  • Takeaways & Limitations

    Unsupervised methods relying on approximately isomorphic monolingual spaces fail for distant languages, where distantly supervised seed lexicons may be preferable.

Abstract

from arXiv · show

Cross-lingual representations of words enable us to reason about word meaning in multilingual contexts and are a key facilitator of cross-lingual transfer when developing natural language processing models for low-resource languages. In this survey, we provide a comprehensive typology of cross-lingual word embedding models. We compare their data requirements and objective functions. The recurring theme of the survey is that many of the models presented in the literature optimize for the same objectives, and that seemingly different models are often equivalent modulo optimization strategies, hyper-parameters, and such. We also discuss the different ways cross-lingual word embeddings are evaluated, as well as future challenges and research horizons.

1. Introduction

Cross-lingual word embeddings represent words from multiple languages in a joint space, supporting multilingual meaning comparison and transfer to low-resource languages. This survey organizes existing models, shows their shared objectives and equivalences, and reviews evaluation, multilingual extensions, and future challenges.

  • Motivation: Cross-lingual word embeddings place words from different languages in a joint representation space.They support reasoning about word meaning across languages and transfer between resource-rich and low-resource languages.
  • Organization: The survey standardizes terminology and notation to make similarities and differences among existing approaches easier to recognize.It uses a common framework to compare models across the literature.
  • Survey scope: The survey proposes a typology that characterizes cross-lingual embedding models by their differentiating features.The typology provides a compact overview and organizes models according to their data requirements and alignment types.
  • Shared objectives: Many cross-lingual embedding models optimize nearly the same objective functions, with variation arising from data, monolingual and regularization objectives, and optimization procedures.The survey also proves that mapping-based, joint, and pseudo-bilingual word-level approaches can optimize roughly the same objective.
  • Further contributions: It critically examines evaluation methods, describes multilingual extensions, and outlines outstanding challenges and future research directions.These contributions extend beyond model taxonomy to evaluation and research planning.

2. Notation and Terminology

The survey establishes shared notation for word, sentence, document, embedding, alignment, and transformation objects. It also expresses most cross-lingual models through a common objective combining monolingual losses with regularization, while emphasizing optimization order as a major distinction.

  • Notation: Embedding matrices Xℓ represent the vocabularies of languages ℓ in d-dimensional spaces.Rows or columns indexed by i refer to word embeddings, while x_wi denotes the embedding associated with word wi.
  • Notation: The survey distinguishes source and target languages, word embeddings, context embeddings, translation mappings, and word co-occurrence matrices.Mapping approaches learn W_s→t to transform source embeddings using translated seed words.
  • Alignment notation: Sentence and document alignment notation represents aligned units with vectors y and z in R^d.The notation separately identifies source and target sentences or documents and their representations.
  • Objective functions: Most cross-lingual embedding objectives combine language-specific monolingual losses Lℓ with a cross-lingual regularization term Ω.This formulation condenses differences between approaches into their regularization assumptions.
  • Optimization: Optimization strategy is a key differentiating factor because joint optimization of multiple non-convex losses is difficult.Most methods optimize losses stepwise with some variables fixed, an approximate procedure without a guarantee of reaching even a local optimum.

3. Monolingual Embedding Models

Cross-lingual embedding methods commonly extend monolingual embedding objectives, so the survey reviews matrix-factorization, predictive, and margin-based approaches. These models differ in training formulation, while several implicitly factorize related co-occurrence statistics.

  • Matrix factorization: LSA learns dense word representations by factorizing a PMI-transformed word-word co-occurrence matrix.Singular value decomposition produces reduced-dimensional word embeddings from the top k singular components.
  • Margin-based models: Max-margin loss trains embeddings to score correct word sequences higher than incorrect sequences.The model compares scores for corpus-observed windows against corrupted alternatives.
  • Predictive models: Skip-gram with negative sampling predicts surrounding context words from a target word while approximating the expensive softmax.Negative sampling makes training more computationally efficient and implicitly factorizes a shifted PMI matrix.
  • Predictive models: CBOW jointly predicts a center word from the surrounding context window and is typically trained with negative sampling.Its objective aggregates the embeddings of context words before predicting the target.
  • Matrix factorization: GloVe learns word and context embeddings by matching their dot product to logarithmic co-occurrence counts.With fixed bias choices, GloVe is equivalent to factorizing a PMI matrix shifted by log |C|.

4. Cross-Lingual Word Embedding Models: Typology

The survey’s typology classifies cross-lingual embedding models by alignment level and data comparability. It uses these data requirements to explain performance variation and organize word-, sentence-, and document-level approaches.

  • Motivation: The survey argues that bilingual supervision data often matters more for performance than the underlying model architecture.Its typology therefore emphasizes data requirements as a major source of variation among methods.
  • Typology dimensions: The typology uses alignment level and data comparability as its two main dimensions.Alignment may involve words, sentences, or documents, while data may be parallel translations or merely comparable sources.
  • Survey organization: Tables and figures summarize the data types and classify models into an exhaustive typology.The survey also provides detailed overviews of monolingual objectives and regularization terms.
  • Word alignment: Word-level alignment commonly uses bilingual dictionaries or automatically aligned parallel corpora.Comparable word-level data is more plentiful but has been used less often and can involve images or other modalities.
  • Sentence alignment: Sentence-level alignment typically relies on parallel corpora such as Europarl, while comparable sentence data can connect related multilingual materials.Examples include translations and image-caption resources that are related without being direct translations.
  • Document alignment: Document-level parallel alignment requires translated documents and is rare, whereas comparable document alignment can use topic- or class-aligned documents.Wikipedia and multilingual classification datasets are examples of comparable document-level resources.

5. A Brief History of Cross-Lingual Word Representations

Cross-lingual word representations predate modern word embeddings and draw on earlier approaches using linguistic labels, lexical contexts, aligned data, and limited bilingual supervision. The survey connects these historical architectures to current embedding models and organizes them by bilingual data requirements.

  • Cross-lingual word representations existed before the popular introduction of word embeddings, with many current research ideas having earlier precedents.
  • Earlier approaches used abstract linguistic labels, cross-lingual word clusters, syntactic or part-of-speech contexts, and lexical features for transfer and representation learning.
  • Traditional context-counting methods mapped high-dimensional co-occurrence vectors into a shared space using seed bilingual dictionaries containing paired context words.
  • Bootstrapping in traditional context-counting approaches anticipated iterative self-learning techniques that reduce bilingual dictionary supervision in mapping-based models.
  • Sentence-aligned word translation probabilities from statistical machine translation can serve as cross-lingual semantic similarity functions instead of embedding cosine similarity.
  • Historical cross-lingual architectures also included pseudo-bilingual corpora, document alignments, latent topical spaces, and document-aligned Wikipedia structures.

6. Word-Level Alignment Models

Word-level alignment models learn cross-lingual representations from aligned words using mapping-based, pseudo-multilingual-corpus, or joint approaches. The survey argues that these approaches are often equivalent apart from optimization strategies, while mapping variants differ in transformations, supervision, refinement, and retrieval.

  • Word-level alignment methods: Word-level alignment models include mapping-based, pseudo-multilingual-corpus, and joint methods, which respectively transform spaces, mix languages in constructed corpora, or jointly optimize monolingual and cross-lingual objectives.
  • Word-level alignment methods: These three approaches are equivalent modulo optimization strategies, according to the survey.
  • Evaluation: Bilingual lexicon induction evaluates models by retrieving target-language words for source queries, typically using nearest-neighbour cosine similarity in a shared space.
  • Mapping methods: Mapping-based approaches learn a transformation from independently trained monolingual spaces and vary by mapping method, seed lexicon, refinement, and nearest-neighbour retrieval.
  • Mapping methods: The four mapping families are regression, orthogonal, canonical, and margin methods, differing in whether they maximize similarity, constrain transformations, project both languages, or maximize translation margins.
  • Regression methods: Regression methods learn a linear transformation by minimizing squared Euclidean distance between mapped source and target seed-word embeddings.
  • Orthogonal methods: Orthogonal methods constrain W so that W⊤W = I, preserving length normalization; the survey reports that this constraint improves over basic regression and is most commonly adopted.
  • Canonical methods: Canonical methods learn separate source and target transformations into a new shared space by maximizing correlation between projected embeddings.

The Seed Lexicon

Mapping-based approaches depend on seed bilingual lexicons, which can be taken from existing resources, induced through weak supervision, or learned without bilingual supervision. The survey describes a progression toward reducing or eliminating manually supplied seed pairs.

  • Mapping-based approaches use seed lexicons as a core component for learning a joint cross-lingual embedding space.
  • Seed lexicons comprise off-the-shelf bilingual lexicons, weakly supervised lexicons, and learned bilingual lexicons.
  • Off-the-shelf approaches initially used frequent-word lexicons, with later work showing that as few as 25 seed pairs can suffice.
  • Weakly supervised lexicons can use cognates, shared numerals, or identically spelled strings and have produced results competitive with off-the-shelf lexicons.
  • Unsupervised methods learn an initial seed lexicon without bilingual supervision and, so far, are all based on the mapping approach.

The Refinement

The survey shows that refinement, retrieval, retrofitting, pseudo-bilingual, joint, and mapping-based approaches often share objectives or become equivalent under particular settings. Their practical differences instead reflect data requirements, optimization choices, computational cost, and coverage of unseen words.

  • The Refinement: Iterative refinement reuses induced translation pairs to learn improved mappings, but one refinement step is often sufficient because later iterations add noisier translations.Refinement is especially relied upon when the seed lexicon is small or learned unsupervised.
  • The Retrieval: CSLS corrects nearest-neighbour retrieval by increasing isolated-vector similarity and decreasing hub similarity, improving bilingual lexicon induction accuracy.It is now commonly used instead of cosine similarity; phrase-based MT retrieval has been shown to outperform CSLS significantly.
  • The Retrieval: Retrofitting captures relations beyond a linear transformation but only refines words present in external semantic resources and backs off to distributional estimates for others.Post-specialisation methods extend this idea by learning a global transformation for words unobserved in those resources.
  • 6.1.2 Word-level Approaches based on Pseudo-bilingual Corpora: Pseudo-bilingual methods optimize an objective similar to mapping-based methods but require training embeddings from scratch on concatenated monolingual corpora, making them more expensive.Mapping-based methods reuse pretrained embeddings and learn the mapping efficiently.
  • 6.1.4 Sometimes Mapping, Joint and Pseudo-bilingual Approaches are Equivalent: Mapping, joint, and pseudo-bilingual approaches can optimize roughly the same objectives and be equivalent modulo optimization strategies and hyper-parameters.The survey uses a common notation to expose these similarities across apparently different models.
  • 6.1.4 Sometimes Mapping, Joint and Pseudo-bilingual Approaches are Equivalent: The survey proves that pseudo-bilingual sampling is equivalent in the limit to Constrained Bilingual SGNS, and retrofitting is equivalent to it when α = Ω∞.These equivalences hold under specified assumptions or hyper-parameter settings.

6.2 Word-Level Alignment Methods with Comparable Data

Comparable-data methods use shared signals other than word-level parallel alignments, including images and part-of-speech equivalence. The survey contrasts these approaches with parallel-data methods and notes that multimodal signals generally complement text rather than replace it.

  • Comparable feature models: Comparable feature models exploit cross-lingual comparability in features such as part-of-speech tags rather than word-level parallel alignments.One approach creates a pseudo-cross-lingual corpus by replacing words with words having equivalent part-of-speech tags.
  • Grounding language in images: Language-grounding models use images as shared cross-lingual signals to estimate similarity between words in different languages.Image sets can be associated with each word, and visual features can provide cross-lingual similarity information.
  • Grounding language in images: Signals from modalities beyond text are currently under-explored and typically work best as additional information rather than as the sole source.The survey mentions auditory and olfactory signals as examples beyond vision.
  • Summary: Parallel word-level data is generally preferred because methods using it have been shown to outperform methods leveraging comparable data.The survey also describes parallel data as relatively easy to obtain for most language pairs.

7. Sentence-Level Alignment Methods

Sentence-level alignment methods learn cross-lingual representations from aligned sentences by adapting monolingual objectives or reconstructing and aligning sentence representations. The survey also highlights that sentence-aligned data is costly for low-resource languages and that model variants differ in their use of monolingual objectives and alignment constraints.

  • Sentence-aligned parallel data is widely available for European languages but more expensive than word-aligned data for low-resource languages or new domains.Sentence alignment requires fine-grained supervision.
  • Bilingual skip-gram: Bilingual skip-gram models jointly optimize monolingual SGNS losses with cross-lingual regularization, while predicting words in source and target sentences.
  • Sentence-level approaches often differ mainly in whether they add monolingual objectives, how they encode alignment, and whether losses are optimized jointly.
  • Compositional sentence models: Compositional sentence models sum word embeddings into sentence representations and train aligned sentences to be close.
  • Extensions: Document composition extends sentence models recursively, with bigram-based nonlinear composition outperforming addition on large datasets but underperforming it on smaller data.
  • Bilingual autoencoder: Bilingual autoencoders reconstruct each source sentence and its translation using language-specific encoder and decoder layers.

8. Document-Level Alignment Models

Document-level alignment models use comparable or pseudo-bilingual documents, concepts, or extensions of sentence-level methods to induce cross-lingual embeddings. The survey concludes that document alignment has so far provided little additional information beyond sentence-level alignment.

  • Comparable document-level alignment is appealing because it is often cheaper to obtain than sentence-aligned parallel data.
  • Pseudo-bilingual document-aligned corpora: Pseudo-bilingual approaches merge words from aligned documents into mixed-language corpora, using random or length-ratio shuffling strategies.
  • Concept-based models: Concept-based methods represent words through shared multilingual topics, concepts, or the Wikipedia concepts they describe.
  • Extensions of sentence-alignment models: Extensions of sentence-alignment models constrain word, sentence, and paragraph representations while jointly optimizing monolingual objectives and cross-lingual regularization.
  • Document-level alignment has so far provided little additional information compared with sentence-level alignment methods.
  • Most approaches combine monolingual losses with regularization, although image-grounded methods instead use pretrained image features and similarity heuristics.

9. From Bilingual to Multilingual Training

Multilingual training extends bilingual embedding methods to shared spaces across several languages, often using a pivot language or multilingual contexts. These extensions enable transfer between language pairs without direct seed lexicons and can provide additional supervision and reduce data sparseness.

  • Multilingual training adds supervision signals from multiple languages, producing better word embedding estimates and supporting multi-source transfer.
  • Mapping-based approaches: Pivot mapping projects L−1 monolingual spaces into one chosen pivot space using L−1 seed translation dictionaries.
  • Mapping-based approaches: Multilingual mapping can induce a shared space for language pairs lacking directly usable bilingual resources.English is commonly selected as the pivot.
  • Mapping-based approaches: Smith et al. report Spanish-Catalan P@1 of 0.82 without a seed lexicon, versus 0.70 averaged across Spanish-English and Catalan-English spaces.
  • Pseudo-bilingual and joint approaches: Pseudo-multilingual models corrupt monolingual corpora so words from multiple languages become context words, while joint models predict multilingual translations from shared contexts.
  • Sentence-level extensions: Multilingual TransGram predicts all words in aligned sentences rather than relying on word alignments, using English as a pivot to reduce bilingual data requirements.

10. Evaluation

Cross-lingual embeddings are evaluated through intrinsic and extrinsic tasks, benchmarks, and downstream applications. The survey highlights limitations in intrinsic evaluations, especially their weak relationship to downstream performance and unresolved dataset issues.

  • Evaluation Scope: Evaluation spans intrinsic and extrinsic tasks, with benchmarks and downstream applications providing complementary evidence about embedding quality.The survey discusses word alignment, dictionary induction, and cross-lingual transfer alongside benchmark resources and associated challenges.
  • Intrinsic Tasks: Intrinsic tasks such as word similarity and multiQVEC evaluate representation characteristics in controlled settings.Word similarity uses human-annotated multilingual word pairs, while multiQVEC correlates embeddings with multilingual linguistic resources.
  • Intrinsic Tasks: Word similarity datasets extend monolingual resources across languages but retain subjectivity, confusion between similarity and relatedness, and limited task relevance.They also lack standardized splits, handle polysemy poorly, and correlate only weakly with downstream performance.
  • Intrinsic Tasks: MultiQVEC and multiQVEC+ quantify linguistic content by correlating embeddings with manually annotated multilingual resources.MultiQVEC+ uses canonical correlation analysis, while multiQVEC extends QVEC with multilingual supersense annotations.
  • Extrinsic Tasks: Extrinsic word alignment and bilingual dictionary induction use nearest-neighbor search in the shared cross-lingual embedding space.Dictionary induction evaluates retrieved source-target pairs against gold dictionaries, which are especially valuable where manually constructed resources are scarce.

11. Applications

Cross-lingual embeddings support transfer across languages in classification, parsing, tagging, dialogue, entity linking, and sentiment tasks. Evaluation evidence indicates that supervision type, feature choice, task, and alignment method all affect observed performance.

  • Applications: Cross-lingual embeddings enable models trained in one language to transfer to another through a common representation space.They can serve directly as features in multilingual NLP models.
  • Applications: Applications include document classification, dependency parsing, POS tagging, named entity recognition, super-sense tagging, semantic parsing, and discourse parsing.These tasks transfer models or representations across languages using shared embeddings.
  • Applications: Cross-lingual embeddings support dialogue state tracking, entity linking, and sentiment analysis in multilingual settings.They provide inputs for dialogue models, link non-English mentions to English Wikipedia, and support multilingual review classification.
  • Applications: Cross-lingual embeddings can improve dialogue state tracking in German and Italian without language-specific training data.A multilingual model trained on all three languages improves tracking performance in English, German, and Italian.
  • Evaluation Challenges: Embedding evaluation remains controversial because many metrics reuse co-occurrences, contexts, or lexical resources involved in embedding induction.Behavioral observations such as gaze or fMRI data are presented as a task-independent alternative.
  • Benchmarks: Benchmark studies find that cheaper supervision can approach more expensive word-level supervision on word similarity, while finer-grained tasks often favor word- and sentence-level information.They also emphasize developing new data sources and leveraging comparable data for low-resource languages and domains.
  • Benchmarks: Performance often depends on the application: orthogonal mapping results on bilingual lexicon induction correlate strongly with some downstream tasks, whereas non-orthogonal methods correlate weakly.Methods optimized directly for bilingual lexicon induction can show reduced downstream performance.

12. General Challenges and Future Directions

The survey identifies unresolved challenges in making cross-lingual embeddings robust across morphology, expressions, meanings, domains, and languages, while also highlighting computational and modeling limits. Future work must address scarce data, richer linguistic units, stronger mappings, and whether all desired constraints can be satisfied.

  • Subword-level information: Subword information could better represent rare and internally complex word forms in morphologically rich languages, but has not yet been incorporated into cross-lingual word representations.Composing representations from lemmas and morphemes is proposed as a relevant direction.
  • Multi-word expressions: Multi-word expressions remain difficult because their meanings can be non-compositional, and they have received scarce attention in cross-lingual settings.Examples include “ad hoc” and “kick the bucket.”
  • Function words: Cross-lingual embeddings model conceptual meaning well but inadequately capture functional meaning, an important limitation for dialogue and pragmatics.The survey illustrates this with the distinction between “Give me a pencil” and “Give me that pencil.”
  • Polysemy: Polysemy can produce O(n × m) false nearest neighbors when source and target languages contain n and m bad word embeddings, respectively.Cross-lingual multi-sense embeddings are identified as a promising response to this amplified problem.
  • Specialized domains: Specialized domains and low-resource languages often lack parallel data, motivating robust representations from few parallel examples and greater use of comparable corpora.The survey also notes limited progress on composing word embeddings into higher-level sentence and document representations.
  • Feasibility: Finding an embedding that satisfies all inter-lingual and intra-lingual constraints is difficult: checking a model is linear in the number of constraints, while existence is NP-hard.Many current methods minimize losses for constraint violations without guaranteeing that the final model satisfies every constraint.
  • Non-linear mapping: Linear mapping can outperform nonlinear alternatives, yet assuming one global linear transformation is overly simplistic because language-specific and neighborhood-specific differences vary across embedding spaces.The survey reports that no model had yet leveraged this intuition into a more effective mapping model.
  • Robust unsupervised approaches: Unsupervised methods assume approximately isomorphic monolingual spaces, an assumption that fails especially for distant languages; identical-string seed lexicons may then be preferable.A large-scale evaluation identified Artetxe et al. (2018a) as the most robust completely unsupervised approach among those discussed.

13. Conclusion

The survey organizes cross-lingual word embedding research through standardized notation and a typology that exposes similarities among models. It also connects models formally, reviews evaluation and multilingual extensions, and identifies future challenges.

  • Conclusion: The survey introduces standardized notation and a typology demonstrating similarities among many cross-lingual word embedding models.This framework supports the survey’s comparative overview.
  • Conclusion: It provides proofs connecting different word-level embedding models and describes evaluation methods and extensions from bilingual to multilingual settings.The survey also outlines challenges and future research directions.
Loading 1706.04902v4…