PaperlayerView exact revision on arXiv

Source-linked AI summary

Cross-Lingual Alignment of Contextual Word Embeddings, with Applications to Zero-shot Dependency Parsing

Tal Schuster, Ori Ram, Regina Barzilay, Amir Globerson

arXiv:1902.09492v2cs.CLcs.LG

TL;DR

Contextual embeddings offer richer representations but are difficult to align across languages because their vectors vary with context. The paper aligns context-independent anchors and transfers that mapping to contextual spaces, improving zero-shot and few-shot dependency parsing over prior state of the art. The method is task-independent in core design, but its broader generalization is conjectured rather than established.

  • Problem

    Aligning multilingual contextual embeddings is difficult because a translated word pair corresponds to multiple context-specific vectors, limiting direct use of dictionary supervision.

  • Method

    The method factorizes contextual spaces into context-independent anchors and context-dependent parts, learns cross-lingual alignment between anchors, and applies it to contextual embeddings.

  • Results

    The method consistently outperforms previous transfer methods in zero-shot and few-shot dependency parsing, including a 6.8 LAS-point absolute improvement over the prior state of the art.

  • Takeaways & Limitations

    Anchor-based alignment supports multilingual contextual transfer with or without dictionaries and for low-resource languages, while improving zero-shot and few-shot dependency parsing.

  • Takeaways & Limitations

    Generalization beyond the paper’s core task is conjectured rather than demonstrated.

Abstract

from arXiv · show

We introduce a novel method for multilingual transfer that utilizes deep contextual embeddings, pretrained in an unsupervised fashion. While contextual embeddings have been shown to yield richer representations of meaning compared to their static counterparts, aligning them poses a challenge due to their dynamic nature. To this end, we construct context-independent variants of the original monolingual spaces and utilize their mapping to derive an alignment for the context-dependent spaces. This mapping readily supports processing of a target language, improving transfer by context-aware embeddings. Our experimental results demonstrate the effectiveness of this approach for zero-shot and few-shot learning of dependency parsing. Specifically, our method consistently outperforms the previous state-of-the-art on 6 tested languages, yielding an improvement of 6.8 LAS points on average.

1 Introduction

The paper addresses the difficulty of aligning contextual embeddings for multilingual transfer by deriving their alignment from context-independent anchors. It applies this approach to low-resource zero-shot and few-shot dependency parsing, reporting gains over prior methods.

  • Contextual embeddings represent richer semantic and syntactic information than traditional context-independent embeddings, but dictionary-based alignment remains unclear because each word has multiple context-specific vectors.
  • The proposed method factorizes contextual spaces into context-independent anchors and context-dependent components, learns multilingual alignment at the anchor level, and applies it to the original contextual spaces.A bilingual dictionary can supervise anchor alignment when available.
  • Limited raw data can produce deficient source-language embeddings, so the method adds an anchor-alignment component to the language-model loss for low-resource languages.The method encourages anchors from source and target languages to be closer in a joint space.
  • 6.8 LAS points is the absolute improvement over the prior state-of-the-art in zero-shot cross-lingual dependency parsing.The evaluation covers zero-shot transfer and compares against Ammar et al. (2016).
  • 5 LAS points is the gain over the top result on the Kazakh treebank with only 38 training trees, demonstrating benefit for few-shot learning.

2 Related work

Prior multilingual transfer methods mainly align context-independent embeddings or rely on supervision such as dictionaries, parallel sentences, annotations, or translated treebanks. This work extends alignment to contextual embeddings, whose richer lexical representations are reported to improve multilingual parsing.

  • Multilingual Embeddings: MUSE and related methods construct multilingual spaces by aligning monolingual, context-independent embedding spaces, with supervised and unsupervised variants.
  • Multilingual Embeddings: Contextual embeddings introduce a new alignment challenge because each token has context-dependent representations rather than one fixed embedding.
  • Multilingual Embeddings: A concurrent method aligns word pairs appearing in the same context using parallel sentences, whereas this paper reports better word translations without that supervision.
  • Multilingual Embeddings: Bilingual dictionaries have also been used to improve embeddings trained on small datasets, including methods that jointly represent or train translated word pairs.
  • Multilingual Parsing: Multilingual parsing has used delexicalized tags, annotation projection, treebank translation, and aligned lexical embeddings, with several approaches requiring supervision.
  • Multilingual Parsing: The paper targets multilingual transfer with contextualized embeddings, motivated by their strong monolingual parsing results and reporting significant gains from richer lexical representations.

3 Aligning Contextual Word Embeddings

The paper aligns contextual embeddings by using context-independent word anchors, supporting supervised, unsupervised, and low-resource multilingual transfer. The approach relies on contextual geometry and preserves contextual relations through linear mappings.

  • 3 Aligning Contextual Word Embeddings: Context-dependent embeddings represent a token with multiple vectors across contexts, forming a point cloud rather than a single word vector.The paper studies how these clouds are distributed and uses ELMo embeddings for the analysis.
  • 3 Aligning Contextual Word Embeddings: Embedding anchors summarize each token’s contextual representations through an average computed over available unlabeled data.The anchor provides one context-independent vector per word for alignment.
  • 3.2 Context-Independent Alignment: Linear alignment maps source contextual vectors into the target space using W s→t, with orthogonality preserving relations among contextual shifts.The orthogonal constraint yields a closed-form Procrustes solution for context-independent alignment.
  • 3.1 The Geometry of Context-Dependent Embeddings: Contextual embeddings for homonyms can form multiple sense-specific clouds, and alignment places these clouds near their corresponding translations.The English word “bear” provides the main example, while “bank” shows that successful alignment can occur even without obvious separated clusters.
  • 3.3 Context-Dependent Alignment: The paper proposes three contextual alignment procedures: one supervised method and two unsupervised methods, including anchor-based and context-based variants.The supervised approach aligns token anchors using a bilingual dictionary, while the unsupervised approaches extend MUSE-style alignment.
  • 3.3 Context-Dependent Alignment: Context-based alignment directly handles homonyms but is empirically less stable than unsupervised anchored alignment.This is the paper’s stated limitation for the context-based variant.
  • 3.4 Learning Anchored Language Models: Anchored language-model training regularizes resource-limited languages by bringing translation anchors closer, reducing overfitting and encouraging cross-lingual alignment.The anchors are representations before context-aware components such as ELMo’s LSTM.

4 Multilingual Dependency Parsing

The parser is extended for multilingual transfer by replacing static embeddings with aligned contextual ELMo representations and sharing parameters across languages. Fixed, pretrained alignment supports both zero-shot and few-shot target-language settings.

  • Baseline Parser: The baseline parser is a first-order graph-based neural edge-scoring model, with a minimum-spanning-tree procedure enforcing valid outputs at test time.Dependency-label scores are defined separately for each edge, and MST decoding produces the final tree.
  • Multilingual Parsing with Alignment: The parser replaces static word embeddings with contextualized ELMo representations aligned to a shared multilingual space.The alignment matrix is learned before parser training and remains fixed during training.
  • Multilingual Parsing with Alignment: All model parameters are shared across languages after their contextual embeddings are mapped into the joint space.For a sentence in language ℓ, the contextual embeddings are transformed using the language-to-joint-space alignment matrix.
  • Multilingual Parsing with Alignment: The same architecture supports single-language and multiple-language training, with target-language data available in few-shot or absent in zero-shot settings.In multi-source settings, embeddings are aligned to English.
  • Multilingual Parsing with Alignment: The alignment and embedding changes occur at the word-embedding level, so they can be applied to other NLP models that use word embeddings.The paper presents the aligned contextual embeddings as an extension of the parsing architecture rather than a parser-specific modification.

5 Experimental Setup

Experiments use unsupervised ELMo embeddings, MUSE-based alignment, and biaffine dependency parsing, comparing supervised and unsupervised variants with contextual and context-independent baselines. The setup fixes the first ELMo layer for parsing and evaluates translation alignment and zero-shot LAS across six languages.

  • Contextual Embeddings: ELMo generates 1024-dimensional embeddings from tokenized Wikipedia data for each language, using 95% of sentences for training and 5% for evaluation.The model uses ELMo's default parameters and randomly shuffles sentences before splitting the data.
  • Alignment: MUSE and its dictionary tables provide the alignment framework, while anchor vectors are computed by averaging representations on an evaluation set.Alignment experiments use the 50k most common words in each language.
  • Dependency Parsing: The dependency parser is an AllenNLP biaffine parser trained on trees from one or multiple languages, with randomly alternated language batches in the multiple-language setting.Dropout is applied to ELMo, Bi-LSTM, and MLP outputs, and early stopping is used.
  • Alignment Evaluation: Table 2 evaluates English word translation with CSLS precision@5 for supervised and unsupervised alignment across German, Spanish, French, Italian, Portuguese, and Swedish.Unsupervised results are also reported after anchor-based refinement, with failed convergence marked by an asterisk.
  • Parsing Evaluation: Table 3 reports zero-shot cross-lingual LAS for six languages, comparing prior methods with aligned FastText, anchor-only, and reduced-supervision baselines.Corresponding UAS results are provided in the appendix.
  • Dependency Parsing: The first ELMo LSTM layer is used alone, with fixed layer weights [0, 1, 0], because it performs as well as learning a layer combination and captures syntactic information.The first-layer choice is supported by experiments on the English treebank and prior findings about lower-layer syntax.
  • Experimental Variants: The main model uses supervised anchored alignment for multilingual pretrained ELMo, alongside aligned FastText, anchor-only, no-dictionary, and no-POS variants.These variants isolate contextualization, dictionary supervision, and POS-tag contributions under the corresponding experimental settings.

6 Results

The experiments show that anchored contextual embeddings improve multilingual alignment and zero-shot or few-shot dependency parsing, including under limited supervision and unlabeled data.

  • Alignment: Anchors substantially improve alignment precision across languages in the REFINE step.Supervised alignments outperform unsupervised ones, while context-based alignment performs inconsistently across languages.
  • Zero-Shot Parsing, Multiple Source Languages: 6.8 LAS points is the margin by which the multilingual parser outperforms previous parsers in six-language zero-shot experiments.The model improves over previous methods even with unsupervised alignment.
  • Zero-Shot Parsing, Multiple Source Languages: Without POS tags, scores remain higher than previous annotated methods in five of six languages, while Portuguese drops by 8.8 LAS points.The result indicates strong transfer without POS annotations but also a language-specific degradation.
  • Zero-Shot Parsing, Multiple Source Languages: 3 points is the average advantage of the context-aware model over FASTTEXT in LAS, with gains across all six languages in UAS and five in LAS.The context-aware model also consistently outperforms models using precomputed context-independent anchors.
  • Few-Shot Parsing, Small Treebanks: 5.05 LAS points separate the proposed algorithm from the best shared-task model on the small Kazakh treebank, with over 10 points over FASTTEXT.The setting uses only 38 training trees, no development set, and no POS tags.
  • Zero-Shot Parsing, Limited Unlabeled Data: 7 UAS points and 9 LAS points are gained from anchored language-model regularization when unlabeled Spanish data are limited.Reducing unlabeled data decreases alignment precision by around 20 points, whereas regularization significantly improves validation perplexity and parsing.

7 Conclusion

The paper concludes that anchor-based alignment makes unsupervised contextual embeddings useful for multilingual transfer. It reports improvements in zero-shot and few-shot dependency parsing and connects the analysis to understanding contextual representations.

  • 7 Conclusion: Anchor-based alignment reduces multilingual contextual-embedding alignment to context-independent alignment and supports settings with or without dictionaries.The method is also designed for low-resource languages.
  • 7 Conclusion: The aligned embeddings improve upon state-of-the-art zero-shot and few-shot cross-lingual dependency-parsing models.The conclusion attributes transfer gains to the contextual nature of the embeddings.
  • 7 Conclusion: The analysis of ELMo context-aware embeddings contributes to understanding their properties.The paper presents these findings as a step toward understanding contextual word embeddings.
  • 7 Conclusion: The authors conjecture that the task-independent method may generalize to other tasks.This is stated as a conjecture rather than an experimentally established result.

A Alignment Results for All Layers

Across ELMo layers, contextual representations produce stronger word-translation anchors than layer-zero context-independent representations, according to the paper’s analysis.

  • A Alignment Results for All Layers: Contextual-layer anchors achieve higher word-translation precision than layer-zero representations in supervised alignment.Layer zero is context independent, whereas layers one and two use anchors derived from LSTM outputs.
  • A Alignment Results for All Layers: The paper conjectures that contextual representations are stronger anchors because the language-model objective trains them to support contextual embeddings.This contrasts with FASTTEXT, which directly optimizes token-based embeddings.

B Additional Parsing Results

The additional-results appendix presents complementary parsing comparisons and layer-wise word-translation alignment results, using UAS and precision metrics.

  • B Additional Parsing Results: Table 6 reports ELMo layer-wise word-translation precision@1 and precision@5 to English using supervised CSLS alignment.It covers German, Spanish, French, Italian, Portuguese, and Swedish.
  • B Additional Parsing Results: Table 6 distinguishes layer-zero character-level representations from layer-one and layer-two anchor alignments derived from LSTM outputs.Layer zero is context independent; layers one and two are contextual.
  • B Additional Parsing Results: Table 7 compares zero-shot cross-lingual parsing results with previous methods using UAS, including aligned FASTTEXT and context-independent baselines.Its bottom rows include models without POS tags and/or using unsupervised anchored alignment.

C Hyperparameters

The experiments use default MUSE hyperparameters for alignment and report the parsing hyperparameters in Table 8.

  • Alignment experiments use the default hyperparameters of the MUSE framework.
  • Table 8 reports the hyperparameter values used in multilingual parsing experiments.
  • The parsing hyperparameters are described as applying throughout the experiments.

D Additional Alignment Example

The additional alignment example examines contextual embeddings for the English word “bank” and its Spanish translations “banco” and “orilla.” Although the meanings do not form obvious reduced-space clusters, alignment places matching contexts near their translations.

  • Additional alignment example: The two meanings of “bank” do not form obvious clusters in the reduced space, possibly because both meanings have the same noun POS tag.
  • Additional alignment example: Alignment nevertheless places embeddings from each context close to the matching Spanish translation.Table 9 presents nearest neighbors for the Spanish anchors “banco” and “orilla,” with full sentences included for context.
  • Additional alignment example: Table 10 presents nearest neighbors after alignment for the Spanish anchors “tener” and “oso” derived from contextual embeddings of English “bear.”The full sentence is presented for context.
  • Figure 3: Figure 3 visualizes aligned contextual embeddings for English “bank” and Spanish “banco” and “orilla” in a two-dimensional PCA space.Anchors are shown as symbols, while contextualized embeddings are shown as dots; colors distinguish the Spanish meanings.
Loading 1902.09492v2…