PaperlayerView exact revision on arXiv

Source-linked AI summary

UDapter: Language Adaptation for Truly Universal Dependency Parsing

Ahmet Üstün, Arianna Bisazza, Gosse Bouma, Gertjan van Noord

arXiv:2004.14327v2cs.CL

TL;DR

Multilingual dependency parsing must balance transfer across languages with the capacity needed to avoid interference, especially across typologically diverse languages. UDapter combines adapters and contextual parameter generation conditioned on typological features, and outperforms strong baselines on most high- and low-resource languages.

  • Problem

    Multilingual models can suffer from interference and limited capacity when scaled across many typologically diverse languages, constraining high-resource performance.

    Source: §1 · p. 1

  • Method

    UDapter uses adapters and contextual parameter generation to adapt language-specific parameters from embeddings conditioned on typological features while sharing model parameters.

    Source: §1 · p. 1, §3.2 · p. 3, §3.3 · p. 4

  • Results

    UDapter outperforms strong monolingual and multilingual baselines on most high-resource and low-resource zero-shot languages.

    Source: §6 · p. 9

  • Takeaways & Limitations

    The analyses identify typological features as crucial for zero-shot languages within UDapter's soft parameter-sharing approach.

    Source: §6 · p. 9

    Contextual parameter generation multiplies adapter parameter count by language-embedding size, making the model larger than the baseline.

    Source: §3.3 · p. 4

Abstract

from arXiv · show

Recent advances in multilingual dependency parsing have brought the idea of a truly universal parser closer to reality. However, cross-language interference and restrained model capacity remain major obstacles. To address this, we propose a novel multilingual task adaptation approach based on contextual parameter generation and adapter modules. This approach enables to learn adapters via language embeddings while sharing model parameters across languages. It also allows for an easy but effective integration of existing linguistic typology features into the parsing network. The resulting parser, UDapter, outperforms strong monolingual and multilingual baselines on the majority of both high-resource and low-resource (zero-shot) languages, showing the success of the proposed adaptation approach. Our in-depth analyses show that soft parameter sharing via typological features is key to this success.

1 Introduction

UDapter addresses capacity and interference challenges in multilingual dependency parsing by combining shared parameters with language-specific adaptation. It uses typological features to condition that adaptation and outperforms strong baselines on most high-resource languages while improving zero-shot results.

1 Introduction

  • Scaling multilingual models across many typologically diverse languages can hurt performance, while limited capacity can lead to over-generalization on high-resource languages.The paper identifies this capacity challenge as the curse of multilinguality.Source: §1 · p. 1
  • UDapter combines adapter tuning with contextual parameter generation, adapting language-specific parameters from language embeddings while sharing parameters across languages.The approach uses a mix of curated and predicted typological features from URIEL rather than learning language embeddings from scratch.Source: §1 · pp. 1–2
  • Typological features substantially increase zero-shot LAS without reducing high-resource performance compared with language embeddings learned from scratch.The paper presents conditioning on existing typological features as a simple, effective contribution to zero-shot parsing.Source: §1 · p. 2
  • UDapter significantly outperforms strong monolingual and multilingual parsers on most high-resource languages and achieves promising improvements on zero-shot languages.Evaluation covers 13 high-resource and 30 genuinely low-resource languages.Source: §1 · p. 2

2 Previous Work

Previous work traces multilingual neural models and cross-lingual dependency parsing, including language conditioning and typological features. Prior findings on typology were mixed, and learned typological embeddings did not consistently help parsing.

Multilingual Neural Networks

  • Multilingual neural-network work moved from dedicated architectures to shared models, then pursued low-resource transfer while preserving high-resource accuracy; adapters also addressed capacity in pretrained models.The transfer–interference trade-off is discussed in neural machine translation, and Pfeiffer et al. (2020) combined language and task adapters for cross-lingual transfer.Source: §2 · p. 2

Cross-Lingual Dependency Parsing

  • Cross-lingual parsing progressed from delexicalized source-language parsers to approaches adding typological features, syntactic embeddings, word clusters, multilingual embeddings, and mBERT-based zero-shot transfer.Kondratyuk and Straka (2019) trained a multilingual parser on concatenated available UD treebanks.Source: §2 · p. 2

Language Embeddings and Typology

  • Language embeddings encode input-language information, while typological-feature results in parsing remained mixed and often relied on a handful of manually selected features.Ammar et al. (2016) found learned typological embeddings underperformed randomly initialized language IDs and hurt zero-shot parsing; UDapter instead uses soft parameter sharing.Source: §2 · pp. 2–3

3 Proposed Model

UDapter combines a pretrained mBERT encoder, adapter layers, and a biaffine parser whose trainable weights are generated from language embeddings.

3 Proposed Model

  • UDapter stacks a biaffine parser on pretrained mBERT, keeping mBERT weights frozen while generating adapter and parser weights from language embeddings.The contextual parameter generator is updated during treebank training.Source: §3 · p. 3

3.1 Biaffine Attention Parser

  • The biaffine parser scores head–tail arcs from encoder representations and uses Chu-Liu/Edmonds to find the highest-scoring valid dependency tree.Label scores are calculated with a biaffine classifier over separate feedforward layers.Source: §3.1 · p. 3

3.2 Transformer Encoder with Adapters

  • Two bottleneck adapters with feedforward projections and GELU are inserted into each transformer layer while the pretrained network weights remain frozen.Adapters capture task- and language-specific information while the original network serves as multilingual memory.Source: §3.2 · pp. 3–4

3.3 Contextual Parameter Generator

  • CPG generates adapter and biaffine-attention parameters from language embeddings, controlling cross-language sharing; its parameter count grows with embedding size.The paper notes that CPG makes the model larger than the baseline and leaves separate low-rank projections for parameter groups untried.Source: §3.3 · p. 4, §3.4 · p. 4

3.4 Typology-Based Language Embeddings

  • A two-layer MLP maps typological features to language embeddings, using syntactic, phonological, and phonetic inventory features, with predicted values where feature data are unavailable.The feature vector draws on URIEL and includes 103 syntactic, 28 phonological, and 158 phonetic inventory features.Source: §3.4 · pp. 4–5

4 Experiments

The experiments evaluate UDapter on 13 high-resource and 30 low-resource languages against monolingual and multilingual baselines. UDapter outperforms these baselines across the high-resource set and shows a small average gain on the low-resource set.

4 Experiments

  • Experiments used 13 high-resource training languages and evaluated 30 low-resource languages, including zero-shot tests; baselines included retrained mono-udify and multi-udify.The comparison also included an adapter-only variant and UDapter-proxy, which uses learned language embeddings instead of typology features.Source: §4 · pp. 5–6

4.1 Results

  • UDapter outperformed monolingual and multilingual baselines on most high-resource languages and showed overall promising improvements on zero-shot languages.The authors describe zero-shot parsing as a difficult setting and characterize the results as a step toward overcoming the positive/negative transfer trade-off.Source: §4.1 · pp. 6–7
  • 9 of 13 high-resource languages set a new state of the art, and UDapter significantly outperformed mono-udify and multi-udify in 11 of 13 languages.The exceptions to significance were English and Italian.Source: §4.1 · p. 6
  • 36.5 vs 36.3 average LAS on the 30 low-resource languages favored UDapter over multi-udify; UDapter also won on 13 of 18 representative languages and was significantly better on 16 of 30.The zero-shot setting has no training data.Source: §4 · p. 5, §4.1 · p. 6
  • UDapter-proxy underperformed UDapter, indicating the importance of typological features in the approach.The proxy used an embedding from the same language family when available for low-resource languages.Source: §4 · p. 6, §4.1 · p. 7

5 Analysis

The analyses find that UDapter’s gains favor languages with less training data, while typological features are especially valuable for zero-shot parsing. Its learned representations preserve some linguistic structure, and contextual parameter generation (CPG) matters beyond simply enlarging adapters.

5.1 Which languages improve most?

  • Gains over multi-udify are higher for high-resource languages with less training data, benefiting low-resource languages without hurting high-resource ones.For zero-shot languages, the difference is +1.2 LAS; UDapter outperforms multi-udify on 13 of 22 languages absent from mBERT training.Source: §5.1 · p. 7

5.2 How much gain from typology?

  • Typological features leave high-resource average LAS similar (87.3 versus 87.1) but raise zero-shot average LAS from 9.0 to 36.5 versus centroid embeddings.The comparison uses 30 zero-shot languages; a same-family proxy embedding also underperforms UDapter.Source: §5.2 · p. 7, Figure 4 · p. 7

5.3 How does UDapter represent languages?

  • The projection layer uses syntactic, phonological, and phonetic-inventory features, rather than relying only on syntax.The feature-weight analysis reports use of all available typological features.Source: §5.3 · p. 7, Figure 4 · p. 7
  • With typological features, UDapter’s embeddings preserve many linguistic similarities and place low-resource languages near typologically similar trained languages.Some similarities are overridden, including Japanese drifting away from Korean; related-language groupings such as Indian and Uralic languages are largely preserved.Source: §5.3 · p. 8, Figure 5 · p. 8
  • Without typological features, learned embeddings form a seemingly random space without linguistic similarities or a principled way to represent additional languages.The comparison includes Arabic far from Hebrew and Korean closer to English than Japanese.Source: §5.3 · p. 8
  • UDapter’s typology-informed embeddings balance linguistic motivation with optimization for in-training language accuracy.This conclusion combines the embedding plots with the parsing results.Source: §5.3 · p. 8

5.4 Is CPG really essential?

  • Doubling adapter size slightly helps high-resource languages but slightly hurts zero-shot performance, whereas adding CPG yields the best results across languages.Applying CPG to the encoder brings most high-resource gains; for low-resource languages, typology-based sharing matters most in biaffine attention.Source: §5.4 · pp. 8–9

6 Conclusion

UDapter combines adapters with contextual parameter generation conditioned on language embeddings projected from curated typological features. It outperforms strong monolingual and multilingual baselines on most high- and low-resource languages, with analyses identifying typological features as crucial for zero-shot performance.

6 Conclusion

  • UDapter combines adapters with contextual parameter generation to adapt language-specific parameters using embeddings projected from curated typological features.Adapters provide task-level adaptation, while contextual parameter generation provides language-specific adaptation.Source: §6 · p. 9
  • UDapter outperforms strong monolingual and multilingual baselines on most high-resource and low-resource zero-shot languages.The authors attribute this balance to per-language capacity and maximum sharing, and conclude that typological features are crucial for zero-shot languages.Source: §6 · p. 9

A.1 Experimental Details

The implementation builds on UDify settings and compares UDapter with retrained baselines under dependency-parsing conditions. UDapter has more trainable parameters than UDify but similar training time.

A.1 Experimental Details

  • UDapter is based on UDify and reuses its optimized hyperparameters without a new search; mono-udify and multi-udify are retrained on 13 high-resource languages for dependency parsing.The adapter-only baseline uses size 1024 for comparison, while final UDapter uses size 256; neither UDapter nor the baselines uses layer attention.Source: §A.1 · p. 13
  • An epoch takes approximately 30 minutes for UDapter and 27 minutes for UDify on a Tesla V100, while their trainable parameter counts are 550M and 191M, respectively.UDapter’s parameter cost depends on language-embedding size rather than the number of languages; separate monolingual UDify models for 13 languages total 2.5B parameters.Source: §A.1 · p. 13
  • Table 4 reports LAS results for UDapter and UDify models across all low-resource languages, marking languages absent from mBERT training data with “*” and indicating no significant UDapter–multi-udify difference with “†”.Source: Table 4 · p. 13

A.2 Zero-Shot Results

The appendix reports LAS results for UDapter, original UDify, and retrained multi-udify across 30 low-resource languages.

A.2 Zero-Shot Results

  • Table 4 reports LAS scores for UDapter, original UDify, and retrained multi-udify on all 30 low-resource languages.Original UDify was trained on all available UD treebanks from 75 languages; its zero-shot scores were obtained by running the pretrained model.Source: §A.2 · p. 13

A.3 Language Details

The appendix documents the language composition and dataset details for training and zero-shot experiments. Training languages come from UD 2.3, while zero-shot languages are selected from UD 2.5 to expand the experimental language set.

A.3 Language Details

  • Tables 5 and 6 provide language codes, data sizes, and family information for training and zero-shot languages.Source: §A.3 · p. 13
  • Training languages are drawn from UD 2.3, with treebank, family, word-order, and training/test data-size details listed in Table 5.Source: Table 5 · p. 14
  • Zero-shot languages are selected from UD 2.5 to increase the number of languages in the experiments, with treebank, family, and test-size details in Table 6.Source: Table 6 · p. 14

Related papers

Loading 2004.14327v2…