PaperlayerView exact revision on arXiv

Source-linked AI summary

75 Languages, 1 Model: Parsing Universal Dependencies Universally

Dan Kondratyuk, Milan Straka

arXiv:1904.02099v3cs.CLcs.LG

TL;DR

UDify addresses the challenge of parsing languages with little annotated data by jointly fine-tuning multilingual BERT across Universal Dependencies tasks and treebanks. The model simultaneously predicts tagging, morphology, lemmas, and dependency trees with task-specific classifiers, achieving strong multilingual results and the greatest benefits for low-resource languages, including zero-shot settings.

  • Problem

    Languages without annotated data are difficult to parse accurately, motivating multilingual transfer across syntactically annotated Universal Dependencies treebanks.

  • Method

    UDify fine-tunes multilingual BERT with task-specific layer attention and simultaneous classifiers for multiple Universal Dependencies tasks.

  • Results

    UDify exceeds state-of-the-art dependency-parsing performance across a large subset of languages while remaining comparable in tagging and lemmatization accuracy.

  • Takeaways & Limitations

    Multilingual learning is most beneficial for low-resource languages, including languages without training sets, and can be further improved with individual-treebank fine-tuning.

  • Takeaways & Limitations

    UDify severely underperforms the baseline on some low-resource languages, although individual-treebank fine-tuning eliminates most of these gaps.

Abstract

from arXiv · show

We present UDify, a multilingual multi-task model capable of accurately predicting universal part-of-speech, morphological features, lemmas, and dependency trees simultaneously for all 124 Universal Dependencies treebanks across 75 languages. By leveraging a multilingual BERT self-attention model pretrained on 104 languages, we found that fine-tuning it on all datasets concatenated together with simple softmax classifiers for each UD task can result in state-of-the-art UPOS, UFeats, Lemmas, UAS, and LAS scores, without requiring any recurrent or language-specific components. We evaluate UDify for multilingual learning, showing that low-resource languages benefit the most from cross-linguistic annotations. We also evaluate for zero-shot learning, with results suggesting that multilingual training provides strong UD predictions even for languages that neither UDify nor BERT have ever been trained on. Code for UDify is available at https://github.com/hyperparticle/udify.

1 Introduction

UDify addresses the difficulty of parsing languages with limited annotations by combining multilingual transfer with a pretrained self-attention model. It predicts multiple Universal Dependencies tasks simultaneously using shared multilingual representations and task-specific classifiers.

  • Multilingual modeling can help overcome the difficulty of accurately parsing languages without annotated training data.Cross-linguistic training can also reduce the cost of maintaining separate models for many languages.
  • Universal Dependencies provides consistent cross-lingual annotations for multi-task prediction, including lemmas, UPOS, UFeats, and dependency structures.
  • UDify uses multilingual BERT to produce contextual embeddings, task-specific layer attention, and simultaneous softmax decoding for UD tasks.
  • The model applies regularization techniques including masking, increased dropout, weight freezing, discriminative fine-tuning, and layer dropout.

3. We train and fine-tune the model on the en-

UDify is trained using the entirety of the available UD training data and evaluated against a strong multilingual parsing baseline. The experiments also examine multilingual benefits, zero-shot prediction, and why pretrained self-attention supports multilingual parsing.

  • UDify is trained by concatenating all available UD training sets.
  • The model is evaluated against UDPipe Future, a winner of the CoNLL 2018 Shared Task on Multilingual Parsing.
  • The experiments analyze which languages benefit most from multilingual training and evaluate zero-shot prediction on treebanks without training data.
  • The implementation uses AllenNLP with PyTorch, and the authors release UDify code and fine-tuned BERT weights.

2 Multilingual Multi-Task Learning

The multilingual multi-task model replaces conventional embedding and encoder components with multilingual BERT, then predicts several UD annotations from shared word-level representations. Its task heads cover tagging, morphological features, lemmatization, and dependency parsing.

  • The architecture uses multilingual BERT with 12 layers, 12 attention heads per layer, and 768-dimensional hidden states, pretrained on 104 languages.
  • BERT wordpiece tokenization is applied within UD word boundaries, and predictions use the first wordpiece representation for each word.
  • UDify replaces UDPipe Future’s embedding, encoder, and projection layers with BERT while retaining task-specific prediction layers and cross-entropy training.
  • UPOS and UFeats are predicted with word-level softmax classifiers, with each complete UFeats string treated as one vocabulary item.
  • Lemmatization is formulated as sequence tagging over edit scripts that transform word forms into lemmas.
  • Dependency parsing uses BERT-based biaffine attention to score arc heads before Chu-Liu/Edmonds tree decoding.

3 Fine-Tuning BERT on UD Annotations

UDify fine-tunes multilingual BERT with task-specific attention and extensive regularization to prevent downstream classifiers from overfitting individual layers or pretrained representations. The procedure uses differentiated optimization for BERT and non-BERT parameters.

  • Regularization is described as crucial for obtaining a high-scoring UDify network when fine-tuning BERT.
  • Task-specific layer attention computes weighted sums of all 12 BERT layers rather than relying only on the final layer.
  • Layer dropout sets each layer-attention weight to −∞ with probability 0.1, encouraging the model to use information across BERT layers.
  • BERT and non-BERT parameters use separate learning rates, and BERT is frozen during the first epoch for training stability.
  • Random input masking can reduce classifier overfitting by forcing reliance on surrounding context.

4 Experiments

The experiments evaluate UDify across Universal Dependencies treebanks, compare it with a retrained UDPipe Future baseline, and probe syntactic structure in BERT before and after fine-tuning.

  • Evaluation setup: The comparison baseline is UDPipe Future retrained with gold segmentation because CoNLL 2018 model tables do not assume gold word segmentation.The authors note that UDPipe Future does not itself use BERT.
  • Evaluation setup: Experiments use the full Universal Dependencies v2.3 corpus but omit five datasets whose training annotations are not freely available.The omitted datasets are Arabic NYUAD, English ESL, French FTB, Hindi English HEINCS, and Japanese BCCWJ.
  • Training setup: Multilingual training concatenates all available training sets, shuffles sentences before each epoch, and feeds mixed-language or mixed-treebank batches for 80 epochs.This setup exposes the model to sentences from any language or treebank within a batch.
  • Probing for syntax: A structural probe is trained on unmodified and fine-tuned BERT representations to test whether fine-tuning makes them more closely match dependency-tree structure.The probe evaluates whether syntax trees can be extracted from contextual embeddings using a linear transformation.

5 Results

Results compare UDify configurations and attention strategies across treebanks, assess zero-shot parsing, and analyze how BERT layers contribute to different UD tasks.

  • Main comparisons: Table 2 compares UDPipe Future with language-specific UDify, multilingual UDify, and multilingual-then-language-specific UDify configurations.The comparison covers selected high-resource and low-resource languages and reports UPOS, UFeats, Lemma, UAS, and LAS.
  • Layer preferences: Figure 2 shows that each UD task learns its own BERT layer-weight pattern rather than sharing one universal preference.The plotted weights are unnormalized attention weights contributing to each BERT layer.
  • Attention strategies: Table 3 averages scores over 89 treebanks to compare task-specific layer attention, global layer attention, and a simple sum of BERT layers.Figure 2 plots the learned layer-attention weights for each task.
  • Zero-shot learning: Zero-shot evaluation covers Breton, Tagalog, Faroese, Naija, and Sanskrit treebanks without training sets.The table distinguishes languages pretrained with BERT by bold formatting.

6 Discussion

UDify performs strongly across multilingual UD parsing, with multilingual training especially helping low-resource and zero-shot languages. Fine-tuning also makes BERT’s representations and attention more closely reflect syntactic dependencies, while task-specific layer attention improves performance.

  • Model Performance: UDify achieves results comparable to state-of-the-art UD parsing, especially excelling at dependency parsing while slightly underperforming on Lemmas and Universal Features.The authors attribute the latter gap to UDPipe Future’s character-level embeddings.
  • Model Performance: Multilingual training can substantially improve low-resource languages, with a second round of treebank-specific fine-tuning often reducing remaining gaps.The benefit is strongest where monolingual training has limited data, whereas already strong languages can sometimes perform worse.
  • Model Performance: Zero-shot fine-tuning produces reasonably high scores for treebanks without training data, including Breton, Tagalog, Faroese, Naija, and Sanskrit.The authors partially attribute this to similarities between these languages and languages seen during training.
  • Model Performance: Task-specific layer attention improves performance more than a global weighted average, with tasks preferring different BERT layers.All tasks favor the final three layers, while Lemmas and UFeats also use information from the first three layers.
  • Effect of Syntactic Fine-Tuning on BERT: Fine-tuning on UD reduces BERT’s UUAS error by 41%, and visual analyses show attention becoming sparser and more sensitive to localized syntactic structure.The syntactic probe and attention visualizations indicate that fine-tuning makes representations more closely resemble annotated dependency trees.
  • Factors that Enable BERT to Excel at Dependency Parsing and Multilinguality: The authors connect robust multilingual dependency parsing to self-attention’s long-range dependency modeling, regularization, and contextual pretraining.They argue that pretrained self-attention provides a syntactic bias that generalizes across languages without major multilingual degradation.

7 Related Work

Prior multilingual parsing systems combined related treebanks or used scalable multilingual representations, often reducing the number of models while improving low-resource performance. UDify extends this direction with a broader multilingual multi-task approach built on pretrained self-attention.

  • Multilingual parsing systems: The Uppsala system combined treebanks across 82 treebanks in a multi-task pipeline using 34 models, improving especially low-resource performance.Its results were no worse than training on each treebank individually.
  • Other multilingual approaches: Other multilingual approaches include translation models for 12 languages and recurrent multilingual embeddings trained on auxiliary tasks such as natural language inference.These efforts demonstrate alternative strategies for scaling multilingual modeling beyond UD parsing.

8 Conclusion

UDify is a single multilingual multi-task self-attention model that produces UD annotations across treebanks while matching or exceeding prior performance in key tasks. Its strongest benefits appear for low-resource and even training-free languages.

  • Conclusion: UDify exceeds state-of-the-art dependency-parsing performance in a large subset of languages while remaining comparable in tagging and lemmatization accuracy.The model produces annotations for any UD treebank supported by the framework.
  • Conclusion: Multilingual training reduces the required model count to one and benefits low-resource languages, including languages without training sets.Monolingual fine-tuning from UDify’s saved BERT weights can improve performance further.

A Appendix

The appendix provides training details, hyperparameter choices, and complete evaluation tables for UDify across Universal Dependencies languages.

  • Appendix: The appendix documents hyperparameter and miscellaneous training choices and presents full UDify evaluation tables across all UD languages.It is intended as a reference for the model’s training and results.

A.1 Hyperparameters

UDify uses concatenated multilingual training with extensive regularization and evaluates performance across treebanks of different sizes. The largest gains over UDPipe Future occur on small treebanks, while more training resources generally improve absolute LAS.

  • A.1 Hyperparameters: UDify concatenates all training sets, shuffles sentences into batches of 32, and trains for 80 epochs with staged learning-rate scheduling.BERT is unfrozen in the second epoch, followed by 8,000 warmup batches and inverse square root decay.
  • A.1 Hyperparameters: UDify applies bucketed batching with sentence-length fuzzing, stronger BERT and layer dropout, and increased wordpiece masking to address overfitting.Attention and hidden dropout increase from 0.1 to 0.2, BERT-layer dropout is 0.5, layer dropout is 0.1, and masking rises from 0.15 to 0.2.
  • A.1 Hyperparameters: The final multilingual model requires approximately 25 days of training on an NVIDIA GTX 1080 Ti, averaging 8 hours per epoch.Half-precision training is used to fit BERT in memory, and performance continues improving after validation begins leveling off.
  • A.2 Training Size Effect on Performance: The largest UDify-over-UDPipe Future LAS improvements tend to occur on treebanks with fewer than 3,000 training examples, while more training resources generally improve absolute LAS.Figure 5 plots LAS differences by training-set size, and Figure 6 reports absolute LAS values.
  • A.3 Miscellaneous Details: Language-specific modeling is not strictly necessary for high-performing cross-lingual dependency-parsing representations, although the authors caution that such techniques could improve the model.A second fine-tuning round can help weaker treebanks, but multilingual training over all treebanks can still outperform it.
Loading 1904.02099v3…