Source-linked AI summary
Multilingual Constituency Parsing with Self-Attention and Pre-Training
Nikita Kitaev, Steven Cao, Dan Klein
TL;DR
Constituency parsing needed broader evidence on contextual pre-training across languages and on scaling large models beyond separate per-language systems. The paper compares pre-training conditions, builds shared multilingual pre-training and fine-tuning methods, and finds benefits across eleven languages while reducing model size through parameter sharing. It also reports limited cross-lingual generalization except when labeled data is limited.
Problem
Prior evidence covered contextual representations mainly through English experiments with ELMo, while large models make separate pre-training and fine-tuning for many languages computationally expensive.
Method
The paper evaluates no pre-training, fastText, ELMo, and BERT, then compares independent multilingual parsers with joint multilingual pre-training and fine-tuning that shares most parameters.
Results
Pre-training benefits all evaluated languages, and the resulting parsers achieve higher F1 scores than previously known systems for all eleven languages.
Takeaways & Limitations
Joint multilingual training is a promising resource-efficient approach for supporting many languages as pre-trained models grow in capacity.
Takeaways & Limitations
Joint training does not exhibit substantial cross-lingual generalization except when labeled data is limited.
Abstract
from arXiv · showhide
We show that constituency parsing benefits from unsupervised pre-training across a variety of languages and a range of pre-training conditions. We first compare the benefits of no pre-training, fastText, ELMo, and BERT for English and find that BERT outperforms ELMo, in large part due to increased model capacity, whereas ELMo in turn outperforms the non-contextual fastText embeddings. We also find that pre-training is beneficial across all 11 languages tested; however, large model sizes (more than 100 million parameters) make it computationally expensive to train separate models for each language. To address this shortcoming, we show that joint multilingual pre-training and fine-tuning allows sharing all but a small number of parameters between ten languages in the final model. The 10x reduction in model size compared to fine-tuning one model per language causes only a 3.2% relative error increase in aggregate. We further explore the idea of joint fine-tuning and show that it gives low-resource languages a way to benefit from the larger datasets of other languages. Finally, we demonstrate new state-of-the-art results for 11 languages, including English (95.8 F1) and Chinese (91.8 F1).
1 Introduction
The paper broadens prior constituency-parsing studies of contextual representations from English and ELMo to multiple pre-training conditions and languages. It also develops multilingual sharing and joint fine-tuning to reduce model size and support lower-resource languages.
- Pre-training benefits all evaluated languages, with BERT outperforming ELMo, ELMo outperforming fastText, and fastText slightly outperforming non-pre-trained baselines.Larger pre-trained model capacity typically yields higher parsing accuracy.
- The study compares independent multilingual parser training with joint fine-tuning that shares one copy of fine-tuned BERT parameters across ten languages.Only a small output-specific fragment remains language-specific.
- 10x smaller models lose an average of only 0.28 F1 under shared multilingual fine-tuning.The comparison is against separately fine-tuned parameter-heavy BERT variants.
- Larger treebanks work better as auxiliary tasks, while only smaller treebanks benefit from paired joint training.The results motivate resource-efficient support for many languages.
- Joint training provides limited cross-lingual generalization except when labeled data is limited.The paper frames this as a boundary on the broader usefulness of joint multilingual training.
2 Model
The parser scores labeled spans with a self-attentive neural architecture and selects the highest-scoring valid constituency tree using a CKY variant. BERT representations are projected into this parser, whose additional layers use word-based tokenization and task-specific self-attention.
- A constituency tree T is represented as a set of labeled spans, with each span defined by its start, end, and label.The parser assigns a score s(T) to each tree that decomposes over spans.
- A neural network transforms word vectors with self-attention and produces per-span labeling scores through a two-layer ReLU MLP classifier.For each span, its hidden vector is constructed by subtracting representations associated with the span boundaries.
- The highest-scoring valid tree is found efficiently with a variant of the CKY algorithm.
- BERT token representations are projected and passed to the parser after retaining the vector for each word’s final WordPiece sub-word.The paper reports that choosing the first rather than last sub-word did not substantially affect English parsing accuracy.
- Additional parser layers remain useful because they use word-based tokens, factored self-attention, and random initialization rather than BERT’s pre-trained sub-word architecture.Passing projected BERT vectors directly to the span classifier hurts parsing accuracy.
3 Comparison of Pre-Training Methods
The study compares pre-training methods and model configurations for English constituency parsing, finding stronger performance from contextual pre-training and larger capacity. It also examines multilingual pre-training and imperfect language-data fit.
- Pre-training methods: 95.70 F1: BERTLARGE outperforms BERTBASE and ELMo on English constituency parsing.BERTBASE and ELMo achieve 95.32 and 95.21 F1, respectively; fastText and no pre-training achieve 93.72 and 93.61 F1.
- Pre-training methods: Pre-training improves parsing over non-pre-trained baselines, with BERT outperforming ELMo and ELMo outperforming fastText.The comparison covers BERT, ELMo, fastText, and training from scratch on treebank data.
- Model configurations: 95.87 F1 vs. 95.66 F1: an ensemble of four English BERT-based parsers improves over the best single model.The gain reflects fine-tuning randomness and variation between BERT versions.
- Model configurations: BERT-based pre-training gains have not yet plateaued with computational resources and model size.This conclusion combines the stronger BERTLARGE result with the ensemble improvement.
- Multilingual pre-training: Less than 0.3 F1: multilingual pre-training trails English-only pre-training at equal model capacity, with 95.24 versus 94.97 F1.The result supports parameter sharing as a resource-efficient approach for multiple languages.
- Multilingual pre-training: 93.57 vs. 93.61 F1: English fine-tuning from Chinese-Wikipedia BERT performs comparably to supervised training from randomly initialized parameters.The pre-training data and subword vocabulary were not a good fit for the English target task.
4 Multilingual Model
The multilingual parser shares BERT and self-attention layers across ten languages while retaining language-specific span classifiers, reducing model size with only a small average accuracy loss. Joint training particularly benefits smaller treebanks, although cross-lingual generalization remains limited when labeled data is plentiful.
- Shared multilingual architecture: Over 1.8 billion parameters would be required for ten independent parsers, whereas the shared multilingual parser reduces model size by 10x.The shared model uses common BERT and self-attention layers plus one 250K-850K-parameter MLP span classifier per language.
- Joint model results: 91.12 vs. 88.32 average F1: joint multilingual training outperforms treebank-only models across all ten languages.The comparison uses the multilingual BERT model on the SPMRL and WSJ test splits.
- Joint model results: 91.12 vs. 91.40 F1: joint fine-tuning loses only slightly to independent fine-tuning despite the sharp model-size reduction.The joint model performs better for Swedish.
- Language compatibility: Smaller treebanks benefit most from joint training, while larger treebanks function better as auxiliary tasks.English, German, and Korean are the strongest auxiliaries; Swedish, French, and Polish gain most but function poorly as auxiliaries.
- Language compatibility: Joint training provides limited cross-lingual generalization, except when labeled data is limited.Accuracy differences reflect both access to more data and competition among multiple objectives sharing parameters.
5 Results
Across eleven languages, the parsers achieve higher F1 scores than previously reported systems. The results also show that pre-training remains beneficial when model parameters are heavily shared across languages.
- Multilingual results: Higher F1 scores than any previously known systems are obtained for all eleven evaluated languages.The languages include nine SPMRL languages, English, and Chinese.
- Multilingual results: Pre-training improves parsing even when parameters are shared heavily across languages.Both the single multilingual model and individual monolingual models outperform previous SPMRL systems without pre-trained contextual representations.
6 Conclusion
The effectiveness of unsupervised pre-trained language representations suggests continued potential for improving syntactic modeling. Joint multilingual training offers a scalable way to provide high-capacity NLP systems for many languages.
- Conclusion: Unsupervised pre-training may continue improving machine learning methods’ ability to model syntax and other aspects of language.The conclusion connects future progress in pre-trained representations with higher-capacity models.
- Conclusion: Joint multilingual training is presented as a promising scalable approach for providing NLP systems to many languages as pre-trained models grow.The motivation is scalability with increasingly high-capacity pre-trained models.