Multilingual Constituency Parsing with Self-Attention and Pre-Training

• Pre-training benefits all evaluated languages, with BERT outperforming ELMo, ELMo outperforming fastText, and fastText slightly outperforming non-pre-trained baselines.Larger pre-trained model capacity typically yields higher parsing accuracy.Source: §1 · p. 1
• The study compares independent multilingual parser training with joint fine-tuning that shares one copy of fine-tuned BERT parameters across ten languages.Only a small output-specific fragment remains language-specific.Source: §1 · p. 1
• 10x smaller models lose an average of only 0.28 F1 under shared multilingual fine-tuning.The comparison is against separately fine-tuned parameter-heavy BERT variants.Source: §1 · p. 1
• Larger treebanks work better as auxiliary tasks, while only smaller treebanks benefit from paired joint training.The results motivate resource-efficient support for many languages.Source: §1 · p. 1
• Joint training provides limited cross-lingual generalization except when labeled data is limited.The paper frames this as a boundary on the broader usefulness of joint multilingual training.Source: §1 · p. 2

• A constituency tree T is represented as a set of labeled spans, with each span defined by its start, end, and label.The parser assigns a score s(T) to each tree that decomposes over spans.Source: §2 · p. 2
• A neural network transforms word vectors with self-attention and produces per-span labeling scores through a two-layer ReLU MLP classifier.For each span, its hidden vector is constructed by subtracting representations associated with the span boundaries.Source: §2 · p. 2
• The highest-scoring valid tree is found efficiently with a variant of the CKY algorithm.Source: §2 · p. 2
• BERT token representations are projected and passed to the parser after retaining the vector for each word’s final WordPiece sub-word.The paper reports that choosing the first rather than last sub-word did not substantially affect English parsing accuracy.Source: §2 · p. 2
• Additional parser layers remain useful because they use word-based tokens, factored self-attention, and random initialization rather than BERT’s pre-trained sub-word architecture.Passing projected BERT vectors directly to the span classifier hurts parsing accuracy.Source: §2 · p. 2

Pre-training methods

• 95.70 F1: BERTLARGE outperforms BERTBASE and ELMo on English constituency parsing.BERTBASE and ELMo achieve 95.32 and 95.21 F1, respectively; fastText and no pre-training achieve 93.72 and 93.61 F1.Source: §3 · p. 2
• Pre-training improves parsing over non-pre-trained baselines, with BERT outperforming ELMo and ELMo outperforming fastText.The comparison covers BERT, ELMo, fastText, and training from scratch on treebank data.Source: §3 · p. 2

Model configurations

• 95.87 F1 vs. 95.66 F1: an ensemble of four English BERT-based parsers improves over the best single model.The gain reflects fine-tuning randomness and variation between BERT versions.Source: §3 · p. 3
• BERT-based pre-training gains have not yet plateaued with computational resources and model size.This conclusion combines the stronger BERTLARGE result with the ensemble improvement.Source: §3 · pp. 2–3

Multilingual pre-training

• Less than 0.3 F1: multilingual pre-training trails English-only pre-training at equal model capacity, with 95.24 versus 94.97 F1.The result supports parameter sharing as a resource-efficient approach for multiple languages.Source: §3 · p. 3
• 93.57 vs. 93.61 F1: English fine-tuning from Chinese-Wikipedia BERT performs comparably to supervised training from randomly initialized parameters.The pre-training data and subword vocabulary were not a good fit for the English target task.Source: §3 · p. 3

Shared multilingual architecture

• Over 1.8 billion parameters would be required for ten independent parsers, whereas the shared multilingual parser reduces model size by 10x.The shared model uses common BERT and self-attention layers plus one 250K-850K-parameter MLP span classifier per language.Source: §4 · pp. 3–4

Joint model results

• 91.12 vs. 88.32 average F1: joint multilingual training outperforms treebank-only models across all ten languages.The comparison uses the multilingual BERT model on the SPMRL and WSJ test splits.Source: §4 · p. 4
• 91.12 vs. 91.40 F1: joint fine-tuning loses only slightly to independent fine-tuning despite the sharp model-size reduction.The joint model performs better for Swedish.Source: §4 · p. 4

Language compatibility

• Smaller treebanks benefit most from joint training, while larger treebanks function better as auxiliary tasks.English, German, and Korean are the strongest auxiliaries; Swedish, French, and Polish gain most but function poorly as auxiliaries.Source: §4 · p. 4, Table 3 · p. 4
• Joint training provides limited cross-lingual generalization, except when labeled data is limited.Accuracy differences reflect both access to more data and competition among multiple objectives sharing parameters.Source: §4 · p. 4

Multilingual results

• Higher F1 scores than any previously known systems are obtained for all eleven evaluated languages.The languages include nine SPMRL languages, English, and Chinese.Source: §5 · p. 5
• Pre-training improves parsing even when parameters are shared heavily across languages.Both the single multilingual model and individual monolingual models outperform previous SPMRL systems without pre-trained contextual representations.Source: §5 · p. 5

Conclusion

• Unsupervised pre-training may continue improving machine learning methods’ ability to model syntax and other aspects of language.The conclusion connects future progress in pre-trained representations with higher-capacity models.Source: §6 · p. 5
• Joint multilingual training is presented as a promising scalable approach for providing NLP systems to many languages as pre-trained models grow.The motivation is scalability with increasingly high-capacity pre-trained models.Source: §6 · p. 5