Source-linked AI summary
Towards Better UD Parsing: Deep Contextualized Word Embeddings, Ensemble, and Treebank Concatenation
Wanxiang Che, Yijia Liu, Yuxuan Wang, Bo Zheng, Ting Liu
TL;DR
Multilingual dependency parsing requires effective modeling across languages, domains, and difficult treebank settings. HIT-SCIR extends a strong prior parser with contextualized embeddings, initialization-based parser ensembling, and selective treebank concatenation, achieving 75.84% averaged LAS and first place in the final evaluation. The authors note scope limitations in lemmatization and morphology tagging and caution that treebank concatenation is not universally beneficial.
Problem
The paper asks how to further improve a competitive POS tagger and dependency parser for multilingual Universal Dependencies parsing.
Method
HIT-SCIR adds deep contextualized word embeddings, ensembles parsers with different initialization, and explores treebank concatenation.
Results
75.84% averaged LAS earned HIT-SCIR first place in the final evaluation.
Takeaways & Limitations
The reported techniques improved parsing performance and produced a large-margin first-place LAS result in the CoNLL 2018 evaluation.
Takeaways & Limitations
The system used baseline lemmatization and morphology tagging, ranking sixth in MLAS and second in BLEX.
Abstract
from arXiv · showhide
This paper describes our system (HIT-SCIR) submitted to the CoNLL 2018 shared task on Multilingual Parsing from Raw Text to Universal Dependencies. We base our submission on Stanford's winning system for the CoNLL 2017 shared task and make two effective extensions: 1) incorporating deep contextualized word embeddings into both the part of speech tagger and parser; 2) ensembling parsers trained with different initialization. We also explore different ways of concatenating treebanks for further improvements. Experimental results on the development data show the effectiveness of our methods. In the final evaluation, our system was ranked first according to LAS (75.84%) and outperformed the other systems by a large margin.
1 Introduction
HIT-SCIR extends Stanford’s strong CoNLL 2017 parsing system with contextualized embeddings, parser ensembling, and treebank concatenation. On the official CoNLL 2018 test set, it achieved first place by LAS.
- System basis: HIT-SCIR builds on Stanford’s winning CoNLL 2017 tagger and parser.The paper frames the system as an extension of a highly competitive prior architecture.
- Extensions: Deep contextualized word embeddings are incorporated into the word representations used by both the POS tagger and parser.The paper identifies ELMo as the contextualized embedding method.
- Extensions: Parser ensembles are formed from models trained with different initialization.The system treats initialization diversity as a separate extension to the baseline parser.
- Treebank concatenation: Treebanks that may help one another are concatenated, including treebanks from related languages or domains.The paper explores different concatenation strategies rather than assuming concatenation always helps.
- Final result: 75.84 averaged LAS earned the system first place in the final CoNLL 2018 evaluation.The result significantly outperformed the other systems by a large margin.
2 Deep Biaffine Parser
The system uses LSTM-based representations to predict POS tags and dependency structures, with biaffine scoring for candidate assignments. Word representations combine token, word2vec, and character-level components.
- Architecture: An LSTM network produces a vector representation for each word before POS-tag and dependency-relation prediction.This is the core representation mechanism inherited from the underlying tagger and parser.
- POS tagging: POS-tag scores are computed from the word representation, with each score indicating the possibility of assigning a corresponding tag.The supplied description identifies the score interpretation but does not specify the complete scoring equation.
- Dependency parsing: The parser represents each word using its word and POS-tag inputs before dependency scoring.The parser’s input representation differs from the tagger’s word-only input.
- Dependency parsing: A biaffine classifier scores pairs of word representations to predict whether a dependency arc exists between them.The scores are evaluated across possible head words.
- Decoding: The decoder may produce cycles, which are handled using an iterative cycle-fixing method.This decoding procedure follows the prior system’s approach.
- Word representation: Each word representation sums a finetuned token embedding, a fixed word2vec embedding, and an LSTM-encoded character representation.The paper gives the composition as v(word) = w_i + p_i + ˆv_i.
3 Deep Contextualized Word Embeddings
The system adds fixed deep contextualized word embeddings to both the tagger and parser, using simplified ELMo representations and dropout during training. Preliminary experiments select between contextual representations based on development performance.
- ELMo construction: ELMo encodes words with an LSTM trained on a language-modeling objective over large-scale raw text.The representation is produced from hidden states of a bidirectional LSTM.
- ELMo integration: The system uses a two-layer bidirectional LSTM and combines its contextual representation with the existing word representation.The projected ELMo vector is added as an additional word embedding for both the tagger and parser.
- Simplifications: Unlike Peters et al. (2018), the system treats ELMo as fixed and does not tune its parameters during tagger and parser training.The authors consequently remove layer-wise attention scores and the scaling factor.
- Representation selection: Development performance determines whether the system uses the contextual representation from the initial layer or another selected representation.Preliminary experiments found the initial hidden representation better on some treebanks.
- Training: Dropout is applied to the projected ELMo vector during training to avoid overfitting.The projection parameters are trained jointly with the tagger and parser.
4 Parser Ensemble
The parser ensemble addresses sensitivity to neural-network initialization by training three differently initialized parsers and averaging their softmax output scores.
- Ensemble construction: Three parsers with different initialization are trained and ensembled by averaging their softmaxed output scores.The approach follows prior findings that initialization-sensitive neural-network ensembles can improve performance.
5 Treebank Concatenation
The system concatenates related treebanks across domains or languages, selecting concatenation according to development or cross-validation performance because it does not always help.
- Motivation: Treebank relationships across domains and language families are used as a basis for concatenation.Prior work and the CoNLL 2017 shared task motivate exploiting these relationships.
- Concatenation settings: The system concatenates same-language treebanks across domains or pairs of typologically or geographically correlated languages.This is the system’s stated exception to the adopted concatenation technique.
- Experiments: Cross-domain concatenation is tested on nl, sv, ko, it, en, fr, gl, la, ru, and sl.These are the languages listed for the cross-domain experiments.
- Selection: Cross-lingual concatenation is tested on ugtr, uk-ru, ga-en, and sme-fi, but vocabulary, grammatical genre, and annotation differences can prevent improvement.Usage is chosen from development performance, or by 5-fold cross validation when small treebanks lack development sets.
6 Low Resources Languages
For extremely small or absent training sets, the system transfers word embeddings and a parser from a richer related language, with special handling for Naija.
- Resource constraints: Five shared-task languages have fewer than 50 training sentences, while four have no training data.These conditions make training reasonable parsers difficult.
- Transfer method: The system maps rich- and low-resource-language fasttext embeddings into a shared space using bilingual word-vector transformation.A parser trained on the source treebank is then used for the target language.
- Embedding alignment: The transformation matrix is learned from fasttext embeddings using tokens shared by the two languages, such as punctuation.The transferred parser uses the transformed pretrained embeddings.
- Source selection: For small treebanks, source languages are selected by testing source-parser performance on target training data.For treebanks without training data, the source is chosen according to language family.
- Exception: Naija is handled by applying the full English en_ewt pipeline to pcm and nsc because it lacks fasttext embeddings.The method treats Naija as an English dialect for this transfer decision.
7 Preprocessing
HIT-SCIR treats sentence segmentation and tokenization as important preprocessing components, using specialized segmentors or tokenizers for difficult treebanks and languages. These choices produced substantial development-set improvements and strong final segmentation and tokenization results.
- Preprocessing is treated as an important factor in final parsing performance, alongside improvements to the tagger and parser.
- Sentence Segmentation: 7.67 F-score improvement was obtained on selected treebanks by replacing the UDPipe sentence segmenter with Uppsala segmentor.The selected treebanks had UDPipe development segmentation F-scores below 95.
- Tokenization for Chinese, Japanese, and Vietnamese: Chinese, Japanese, and Vietnamese tokenization is modeled as character-level word-boundary labeling using features from large-scale unlabeled data.The SCIR tokenizer was developed because these languages lack explicit word-boundary markers.
- Tokenization for Chinese, Japanese, and Vietnamese: 7.5 average tokenization F-score improvement over the UDPipe baseline was achieved on the related treebanks.The tokenizer combines PMI, character ELMo, and character bigram embeddings.
- Thai: Thai preprocessing uses whitespace as the sentence delimiter and lexicon-based forward maximum matching for word segmentation.The lexicon preserves the top 10% most frequent words derived from fasttext embeddings.
- Lemmatization and Morphology Tagging: The system uses only the baseline model for lemmatization and morphology tagging, ranking sixth in MLAS and second in BLEX.
8 Implementation Details
The implementation combines pretrained word embeddings, language-specific ELMo models, and ensembles of segmentation models. Figure 1 organizes treebanks by size to examine ELMo’s effects.
- Pretrained Word Embeddings: Large-language treebanks use 100-dimensional shared-task pretrained embeddings, while small or low-resource treebanks use 300-dimensional fasttext embeddings.
- ELMo Effects: Figure 1 sorts treebanks from smallest to largest to display the effects of ELMo.
- Pretrained Word Embeddings: All embeddings retain only the 10% most frequent words, with French embeddings used for the Old French exception.
- ELMo: ELMo parameters are trained as bidirectional language models on 20-million-word samples of each language’s released raw text.The implementation follows Peters et al.’s hyperparameters and uses sample softmax for large-vocabulary training.
- Biaffine Parser: The biaffine parser follows Dozat et al.’s hyperparameters and applies 33% dropout to projected vectors when trained with ELMo.
- SCIR Tokenizer: The SCIR tokenizer uses 50-dimensional character bigram embeddings and ensembles five single segmentors.
- Uppsala Segmentor: The Uppsala segmentor uses default settings and ensembles three single segmentors in its final model.
9 Results
The experiments evaluate ELMo, parser ensembling, treebank concatenation, and preprocessing improvements. Gains are strongest for smaller treebanks, while cross-lingual concatenation is generally ineffective.
- Effects of ELMo: No clear relation appears between treebank size and the gains from ELMo.The study examines error reduction across treebanks but finds no consistent size-related trend.
- Effects of Ensemble: 0.55% averaged LAS improvement and 4.0% averaged error reduction result from parser ensembling, with smaller treebanks benefiting more.The authors attribute the stronger gains on small treebanks to improved generalization against overfitting.
- Effects of Treebank Concatenation: Cross-domain concatenation generally helps small treebanks, while large treebanks show no improvement or may worsen.For small treebanks without development sets, concatenation improves most cases except gl treegal.
- Effects of Treebank Concatenation: Cross-lingual concatenation improves performance only on uk iu, supporting the need for more sophisticated transfer methods.The evaluated cross-lingual results cover development-set and 5-fold cross-validation settings.
- Effects of Better Preprocessing: Word segmentation performance is almost linearly correlated with final parsing performance, although sentence-segmentation improvements have exceptions.Better preprocessing can coincide with lower final parsing performance for el gdt and pt bosque.
- Parsing Strategies and Test Set Evaluation: 75.26 macro-averaged LAS is achieved without parser ensembling, while the selected full system uses ELMo, ensemble, and concatenation choices.The final strategies are selected using development-set performance and cross-validation.
10 Conclusion
The system extends a previous winning architecture with contextualized embeddings, parser ensembling, and treebank concatenation. It achieves 75.84% averaged LAS and first place in the final evaluation.
- Conclusion: The system improves on last year’s winning system by adding deep contextualized word embeddings, parser ensemble, and treebank concatenation.The paper reports these as its principal extensions to the prior system.
- Conclusion: 75.84% averaged LAS earns the system first place in the final evaluation.The authors state that the development results show the effectiveness of their methods.
11 Credits
The paper credits foundational Universal Dependencies resources, baseline and parsing systems, and released contextual-embedding code.
- Credits: The credits acknowledge the Universal Dependencies papers and datasets, baseline UDPipe, contextual word-embedding code, biaffine parsing code, and segmentation work.The cited contributors include Nivre et al., Straka et al., Peters et al., Dozat et al., and de Lhoneux et al.