Source-linked AI summary
Universal Dependency Parsing from Scratch
Peng Qi, Timothy Dozat, Yuhao Zhang, Christopher D. Manning
TL;DR
Dependency parsing research often treats parsing separately from imperfect upstream components, motivating an end-to-end solution. The paper builds a neural raw-text-to-CoNLL-U pipeline covering all required stages and finds competitive performance on big treebanks, with substantially stronger low-resource results after correcting a tokenizer preprocessing bug.
Problem
Dependency parsing is commonly studied in isolation even though upstream tokenization, segmentation, tagging, and lemmatization systems remain imperfect.
Method
The authors build a neural pipeline that converts raw text into CoNLL-U annotations through tokenization, sentence and word segmentation, POS/UFeats tagging, lemmatization, and dependency parsing.
Results
The corrected system would have ranked 2nd, 1st, and 3rd on LAS, MLAS, and BLEX, respectively, and outperformed all submission systems on every official metric for low-resource treebanks.
Takeaways & Limitations
A single neural pipeline can achieve competitive performance on big treebanks and very strong performance across low-resource treebank categories after preprocessing correction.
Takeaways & Limitations
The submitted system contained a tokenizer data-preparation bug that caused sentence segmentation to overfit to newlines and negatively affected low-resource generalization.
Abstract
from arXiv · showhide
This paper describes Stanford's system at the CoNLL 2018 UD Shared Task. We introduce a complete neural pipeline system that takes raw text as input, and performs all tasks required by the shared task, ranging from tokenization and sentence segmentation, to POS tagging and dependency parsing. Our single system submission achieved very competitive performance on big treebanks. Moreover, after fixing an unfortunate bug, our corrected system would have placed the 2nd, 1st, and 3rd on the official evaluation metrics LAS,MLAS, and BLEX, and would have outperformed all submission systems on low-resource treebank categories on all metrics by a large margin. We further show the effectiveness of different model components through extensive ablation studies.
1 Introduction
The paper presents a complete neural pipeline that processes raw text through all CoNLL 2018 UD Shared Task stages. It combines neural and symbolic methods and reports competitive performance, especially after correcting a preprocessing bug.
- The system processes raw text through tokenization, sentence and word segmentation, POS/UFeats tagging, lemmatization, and dependency parsing.It is designed as a complete pipeline for all tasks required by the Shared Task.
- The authors combine symbolic statistical knowledge with flexible neural systems to improve robustness across pipeline components.The contributions include a frequency-informed approach, joint POS/UFeats prediction, a lemmatizer edit classifier, and parser extensions for linearization.
- A biaffine classifier jointly predicts POS and morphological features to improve prediction consistency.
- An edit classifier enhances the sequence-to-sequence lemmatizer’s robustness on rare sequences.
- After fixing a bug, the corrected system would have ranked 2nd, 1st, and 3rd on LAS, MLAS, and BLEX, respectively, while outperforming all systems on low-resource treebanks.The paper also reports extensive ablation studies evaluating the effectiveness of its novel methods.
2 System Description
The system is a raw-text-to-CoNLL-U neural pipeline covering tokenization, segmentation, tagging, lemmatization, and dependency parsing. Its tokenizer combines hierarchical BiLSTM tagging with token-level context, while symbolic lexicons and neural sequence-to-sequence models handle multi-word token expansion and lemmatization.
- Tokenizer: The tokenizer outputs CoNLL-U sentences through joint tokenization and sentence segmentation, followed by multi-word token expansion into syntactic words.These two subsystems prepare word lists for downstream processing.
- Tokenizer: Tokenization and sentence segmentation assign five hierarchical tags to character- or syllable-level units using BiLSTMs.The tags are EOT, EOS, MWT, MWS, and OTHER; Vietnamese uses syllables as units.
- Tokenizer: A two-layer BiLSTM incorporates token-level context through CNN-enhanced representations and gates that suppress non-token boundaries.The first layer operates on raw units, while gating propagates selected hidden states upward.
- Tokenizer: Multi-word token expansion combines a frequency lexicon with a neural character-level sequence-to-sequence model for flexible or unseen patterns.Evaluation tries the dictionary, then a lowercased lookup, and finally the neural predictor.
- POS/UFeats Tagger: The tagger uses a highway BiLSTM with pretrained, frequent-word, and character-level embeddings, predicting UPOS, XPOS, and UFeats with affine or biaffine classifiers.The biaffine XPOS classifier conditions on UPOS to improve consistency, with alternatives for very large or structured tag spaces.
- Lemmatizer: The lemmatizer uses word-based dictionaries before falling back to a neural model, preserving case because it is more relevant for lemmatization than for MWT expansion.The primary dictionary is keyed by word and UPOS; the fallback uses the word alone.
3 Training Details
The pipeline is trained and evaluated as linked components, with special procedures for missing treebank data and explicit hyperparameter choices for each module. Development evaluations generally use oracle upstream annotations, while deployment follows the full raw-text pipeline.
- Pipeline Training and Evaluation: The deployed pipeline sequentially tokenizes and expands raw text, tags words, lemmatizes them, and predicts dependency heads and labels from all annotations.This ordering defines the interfaces between the system’s components.
- Pipeline Training and Evaluation: Most component development evaluations assume oracle related annotations, whereas the dependency parser is the exception.The tokenizer, expander, and downstream tasks receive specified gold segmentations or annotations during isolated evaluation.
- Low-Resource Treebanks: Treebanks without training data use models from the largest available treebank in the same or a related language, with UDPipe 1.2 used for Thai.Examples include English-EWT for Naija and Irish-IDT for Breton.
- Dependency Parser: The parser models relative head-dependent location by factoring it into linear order and distance, with Cauchy-based distance modeling described as more numerically stable during training.Distances are predicted with 1 + softplus and the Cauchy choice avoids the alternatives’ roughly proportional penalties.
- Hyperparameters: The tokenizer uses 64d directional BiLSTM states, 32d character embeddings, dropout p = .33, and input-unit replacement with p = .33.Its gating mechanism is also perturbed by randomly setting gates to 1 with p = .02.
- Hyperparameters: The lemmatizer uses 100d encoder states, 50d character embeddings, a 200d decoder, dropout p = .5, and jointly trains edit and sequence-to-sequence losses.Training runs for up to 60 epochs with standard Adam hyperparameters.
- Hyperparameters: The tagger and parser use shared settings including 75d uncased frequent-word embeddings, while the tagger has 2-layer 200d and the parser 3-layer 400d BiLSTMs.Embedding replacement uses p = .33, feedforward dropout p = .5, and recurrent dropout p = .25, except p = .5 in the tagger BiLSTM.
4 Results
The pipeline achieves competitive macro-averaged performance, with especially strong results on big treebanks, but an input-processing bug harmed low-resource treebanks. After correcting the tokenizer, the system would rank highly overall and outperform all submissions on low-resource treebanks.
- Competitive performance across nearly all macro-averaged metrics places the system among the strongest single-system submissions.The comparison covers all treebanks and big treebanks, where the system achieves top performance on several metrics.
- UFeats F1 reaches the top on big treebanks, while consistency modeling improves the combined AllTags advantage over individual tagging metrics.The results suggest stronger overall annotation consistency even when reference systems are sometimes more accurate on individual tasks.
- The largest gap between all-treebank and big-treebank results comes from sentence segmentation, whose errors affect every downstream component.Downstream components process words at the sentence level, so tokenizer errors propagate through the pipeline.
- 2nd, 1st, and 3rd place on LAS, MLAS, and BLEX, respectively, would have been achieved by the corrected Stanford+ system.The corrected evaluation replaced only the tokenizer; on big treebanks, all metrics changed by at most 0.02% F1.
- All low-resource treebanks would have been outperformed by Stanford+ on every official metric by a large margin.The correction addressed a tokenizer data-preparation issue that caused the sentence segmenter to overfit to newlines.
5 Analysis
The ablation studies evaluate individual tokenizer, tagger, lemmatizer, parser, and pipeline components using macro-averaged dev or test F1. Results show gains from neural lemmatization, tag consistency modeling, parser location terms, and selected tokenizer components.
- Experimental setup: Ablations use oracle access to all other components, with macro-averaged F1 reported on development sets except for pipeline test results.Treebank-level paired bootstrap tests use 10^5 samples to assess significance.
- Tokenizer: +34.89 Words F1: convolutional residual connections help Hebrew MWT identification, while seq2seq improves word segmentation by +10.08 on Hebrew and +4.19 on Arabic.Gating and convolution have mixed treebank-level effects, whereas seq2seq helps every treebank where it makes a meaningful difference.
- POS/UFeats tagger: The biaffine tagger improves XPOS and UFeats prediction and is markedly more consistent across tags than an affine classifier with shared representations.UPOS performance is not significantly affected by the added parameters; the PMI analysis measures consistency beyond individual tag accuracies.
- Lemmatizer: +6.77 F1: adding the edit classifier and seq2seq module substantially improves lemmatization over a dictionary-only baseline across treebank groups.The full lemmatizer, including dictionaries, achieves the best overall performance; the largest reported gain is +4.55 F1 on Upper Sorbian-UFAL.
- Dependency parser: Parser distance and linearization terms both contribute significantly to performance, with the distance term contributing slightly more.The analysis excludes six treebanks with very small development sets.
6 Conclusion & Future Directions
Stanford presents a neural pipeline covering every stage from raw text to dependency parses, with strong performance on both large and low-resource treebanks after correcting a preprocessing bug. The authors identify context-sensitive word embeddings as a major opportunity for further gains, though they require substantial resources.
- The system uses neural components for every stage from raw text to dependency parses and was highly competitive on big treebanks.
- After correcting a preprocessing bug, the system would have outperformed all official systems on every metric for low-resource treebank categories.
- Context-sensitive word embeddings such as ELMo and ULMfit are identified as a major opportunity for further gains.The paper notes that incorporating these embeddings requires a large resource investment.