Source-linked AI summary
Universal Dependencies v2: An Evergrowing Multilingual Treebank Collection
Joakim Nivre, Marie-Catherine de Marneffe, Filip Ginter, Jan Hajič, Christopher D. Manning, Sampo Pyysalo, Sebastian Schuster, Francis Tyers, Daniel Zeman
TL;DR
Universal Dependencies develops cross-linguistically consistent annotation across word segmentation, morphology, and syntax. This paper updates the UD v2 guidelines, documents changes from UD v1, and surveys the expanding treebank collection and its geographic coverage.
Problem
Cross-linguistic treebank annotation requires shared categories and guidelines while accommodating typologically diverse languages and constructions.
Method
The paper describes UD v2 annotation guidelines, including morphological and syntactic changes, and reviews available treebank resources.
Results
UD treebanks span all inhabited continents, with later releases extending beyond the near-European coverage of UD v1.0.
Takeaways & Limitations
UD treebanks are widely used in NLP and linguistic research, while broader language coverage, more data, and continued consistency work remain priorities.
Takeaways & Limitations
Enhanced dependency representations remain optional and are implemented by only 24 treebanks as of UD release v2.5.
Abstract
from arXiv · showhide
Universal Dependencies is an open community effort to create cross-linguistically consistent treebank annotation for many languages within a dependency-based lexicalist framework. The annotation consists in a linguistically motivated word segmentation; a morphological layer comprising lemmas, universal part-of-speech tags, and standardized morphological features; and a syntactic layer focusing on syntactic relations between predicates, arguments and modifiers. In this paper, we describe version 2 of the guidelines (UD v2), discuss the major changes from UD v1 to UD v2, and give an overview of the currently available treebanks for 90 languages.
1. Introduction
Universal Dependencies is an open community project for consistent multilingual treebank annotation, supporting multilingual parsing and crosslingual learning. This paper updates the project through UD v2 guidelines and resource coverage.
- UD develops cross-linguistically consistent treebank annotation for many languages to support multilingual parser development and research on parsing and crosslingual learning.
- The project combines universal categories and guidelines with language-specific extensions when necessary.
- More than 300 researchers contribute to a rapidly growing project approaching 100 represented languages.
- The paper describes UD v2, emphasizes major changes from UD v1, and surveys existing treebank resources.
- UD resources supported multilingual dependency-parsing shared tasks, producing parsers that handle many languages and parse raw text.
2. Annotation Scheme
The UD annotation scheme represents syntactic words lexically, assigns standardized morphological information, and links words through typed dependencies. It combines universal inventories with mechanisms for language-specific variation and cross-linguistic comparison.
- Word segmentation: UD treats dependency relations as holding between words and represents morphological features as word properties without morpheme segmentation.Clitics may be split and contractions undone, creating multiword tokens when one orthographic token corresponds to multiple syntactic words.
- Morphological annotation: Each syntactic word receives a lemma, a part-of-speech tag, and features describing lexical and grammatical properties.The universal part-of-speech inventory contains 17 tags, while features provide finer-grained classification.
- Syntactic annotation: UD uses typed dependency relations in a basic tree, prioritizing direct predicate-argument and modifier links between content words.This design makes shared grammatical relations more transparent across languages with different word order, function words, or inflection.
- Cross-linguistic comparison: Parallel Czech, English, and Swedish sentences share core syntactic relations despite differing in definiteness, passive, and oblique-modifier encoding.The figure highlights common relations separately from morphosyntactic encoding differences.
- Syntactic annotation: The scheme provides 37 universal relation types covering clause-level and noun-phrase dependencies alongside special relations for function words and phenomena such as coordination and ellipsis.Enhanced dependencies can add or modify relations for a more complete basis for semantic interpretation, while the basic representation is obligatory.
3. Changes from UD v1 to UD v2
UD v2 revises tokenization, morphological features, syntactic relations, coordination, ellipsis, and enhanced dependencies to improve cross-linguistic consistency and downstream utility.
- Tokenization and Word Segmentation: UD v2 permits word-internal spaces for languages whose writing systems mark subword units and for explicitly documented exceptions such as numbers and abbreviations.The first policy avoids analyzing languages such as Vietnamese with fixed multiword expressions; the second permits multitoken words for convenience.
- Part-of-Speech Tags: The coordinating-conjunction tag changes from CONJ to CCONJ, while AUX, PART, and PRON/DET receive revised usage guidelines.AUX expands to copulas and nonverbal TAME particles, PART is restricted to documented items, and PRON/DET becomes more flexible cross-linguistically.
- Morphological Annotation: The universal morphological inventory adds features and values and renames or removes some existing ones to cover new languages and align with UniMorph.These revisions are summarized in Table 2.
- Syntactic Annotation: UD v2 revises predicate-dependent relations by replacing DOBJ with OBJ, subsuming passive labels under NSUBJ, CSUBJ, and AUX, and adding more explicit criteria.Passive distinctions can be retained with subtypes such as NSUBJ:PASS, CSUBJ:PASS, and AUX:PASS.
- Coordination: UD v2 attaches coordinating conjunctions and punctuation to the immediately succeeding conjunct, rather than to the first conjunct as in UD v1.The CONJ relation itself remains essentially symmetric, linking non-first conjuncts to the first one.
- Ellipsis: For gapping, UD v2 uses ORPHAN to represent anomalous sibling-like dependencies, while enhanced representations can fully resolve the predicate-argument structure.The new analysis preserves clause integrity and introduces fewer non-projective dependencies than the UD v1 REMNANT analysis.
- Functional and Enhanced Relations: UD v2 adds CLF, extends AUX, subsumes AUXPASS, restricts COP, and removes NEG as a universal relation in favor of a polarity feature.UD v2 also introduces guidelines for enhanced dependency graphs containing optional additional relations and nodes that make implicit relations explicit.
- Enhanced Dependencies: Enhanced graphs are optional, but as of UD v2.5 only 24 treebanks included them and fewer implemented all five enhancements.Many enhanced graphs can be computed automatically from basic dependency trees.
4. Available Treebanks
UD v2.5 contains 157 treebanks representing 90 languages, with substantial variation in data volume and broader geographic coverage than earlier releases. Coverage reaches all inhabited continents, although large gaps remain.
- Available resources: 157 treebanks represent 90 languages in UD release v2.5.The release reports the number of treebanks and represented languages.
- Available resources: Data volume varies from 36 sentences and 321 words for Skolt Sámi to over 200,000 sentences and nearly 4 million words for German.The paper emphasizes considerable heterogeneity in available annotated data across languages.
- Language coverage: The proportion of Indo-European languages decreased from 60% in v2.1 to 53% in v2.5.This change indicates a shift in the language-family composition of the collection.
- Geographic coverage: UD coverage expanded from being almost completely restricted to Europe in v1.0 to representing all inhabited continents by v2.5.The map tracks additions in v1.0, v2.0, and v2.5 using different colors and symbols.
- Geographic coverage: Despite this expansion, large white areas remain on the map, indicating continuing geographic gaps in coverage.The figure uses approximate language locations based on speaker populations or treebank creation sites.
5. Conclusion
UD treebanks have become widely used in NLP and typologically oriented linguistic research. The project’s next priorities are broader language and data coverage while maintaining cross-linguistic consistency.
- Impact: UD treebanks are now widely used in NLP and especially typologically oriented linguistic research.The conclusion characterizes this use as a major development over the project’s first five years.
- Future priorities: Future work includes adding languages to improve coverage of major language families.The paper also identifies the need for more annotated data in existing languages.
- Future priorities: More annotated data for existing languages would make the resources more useful for NLP and linguistic studies.This is stated as a separate future priority from adding languages.
- Future priorities: Applying shared categories and guidelines consistently across increasingly typologically diverse languages remains a long-term challenge.The conclusion presents continued community efforts as addressing this challenge.