Source-linked AI summary
UniMorph 2.0: Universal Morphology
Christo Kirov, Ryan Cotterell, John Sylak-Glassman, Géraldine Walther, Ekaterina Vylomova, Patrick Xia, Manaal Faruqui, Sabrina J. Mielke, Arya D. McCarthy, Sandra Kübler, David Yarowsky, Jason Eisner, Mans Hulden
TL;DR
UniMorph 2.0 addresses the need for broadly usable morphological resources across the world’s languages by improving collection, annotation, and dissemination. It combines grouped human annotation of Wiktionary with external-language data and supplementary gloss resources, producing a large, universally formatted resource released for collaborative use.
Problem
Complex morphology is widespread across languages, but multilingual NLP requires resources that represent inflection consistently, including for low-resource languages.
Method
The project groups Wiktionary tables by structure, annotates representative paradigms, maps data to the UniMorph schema, and supplements it with external sources and generated English glosses.
Results
The release contains 952,530 unique lemmas across 350 languages and provides standardized morphological data, supplementary resources, and English glosses.
Takeaways & Limitations
UniMorph 2.0 provides a collaborative, permissively licensed resource for multilingual morphological research and has supported the CoNLL 2017 Morphological Learning Shared Task.
Takeaways & Limitations
UniMorph 1.0’s heuristic extraction was error-prone because it could not fully cover Wiktionary’s inconsistent formatting.
Abstract
from arXiv · showhide
The Universal Morphology UniMorph project is a collaborative effort to improve how NLP handles complex morphology across the world's languages. The project releases annotated morphological data using a universal tagset, the UniMorph schema. Each inflected form is associated with a lemma, which typically carries its underlying lexical meaning, and a bundle of morphological features from our schema. Additional supporting data and tools are also released on a per-language basis when available. UniMorph is based at the Center for Language and Speech Processing (CLSP) at Johns Hopkins University in Baltimore, Maryland and is sponsored by the DARPA LORELEI program. This paper details advances made to the collection, annotation, and dissemination of project resources since the initial UniMorph release described at LREC 2016. lexical resources} }
1. Introduction
UniMorph addresses the challenge of complex morphology across languages by releasing universally annotated inflectional data. The project targets NLP applications and includes many low-resource languages.
- UniMorph releases inflected forms linked to lemmas and universal morphological feature bundles for multilingual NLP.
- Explicit morphology modeling has improved downstream tasks including machine translation, speech recognition, parsing, keyword spotting, and word embeddings.
- The resource includes inflected open-class words from many languages, including low-resource languages such as Quechua, Navajo, and Haida.
2. Wiktionary Extraction
UniMorph 2.0 replaces error-prone heuristic correction with grouped, human annotation of Wiktionary paradigms while preserving broad multilingual coverage. The resulting resource contains 952,530 lemmas across 350 languages and is linguistically verified.
- UniMorph 1.0 used statistical, visual, and positional heuristics to normalize inconsistent Wiktionary tables into a consistent tabular format.
- The extraction workflow identifies inflected forms, links them to descriptors, and converts descriptors into universal feature annotations.
- 952,530 unique noun, verb, and adjective lemmas across 350 languages comprise the resulting UniMorph dataset.
- UniMorph 1.0 was error-prone because its heuristics could not fully cover Wiktionary’s inconsistent formatting, causing incorrect feature-vector links.
- UniMorph 2.0 groups parsed HTML tables by structure and cell count, then has annotators label a sample table for each group and applies the mapping across that group.
- A second set of expert or native-speaker adjudicators verified the initial annotations, and the schema provides over 200 features across 23 dimensions of meaning.
3. Non-Wiktionary Data Sources
UniMorph 2.0 supplements Wiktionary-derived data with morphological resources from several external projects and manually or computationally prepared sources. These datasets are reformatted and mapped to the UniMorph standard.
- Additional data covers Khaling, Kurmanji Kurdish, Sorani Kurdish, Haida, and Basque from non-Wiktionary sources.
- Custom scripts reformat the additional datasets, while dataset-specific annotations are manually mapped to the UniMorph schema.
4. Supplementary Structured Data
UniMorph 2.0 adds bare-lemma listings and English multi-word glosses to complement its annotated morphological forms. The gloss-generation pipeline converts feature vectors into templates and fills them with generated English inflections.
- UniMorph 2.0 scrapes bare lemmas with part-of-speech labels and provides English multi-word glosses for complex inflected wordforms.
- The gloss pipeline converts UniMorph feature vectors into language-independent English templates such as V;1;PL;PST;IPFV → ‘(we) were VBG.’
- Given an inflected lemma, feature vector, and lemma translation, the system selects a gloss template and generates the corresponding English inflection.
- Generating complex multi-word tenses and robust English inflections from diverse, noisy translation dictionaries is identified as both a challenge and a strength.
5. Community Features
UniMorph is organized as a collaborative, openly released project with downloadable data, tools, and a universal annotation schema. Its resources include per-language counts of lemmas and forms.
- UniMorph releases its data and tools on a rolling basis under a permissive open-source license.
- The project provides a high-level portal and a separate hub for downloadable data and resources.
- Table 2 reports the total numbers of lemmas and forms available for each language.
6. Conclusion
UniMorph 2.0 contributes a large, high-quality multilingual morphology database built through type-based annotation and a universal tagging framework. Its openly released resources supported the CoNLL 2017 Shared Task on Morphological Learning.
- UniMorph 2.0 releases a large, high-quality database of morphological paradigms across typologically diverse languages.
- A type-based annotation procedure extracts substantial Wiktionary data while reducing human annotation effort and handling formatting variation across languages.
- UniMorph prescribes universal tagging and data formats, and its resources supported the first CoNLL shared task on morphology.