Source-linked AI summary
CoNLL-SIGMORPHON 2017 Shared Task: Universal Morphological Reinflection in 52 Languages
Ryan Cotterell, Christo Kirov, John Sylak-Glassman, Géraldine Walther, Ekaterina Vylomova, Patrick Xia, Manaal Faruqui, Sandra Kübler, David Yarowsky, Jason Eisner, Mans Hulden
TL;DR
The paper studies supervised morphological generation across 52 typologically diverse languages under sparse and dense learning scenarios. It organizes shared-task evaluations for individual-form inflection and paradigm completion across resource conditions. Neural systems achieved strong performance with small datasets when supported by suitable biasing or data augmentation, while complementary system behaviors left room for improvement.
Problem
Computational morphology has received less attention despite rich morphology being common across languages and supporting many human language technology tasks.
Method
The shared task evaluates supervised individual-form inflection and paradigm completion across 52 languages with low-, medium-, and high-resource data conditions.
Results
Over 50% exact-match accuracy was achieved with 100 examples and 80% with 1,000 examples, versus 38% for a simple inflectional-rule baseline.
Takeaways & Limitations
Appropriate inductive bias, unlabeled data, or synthetic data can enable strong neural morphological generation from small training datasets.
Takeaways & Limitations
The best systems did not use external data, and effective incorporation of unannotated monolingual corpora remains an open question, especially in lower-resource settings.
Abstract
from arXiv · showhide
The CoNLL-SIGMORPHON 2017 shared task on supervised morphological generation required systems to be trained and tested in each of 52 typologically diverse languages. In sub-task 1, submitted systems were asked to predict a specific inflected form of a given lemma. In sub-task 2, systems were given a lemma and some of its specific inflected forms, and asked to complete the inflectional paradigm by predicting all of the remaining inflected forms. Both sub-tasks included high, medium, and low-resource conditions. Sub-task 1 received 24 system submissions, while sub-task 2 received 3 system submissions. Following the success of neural sequence-to-sequence models in the SIGMORPHON 2016 shared task, all but one of the submissions included a neural component. The results show that high performance can be achieved with small training datasets, so long as models have appropriate inductive bias or make use of additional unlabeled data or synthetic data. However, different biasing and data augmentation resulted in disjoint sets of inflected forms being predicted correctly, suggesting that there is room for future improvement.
1 Introduction
The shared task targets computational morphological inflection across typologically diverse languages, addressing a field whose importance is underscored by the prevalence of rich morphology worldwide. It evaluates robust systems under varied resource conditions and reports strong neural performance with small datasets when augmented or appropriately biased.
- Morphology supports HLT tasks including machine translation, speech recognition, parsing, keyword spotting, and word embeddings.
- 80% of the world’s languages mark verb tense morphologically, while 65% mark grammatical case.
- The task covered 52 languages spanning diverse language families, morphological properties, and low-, medium-, and high-resource conditions.Many included languages were extremely low-resource; the inflection conditions generally used 100, 1,000, or 10,000 training examples.
- The shared task promoted robust systems for morphological inflection and paradigm cell filling using varying amounts of training data.Its training, development, and test data were released publicly.
- Encoder-decoder recurrent neural networks performed very well with small training sets when augmented with mechanisms for low-resource learning.
2 Task and Evaluation Details
The shared task compares supervised morphological generation in two learning scenarios: sparse individual-form inflection and completion of partially observed paradigms. Systems are evaluated across 52 languages, multiple training sizes, and complementary accuracy and distance metrics.
- 2.1 Sub-Task 1: Inflected Form from Lemma: Sub-task 1 maps a lemma and morphosyntactic feature bundle to a requested inflected form.Training examples contain individual forms sparsely sampled from many paradigms, and unannotated corpus data was also provided.
- The two sub-tasks model different practical learning settings: sparse exposure for downstream NLP and extrapolation from a few complete paradigms.The latter reflects compiled grammatical resources, fieldwork with native informants, or classroom generalization.
- 2.2 Sub-Task 2: Paradigm Completion: Sub-task 2 completes missing cells in sparsely populated paradigms using complete paradigms as training data.It is also called the paradigm cell filling problem.
- 2.2 Sub-Task 2: Paradigm Completion: Sub-task 2 evaluates full-paradigm accuracy, whereas sub-task 1 provides only a few inflected forms per lemma in training.Both tasks require predicting multiple inflections, but sub-task 2 supplies complete training paradigms.
- 2.3 Evaluation: Training conditions ranged from a few complete paradigms to dozens, while each system was evaluated across its language-specific test set.
- 2.3 Evaluation: The evaluation measured overall 1-best accuracy, average Levenshtein distance, and full-paradigm accuracy.Metrics were averaged across all 52 languages for each system and resource condition; full-paradigm accuracy is chiefly meaningful for sub-task 2.
3 Data
The task assembled multilingual morphological data from diverse sources and sampled it into resource-controlled training, development, and test sets. Sampling used frequency estimates for sub-task 1 and paradigm-level counts for sub-task 2, with known ambiguity and unequal exemplar counts.
- The dataset covered 52 genealogically and morphologically diverse languages, including low-resource languages such as Quechua, Navajo, Haida, and language isolates.
- Each example represented a lemma, UniMorph feature bundle, and inflected form, while Wiktionary tables supplied full paradigms for many languages.
- String-match frequency estimates overcounted ambiguous forms, producing an average of 1.43 triples per inflected-form string across languages.
- Sub-task 1 training sets contained 100, 1,000, or 10,000 sampled triples in low, medium, and high conditions, respectively, with nested sets favoring higher-count forms.
- Sub-task 2 sampled paradigms by Wikipedia frequency, while test paradigms exposed about 15 of their slots and required predicting the remaining forms.
4 Previous Work
Previous work addressed morphological inflection with finite-state, feature-based, Bayesian, and neural approaches. Neural sequence-to-sequence systems had generally achieved the strongest results, while paradigm completion had received comparatively less attention.
- Earlier systems reduced inflection to pattern selection with finite-state generalization and feature-rich classification.
- Neural sequence-to-sequence models, including attention-based systems, were generally the most successful recent approaches to inflection generation.
- Other approaches modeled paradigms using Markov random fields or Bayesian networks with structured dependencies among paradigm cells.
5 The Baseline System
The baseline system learned feature-bundle-specific prefix and suffix edits from aligned lemma–form pairs, then applied them to generate inflections. It handled prefix-heavy languages by reversing strings but did not exploit partially observed paradigms.
- The baseline aligned each lemma with its inflected form using minimum-cost edits, then divided the alignment into prefix, stem, and suffix regions.
- It extracted prefix-changing and suffix-changing rules and associated them with the exact morphosyntactic feature bundle that produced each training form.
- At test time, the system applied the longest matching suffix rule and the most frequent prefix rule for the requested feature bundle.
- The baseline had no generalization across feature bundles and generated missing paradigm forms independently from the lemma.
- For largely prefixing languages, reversing strings enabled more expressive changes at the left edge after classifying languages by prefix-versus-suffix changes.
6 System Descriptions
Most submissions used neural models, but systems differed substantially in alignment, reranking, and especially auxiliary-data strategies. Oracle results indicate that these approaches often made complementary correct predictions.
- The task received 11 team submissions representing 25 unique systems including the baseline, while all but one submitted system included a neural component.
- Despite similar neural architectures, individual performances differed substantially, with auxiliary training-data methods emerging as a major source of variation.
- Systems varied between soft attention and hard monotonic alignment, with the winning CLUZH system extending a successful monotonic-alignment approach.
- Teams used reranking, unlabeled corpora, synthetic forms, and other augmentation strategies, including copying biases and hallucinated training pairs.
7 Performance of the Systems
Performance was strongest with plentiful data, while low- and medium-resource success depended on inductive biases and produced complementary errors across systems.
- External resources yielded approximately 1% gains for UE-LMU in Medium and 3% for UA in Low, with limited overall impact.CMU was outranked by several systems without external resources, and UA’s submissions covered only a small handful of languages.
- High-resource systems performed very well, but standard encoder-decoder architectures underperformed the training-data-only baseline in low and medium conditions.Systems that surpassed the baseline biased networks toward language-specific inflectional patterns using synthetic-data pretraining.
- 15% and 10% were the approximate Medium- and Low-condition gains of the Ensemble Oracle over individual systems.The oracle was correct whenever any submitted system was correct, indicating substantially different generalization patterns.
- 71% was the low-data Feature Combination Oracle upper bound, leaving 29% of test feature bundles unseen during training.Systems could still generalize by decomposing bundles into individual features; several recurrent models used separate feature inputs.
- Neural systems correctly generalized to unseen feature combinations, sometimes surpassing the atomic feature-bundle upper bound.This result demonstrates that modeling individual morphological features supports predictions beyond combinations observed in training.
8 Future Directions
The paper identifies future work in phonological inflection, using unlabeled and cross-lingual data, selecting informative paradigm cells, and standardizing morphology tasks.
- Future systems could model morphological transductions over pronunciations rather than spellings, including phonological changes and predictable allophonic distinctions.Orthography often fails to reflect these pronunciation patterns, making this direction more challenging.
- Effectively incorporating external unannotated monolingual corpora remains an open question, especially for lower-resource inflection and reinflection.The best competition systems did not use external data, while heavy use produced little gain; cross-lingual information was helpful in related work.
- Active learning should investigate which paradigm cells to annotate first, with diagnostic forms suggested as potentially informative.Speakers may also rely strongly on pattern frequencies, motivating experiments with plausible real frequency distributions.
- Standardized shared-task datasets could support fair comparison for supervised morphological segmentation, tagging, and other underexplored morphology tasks.Many morphology tasks have not yet received shared-task treatment despite the value of standardized data for comparing methods.
9 Conclusion
The shared task evaluated neural and other systems for inflection and paradigm completion across 52 languages and varied data conditions. Neural encoder-decoder systems performed strongly with abundant data and, with suitable biasing or augmentation, also achieved substantial accuracy in low-resource settings, while complementary errors leave room for improvement.
- 24 systems entered inflection, while 3 entered paradigm completion, and all but one used neural models.
- Over 90% exact-match accuracy was achieved on held-out forms in many languages when training data was plentiful.This reinforced the strong performance of encoder-decoder architectures observed in the 2016 shared task.
- Over 50% exact-match accuracy was achieved with 100 examples and 80% with 1,000 examples, compared with 38% for a simple inflectional-rule baseline.Without data augmentation, baseline neural systems in the low-data condition reached only 0–1% accuracy.
- Oracle ensembling showed substantial remaining room for improvement, particularly in low-resource settings.
- The released training, development, and test sets provide a benchmark for future research on inflectional morphology and string-to-string transduction.