Source-linked AI summary
The CoNLL--SIGMORPHON 2018 Shared Task: Universal Morphological Reinflection
Ryan Cotterell, Christo Kirov, John Sylak-Glassman, Géraldine Walther, Ekaterina Vylomova, Arya D. McCarthy, Katharina Kann, Sabrina J. Mielke, Garrett Nicolai, Miikka Silfverberg, David Yarowsky, Jason Eisner, Mans Hulden
TL;DR
The shared task evaluated supervised morphological generation through isolated-form inflection and a harder cloze-style task requiring inflection in sentential context. Across 103 languages and seven context-task languages, newer approaches improved low-resource inflection results, while the cloze task remained difficult to improve over simple baselines.
Problem
The shared task examined supervised morphological generation for isolated inflection and the harder problem of selecting an appropriate inflected form in sentential context.
Method
The competition compared submitted systems across two tasks, three resource conditions, and datasets covering 103 languages for inflection and seven languages for contextual cloze inflection.
Results
41 of the 52 languages shared with 2017 improved in low-resource inflection, while the cloze task generally showed limited improvement over neural and lemma-copying baselines.
Takeaways & Limitations
The released datasets support future research on morphological learning in both isolated word inflection and sentential context.
Takeaways & Limitations
Clear improvements over the neural baseline remained difficult in the contextual task, possibly because of limited training data.
Abstract
from arXiv · showhide
The CoNLL--SIGMORPHON 2018 shared task on supervised learning of morphological generation featured data sets from 103 typologically diverse languages. Apart from extending the number of languages involved in earlier supervised tasks of generating inflected forms, this year the shared task also featured a new second task which asked participants to inflect words in sentential context, similar to a cloze task. This second task featured seven languages. Task 1 received 27 submissions and task 2 received 6 submissions. Both tasks featured a low, medium, and high data condition. Nearly all submissions featured a neural component and built on highly-ranked systems from the earlier 2017 shared task. In the inflection task (task 1), 41 of the 52 languages present in last year's inflection task showed improvement by the best systems in the low-resource setting. The cloze task (task 2) proved to be difficult, and few submissions managed to consistently improve upon both a simple neural baseline system and a lemma-repeating baseline.
1 Introduction
Morphological inflection expresses syntactic and semantic properties on word forms, which are systematically related across a word’s paradigm. The shared task evaluates systems that generate an inflected form from a source form and target morphosyntactic description.
- Morphological inflection expresses some syntactic and semantic properties on word forms.
- Training examples map a lemma and morphosyntactic feature bundle to an inflected form that must be predicted at test time.
- Inflected forms are systematically related, allowing speakers to predict unseen forms such as the plural of wug.
- Task 1 provides a source form and target morphosyntactic description, requiring systems to generate the actual inflected form.For example, the source form run can be converted to its present participle, running.
2 Tasks and Evaluation
The shared task extended sparse-data morphological generation to 103 languages and added a seven-language cloze task requiring inflection in sentential context. Task 2 varied contextual supervision across two tracks and three data conditions.
- The first task extended the 2017 inflection task from 52 to 103 languages using sparse morphological-generation data.Participants received examples of inflected forms and generated another form from a lemma and morphosyntactic feature bundle.
- Task 1: Task 1 required generating an inflected form from an input lemma and desired output tags.
- Task 1: Training data consisted of individual forms sparsely sampled from many large paradigms, with more frequent forms more likely to appear.
- Task 1: Systems were evaluated after training on 10^2, 10^3, and 10^4 lemma/MSD/inflected-form triplets in low, medium, and high conditions.
- Task 2: Inflection in Context: Task 2 asked systems to inflect a lemma in context, requiring evidence from morphology, morphosyntax, semantics, pragmatics, and potentially register or genre.
- Task 2: Inflection in Context: In track 1, systems observed context forms, lemmata, and MSDs; in track 2, they observed only context forms, while the target MSD was hidden in both.
- Task 2: Inflection in Context: Track 1 supplied full morphosyntactic annotation, whereas track 2 supplied lemmata for selected tokens and left remaining tokens without annotation.
- Task 2: Inflection in Context: Both tracks used low, medium, and high training conditions of approximately 10^3, 10^4, and 10^5 tokens, with fewer effective examples in track 2.External semi-supervised data were forbidden.
3 Data
The shared task assembled multilingual morphological data from Wiktionary and Universal Dependencies, converted into the UniMorph schema and sampled into varied resource conditions. Task 2 used contextual UD sentences, while manual annotation supplied plausible alternatives for evaluation.
- The shared task covered 103 genealogically diverse languages, including isolates and languages from roughly 20 language stocks.
- Task 1 represented examples as lemma, UniMorph feature bundle, and inflected-form triples extracted from Wiktionary paradigms.
- Task 1 data were sampled into train, development, and test splits using frequency-weighted distributions over paradigm triples.
- Task 2 used Universal Dependencies treebanks, retaining word forms, lemmata, part-of-speech tags, and morphosyntactic descriptions while discarding dependency trees.
- Task 2 evaluated reconstruction of original forms and prediction of manually annotated contextually plausible alternatives across seven languages.
- UD morphosyntactic descriptions were deterministically converted to UniMorph, but UD includes finer-grained features such as inherent noun gender absent from UniMorph.
4 Baselines
The shared task used rule-based and neural baselines for morphological reinflection. The neural baseline encoded lemma characters together with contextual information and decoded the target form with attention.
- The Task 1 baseline extracted prefix and suffix transformation rules from observed citation-form inflections and applied the longest matching rule at generation time.
- The neural baseline used a bidirectional LSTM encoder and additive-attention decoder to generate an inflected form from encoded input representations.
- In track 1, the encoder additionally used left and right context lemmata and morphosyntactic descriptions.
- In track 2, the encoder combined input characters with left and right context-word embeddings and a whole-token lemma embedding.
- The neural baseline used 100-dimensional embeddings and single-layer encoder and decoder LSTMs, trained for 20 epochs with Adam.
- The copy baseline simply returned the input lemma, exploiting the frequency of lemma forms and providing a strong small-data comparison in some languages.
5 Results
Task 1 systems improved substantially over 2017, especially in low-resource settings, while Task 2 remained difficult despite varied neural architectures and contextual modeling strategies.
- Task 1 results: 41 of 52 shared languages improved in top-system performance under the low-resource condition compared with 2017.
- Task 1 results: Edit-sequence generation and data augmentation generally improved performance in low-data settings, where standard encoder-decoder models suffer from sparsity.
- Task 1 results: The ensemble oracle exceeded any individual system by approximately 10% in medium-resource and 25% in low-resource conditions.
- Task 1 results: Feature-bundle decomposition enabled strong generalization beyond holistic feature memorization, with several systems surpassing the feature-combination oracle on Basque.
- Task 2 results: Task 2 systems used contextual MSD prediction, subword and longer-range context encoders, context attention, multilingual training, and beam search.
- Task 2 results: The best Track 1 system outperformed the best Track 2 system in every data setting, while the performance gap between tracks grew from 3.8 to 13.6 percentage points.
6 Future Directions
Future work targets richer representations, broader multilingual learning, and more data for contextual inflection. The results also suggest that simpler evaluation could expand language coverage.
- Representations: IPA-like representations could address challenges posed by languages with idiosyncratic orthographies.The paper proposes transductions over pronunciations as a future direction.
- Additional data: Unannotated monolingual corpora were absent from both tasks and could support semi-supervised learning, especially for low-resource cloze settings.Suggested approaches include pretrained embeddings or models dedicated to morphological inflection.
- Multilingual learning: Only one team trained a multilingual inflection model, leaving cross-language transfer as an open research direction.Future evaluation should disentangle transfer from the effect of simply providing more data that biases networks toward copying strings.
- Dataset design: Future shared tasks should consider datasets that more accurately reflect learner exposure.The paper identifies this as an important design consideration for both first- and second-language learning settings.
- Evaluation: Similar rankings for original and plausible forms justify using the simpler original-form evaluation in future shared tasks.Removing manual annotation could allow test sets covering a wider variety of languages.
7 Conclusion
The 2018 shared task broadened morphological generation to 103 languages and added a seven-language cloze task. Neural systems dominated, but beating simple baselines in low-data settings required substantial architectural modifications.
- Scope: The shared task covered 103 languages in inflection and introduced a cloze-test task covering 7 languages.Task 1 received 27 submissions and Task 2 received 6 submissions.
- Conclusion: Neural network models prevailed in both tasks, while modified architectures were needed to beat simple low-data baselines.The conclusion identifies this pattern across both the inflection and cloze tasks.
- Resources: The released training, development, and test sets support future research on morphological learning in isolation and sentential context.The resources cover both isolated word inflection and contextual inflection.
A Detailed Task 1 Results
Task 1 reports detailed per-system and per-language results across three metrics and three data conditions. The evaluation also compares submissions with feature-combination and ensemble oracles.
- Evaluation: The results section presents three metrics for every system-language combination, ordered by average per-form accuracy within each subtask and data condition.The listed metrics include per-form accuracy and Levenshtein distance; the supplied passages identify these two explicitly.
- Statistical comparisons: Scores identify the highest-scoring non-oracle systems and statistically indistinguishable alternatives using a sign test with p >= 0.05.Additional markers indicate significant improvements over the feature-combination oracle or no significant difference from the ensemble oracle.