Source-linked AI summary
The SIGMORPHON 2019 Shared Task: Morphological Analysis in Context and Cross-Lingual Transfer for Inflection
Arya D. McCarthy, Ekaterina Vylomova, Shijie Wu, Chaitanya Malaviya, Lawrence Wolf-Sonkin, Garrett Nicolai, Christo Kirov, Miikka Silfverberg, Sabrina J. Mielke, Jeffrey Heinz, Ryan Cotterell, Mans Hulden
TL;DR
The shared task addresses limited evidence and resources for modeling morphology, including low-resource inflection and the need to analyze forms in context. It evaluates cross-lingual inflection transfer and contextual lemmatization and tagging. Task 1 transfer gains were generally modest and positively correlated with language similarity, while the most successful Task 2 systems used varied contextual architectures and embeddings.
Problem
Large inflectional paradigms and limited supervised data make morphological modeling difficult, while contextual analysis must account for forms whose category depends on sentence context.
Source: §1 · p. 1, §2.2 · p. 2Method
The shared task evaluates low-resource inflection using high-resource-language transfer and sentence-level lemmatization and morphosyntactic tagging in context.
Source: §1 · p. 1Results
Task 1 cross-lingual training gains were generally modest and positively correlated with language similarity; the most successful Task 2 systems used variations of multiheaded attention, multi-level encoding, multiple decoders, and ELMo or BERT contextual embeddings.
Source: §7 · p. 13Takeaways & Limitations
The released training, development, and test sets are intended as a benchmark for future research on inflectional morphology and string-to-string transduction.
Source: §7 · p. 13Evaluation of multilingual inflection transfer may need to disentangle learned cross-language transfer from the effect of having more data on copying strings.
Source: §6 · p. 13
Abstract
from arXiv · showhide
The SIGMORPHON 2019 shared task on cross-lingual transfer and contextual analysis in morphology examined transfer learning of inflection between 100 language pairs, as well as contextual lemmatization and morphosyntactic description in 66 languages. The first task evolves past years' inflection tasks by examining transfer of morphological inflection knowledge from a high-resource language to a low-resource language. This year also presents a new second challenge on lemmatization and morphological feature analysis in context. All submissions featured a neural component and built on either this year's strong baselines or highly ranked systems from previous years' shared tasks. Every participating team improved in accuracy over the baselines for the inflection task (though not Levenshtein distance), and every team in the contextual analysis task improved on both state-of-the-art neural and non-neural baselines.
1 Introduction
Language models must handle morphology as well as syntax, but large inflectional paradigms and limited supervised data make this difficult, especially for many languages. The shared task addresses this through low-resource inflection with cross-lingual transfer and contextual lemmatization and morphosyntactic tagging.
1 Introduction
- Human listeners often identify disfluency even when a sentence’s meaning remains mostly clear.Fluent language combines meaning conveyed through word choice and tone with structural rules expressed through syntax and morphology.Source: §1 · p. 1
- Strong language models can reconstruct common syntax but remain insufficient for properly modeling morphology.Many languages use large inflectional paradigms to mark morphosyntactic information on both function and content words.Source: §1 · p. 1
- Large, complex inflectional paradigms require systems to produce many forms, while the supervised data needed for strong inflection systems is infeasible to obtain for many languages.Archi verbs can take thousands of forms, many of which may never occur in billions of lines of input.Source: §1 · p. 1
- The shared task targets low-resource inflection using related high-resource languages and contextual lemmatization with morphosyntactic tagging.The paper reports that the best submitted systems represent the state of the art in their respective tasks.Source: §1 · p. 1
2 Tasks and Evaluation
Task 1 evaluates whether systems can use high-resource-language data to inflect words in a low-resource language. Task 2 evaluates sentence-level lemmatization and morphosyntactic tagging, where context helps resolve the categories and lemmas of word forms.
2.1 Task 1: Cross-lingual transfer for morphological inflection
- Task 1 asks systems to inflect low-resource-language lemmas from morphosyntactic features, using limited target-language examples alongside more data from a related or unrelated language.The task permits fine-tuning, joint modeling, or other transfer methods; exact-match accuracy and average Levenshtein distance evaluate predictions.Source: §2.1 · p. 2
2.2 Task: Morphological analysis in context
- Context-agnostic inflection is useful for evaluating morphological quality, but morphology is not learned in isolation.The task builds on the prior contextual inflection challenge while requiring full-sentence analysis.Source: §2.2 · p. 2
2.2 Task 2: Morphological analysis in context
- Task 2 requires systems to lemmatize and assign a morphosyntactic description to every word in a sentence.Unlike the previous year’s single-word-in-context inflection task, this task analyzes the full sentence; context can change a form’s category and lemma.Source: §2.2 · p. 2
- Context matters because identical word forms can realize different inflectional categories, affecting lemmatization decisions.In the example, “barking” is an adjective with the lemma “barking” in “The barking dogs kept us up all night.”Source: §2.2 · p. 2
3 Data
Task 1 data pair limited low-resource training examples with a larger high-resource set, sampled from language paradigms. Task 2 used Universal Dependencies treebanks, converted to a common morphological schema with some source errors left uncorrected.
3.1 Data for Task 1
- Task 1 provided data for 100 language pairs spanning 79 languages, with 20 pairs distantly related or unrelated.Most language data came from English Wiktionary; the exceptions listed were Basque, Kurmanji, Murrinhpatha, and Sorani.Source: §3.1 · pp. 2–3
- Task 1 examples are lemma, feature-bundle, and inflected-form triples, with separate high- and low-resource training files and features coded using the UniMorph Schema.Paradigms were extracted from Wiktionary tables using a template annotation procedure; the datasets were sampled from distributions informed by inflected-form counts in a February 2017 Wikipedia dump.Source: §3.1 · p. 3
- Task 1 sampling used the first 100 of 12,000 sampled triples for low-resource training and the first 10,000 for high-resource training; the remaining 2,000 were split into development and test sets.Development and test sets were shuffled to be similar to each other and generally contained lower-count triples than training data; some languages had fewer examples, and some splits used 50 or 100 examples when data were scarce.Source: §3.1 · p. 3, §3.2 · p. 3
3.2 Data for Task 2
- The training setup reflects generalization from frequent annotated forms to less frequent forms, which tend to appear in development and test data.The paper also notes that unsupervised methods tend to generalize from frequent to less frequent words.Source: §3.1 · p. 3, §3.2 · p. 3
- Task 2 used the predefined train, development, and test splits from 107 Universal Dependencies monolingual treebanks, converted to the UniMorph schema using a deterministic mapping.The organizers did not individually correct errors in the source material; some languages received additional preprocessing.Source: §3.2 · p. 3
- Task 2 data were not identical across all preprocessing steps: Finnish morpheme boundaries were removed and Russian GSD lemmas lowercased.The development and test data had all fields except form and sentence index struck.Source: §3.2 · pp. 3–4
4 Baselines
The Task 1 baseline compares four neural sequence-to-sequence inflection models that vary in their attention and alignment assumptions. Task 2 baselines include non-neural and neural joint tagging-and-lemmatization systems.
4.1 Task 1 Baseline
- The Task 1 baselines are four neural sequence-to-sequence models with soft, non-monotonic hard, monotonic hard, or offset-based monotonic hard attention.They encode with a biLSTM and decode left-to-right with an LSTM; hard-attention training uses dynamic programming to marginalize latent alignments exactly.Source: §4.1 · p. 4
4.2 Task 2 Baselines
- The non-neural Lemming baseline jointly performs morphological tagging and lemmatization with a globally normalized second-order linear-chain CRF.It prunes lemma choices using pre-extracted edit trees to calculate the partition function efficiently.Source: §4.2 · p. 4
- The supplied team-score table averages scores across languages, but scores marked * are incomparable because they cover only a subset of languages.The † marker indicates submissions that used additional training resources.Source: Table 2 · p. 4
- The neural Task 2 baseline jointly tags and lemmatizes, using maximum likelihood to account for exposure bias and jackknifing to expose the lemmatizer to tagger errors during training.Its character-level biLSTM embeds words, a word-level biLSTM predicts tag sequences, and a sequence-to-sequence lemmatizer uses decoded tags as an additional attribute.Source: §4.2 · p. 4
5 Results
The results cover 30 submissions across two tasks. Task 1 approaches reported pair-dependent or modest cross-lingual gains, while Task 2 teams reported strong outcomes using pretrained embeddings and other contextual-analysis methods.
5.1 Task 1 Results
- UAlberta found cognate projection promising for certain language pairs; Tuebingen reported modest cross-lingual gains, while AX-Semantics combined datasets with optional target-focused adaptation.Source: §5.1 · p. 7
- CMU’s character-copy training significantly outperformed the Task 1 baseline, with further gains from fine-tuning and augmentation.Cross-lingual gains were highly correlated with linguistic similarity; hallucinated forms and multiple related target languages further improved the model.Source: §5.1 · p. 7
- IT-IST’s gated dual attention and SparseMax jointly learned stem and affix modifications, improving significantly over the baseline.Source: §5.1 · p. 7
5.2 Task 2 Results
- ITU slightly improved baseline lemmatization but significantly improved tagging, while RUG reported large gains with ELMo and further gains from related-language joint training..Source: §5.2 · p. 7
- CMU’s Task 2 system predicted POS before individual features, and experiments found little gain from adding gold typological information.Source: §5.2 · p. 7
- UFAL-Prague ranked first in lemmatization and second in tagging, while Charles-Saarland achieved the highest overall tagging accuracy.UFAL-Prague replaced pretrained embeddings with BERT; Charles-Saarland fine-tuned multilingual BERT on a concatenation of all available languages.Source: §5.2 · pp. 7–8
- CUNI–Malta used edit actions for lemmatization and a bidirectional LSTM with two-stage decoding for feature tagging.Source: §5.2 · p. 8
- Edinburgh used a character-based LSTM encoder-decoder on three-word context chunks; CBNU and Ohio State used distinct pipelines or decoders for tagging.Source: §5.2 · pp. 7–8
6 Future Directions
The section identifies future work in multilingual transfer, data design, and representation for inflection, alongside cross-lingual analysis for contextual morphology. It also points to phonological representations as a possible alternative to orthography.
6 Future Directions
- The authors identify typology, multiple helper languages, and unannotated monolingual corpora as avenues for improving multilingual morphology research.The shared task used only one helper language and included no unannotated monolingual corpora; such data may affect low-resource performance.Source: §6 · pp. 8, 13
- For inflection, the authors suggest moving beyond orthographic forms to IPA-like representations and pronunciation transductions.Languages with idiosyncratic orthographies may pose new challenges for this approach.Source: §6 · p. 13
- Only one team tested multilingual inflection, and evaluation must distinguish learned cross-language transfer from extra-data bias toward copying strings.The authors note that evaluating this distinction could be difficult; Jin and Kann (2017) addressed it.Source: §6 · p. 13
- Future shared-task datasets should reflect learner exposure and capture inflectional categories not reliably specified by lemmas.The authors cite Romanian verbal inflection classes and German nominal gender as examples.Source: §6 · p. 13
- The authors propose testing which representations matter for adapting morphology to new languages, including their role in avoiding catastrophic forgetting.They suggest probing representations in the style of Thompson et al. (2018), with a view toward systems that avoid catastrophic forgetting as they learn new languages (Thompson et al., 2019).Source: §6 · p. 13
- For contextual analysis, they propose cross-lingual analysis and using its results in downstream tasks..Source: §6 · p. 13
7 Conclusions
The shared task evaluated low-resource inflection with cross-lingual transfer and contextual lemmatization and morphological analysis across many languages. Inflection transfer gains were generally modest and associated with language similarity, while the most successful contextual-analysis systems used varied architectures and contextual embeddings.
7 Conclusions
- The evaluation covered 100 language pairs in 79 languages for inflection and 107 treebanks in 66 languages for analysis, with 14 and 16 systems submitted, respectively.All submitted systems used neural network models.Source: §7 · p. 13
- Task 1 gains from cross-lingual training were generally modest and positively correlated with language similarity; task 2's most successful systems used varied attention, encoding, decoder, and contextual-embedding approaches.The task 2 approaches included multiheaded attention, multi-level encoding, multiple decoders, and ELMo and BERT contextual embeddings.Source: §7 · p. 13
- The authors released training, development, and test sets as benchmarks for future work on inflectional morphology and string-to-string transduction..Source: §7 · p. 13