Source-linked AI summary
Joint Lemmatization and Morphological Tagging with LEMMING
Thomas Muller, Ryan Cotterell, Alexander Fraser, Hinrich Schütze
TL;DR
Inflected forms need to connect to lexical resources, and lemmatizing unknown forms can require global lemma features; tagging and lemmatization can also inform one another. LEMMING jointly models token-level lemmata and morphological tags without requiring morphological dictionaries, and reports state-of-the-art lemmatization on six languages alongside benefits from joint modeling.
Problem
Lemmatization links inflected forms to lexical resources and addresses sparsity, while unknown forms can require global lemma features that previous work did not support.
Source: §1 · p. 1Method
LEMMING is a modular log-linear model that jointly predicts token-level lemmata and morphological tags using edit-tree candidates and supports arbitrary global lemma features.
Source: §1 · p. 1, §2.1 · p. 2, §3 · p. 3Results
LEMMING sets a new state of the art in token-based statistical lemmatization on six languages, and joint modeling significantly improves tag-plus-lemma accuracy in four of six.
Source: §1 · p. 1Takeaways & Limitations
The authors conclude that joint modeling benefits both lemmatization and tagging, and that LEMMING achieves state-of-the-art token-based lemmatization across six languages.
Source: §6 · p. 5The reported lemmatization-accuracy and unknown-form measures ignore capitalization.
Source: §5 · p. 4
Abstract
from arXiv · showhide
We present LEMMING, a modular log-linear model that jointly models lemmatization and tagging and supports the integration of arbitrary global features. It is trainable on corpora annotated with gold standard tags and lemmata and does not rely on morphological dictionaries or analyzers. LEMMING sets the new state of the art in token-based statistical lemmatization on six languages; e.g., for Czech lemmatization, we reduce the error by 60%, from 4.05 to 1.58. We also give empirical evidence that jointly modeling morphological tags and lemmata is mutually beneficial.
1 Introduction
Lemmatization links inflected forms to lexical resources and helps address sparsity, but unknown forms can require global lemma features that earlier work did not support. Because tags can inform lemmatization and lemmas can inform tagging, the paper proposes a joint token-level model.
1 Introduction
- Lemmata connect inflected forms to lexical resources and help address sparsity, while tags can clarify lemmas and lemmas can help resolve tags under syncretism.For example, German plural noun phrases do not mark gender, making the lemma relevant to gender tagging.Source: §1 · p. 1
- Global lemma features are needed for ambiguous unknown-word transformations, such as Spanish verb forms whose shared ending does not identify the paradigm.Previous work cited here did not support such features.Source: §1 · p. 1
- The paper presents a joint log-linear model for token-level morphological analysis and lemmatization that handles unknown forms without morphological dictionaries or analyzers.It can be trained on corpora annotated with gold-standard tags and lemmata.Source: §1 · p. 1
- Across six languages, LEMMING sets a new state of the art in token-based statistical lemmatization; joint modeling significantly improves tag-plus-lemma accuracy in four languages.For Czech, lemma errors fall by more than 37% and tag-plus-lemma errors by more than 6%.Source: §1 · p. 1
2 Log-Linear Lemmatization
LEMMING formulates lemmatization as a log-linear choice among candidate lemmata, using edit trees to generate candidates and a feature set that can include global lemma information. Its feature design combines edit-tree, alignment, lemma, and dictionary signals.
2 Log-Linear Lemmatization
- The log-linear model scores candidate lemmata using hand-crafted features and a support function that determines which candidates receive non-zero probability.The support must balance computational cost against the risk of excluding the correct lemma.Source: §2 · p. 2, §2.1 · p. 2
2.1 Candidate selection
- Edit trees recursively represent prefix and suffix transformations around a longest common substring, using a substitution operation when no common substring exists.The candidate set applies reusable edit trees to a form and adds lemmata observed for that form in training.Source: §2.1 · p. 2
2.2 Features
- The feature design combines edit-tree and form-affix features with alignments, lemma features, dictionary signals, and capitalization variants.Alignment pairs can be joined with form and lemma character context; lemma prefixes and suffixes provide additional features.Source: §2.2 · p. 2
3 Joint Tagging and Lemmatization
The joint model combines a pruned higher-order CRF for morphological tags with lemmatization in a tree-structured CRF. It uses global normalization, belief propagation for inference, and SGD for parameter estimation.
3 Joint Tagging and Lemmatization
- Morphological tag sequences are modeled with MARMOT, a pruned higher-order CRF whose features include words, affixes, shape, and immediate lexical context.Pruning avoids the exponential runtime of higher-order models.Source: §3 · p. 3
- The model combines lemmatization and higher-order tagging in a tree-structured CRF that globally scores sequences of forms, lemmata, and morphological-plus-POS tags.Lemma and tag cliques use separate feature functions and weight vectors; inference uses belief propagation and parameter estimation uses SGD.Source: §3 · p. 3
- Initializing the joint model with parameters from a pretrained tagging model greatly improved its results.Source: §3 · p. 3
4 Related Work
Related systems also model lemmatization through candidate transformations, but differ in their prediction setting and resource requirements. LEMMING targets token-level tagging and lemmatization in context, including words absent from a morphological dictionary.
4 Related Work
- MORFETTE generates lemma candidates from edit operations and trains separate maximum-entropy Markov models for tagging and lemmatization, queried by beam search.The described related systems include work on tagging, lemmatization, and generation.Source: §4 · p. 3
- Toutanova and Cherry predict possible lemma and coarse-POS sets for word types, unlike LEMMING's fine-grained tagging of tokens in context.The passage says direct comparison is not possible because the tasks differ.Source: §4 · p. 3
- LEMMING uses Toutanova and Cherry's string-to-string transduction model as a standalone baseline, while its edit-tree candidate approach supports efficient training and application.The passage attributes this efficiency to high candidate coverage with a low average number of lemma candidates.Source: §4 · p. 3
- Smith et al.'s dictionary-dependent model jointly handles tagging, segmentation, and lemmatization but is limited to dictionary entries; LEMMING can analyze OOV words using annotated training data.Source: §4 · p. 3
5 Experiments
Experiments compare LEMMING’s pipeline and joint models with baselines across six languages. The best models improve on baselines, while joint modeling generally improves lemma accuracy over the best pipeline systems.
5 Experiments
- Experiments cover English, German, Czech, Hungarian, Latin, and Spanish, with training limited to the first 100,000 tokens per language.The data uses shared-task splits for German, Hungarian, Spanish, and Czech, SANCL for English, and an 8/1/1 Latin split.Source: §5 · pp. 3–4
- Lemma accuracy for the joint model over the best pipeline systems improves by .1 (Spanish), >.3 (German, Hungarian), and ≥.96 (Czech, Latin).Table 2 reports test results for LEMMING-J and pipeline systems; the prose notes an English tie in lemma accuracy and small drops in tag and joint accuracy.Source: Table 2 · p. 4, §5 · p. 5
- The baseline comparison includes MORFETTE, SIMPLE, and JCK; models are trained on gold attributes and tested on MORFETTE-predicted attributes for the initial experiments.Those initial experiments use POS and ignore additional morphological attributes.Source: §5 · p. 4
- Adding alignment and lemma features consistently beats all baselines and edit-tree; adding dictionary features then yields the best model, significantly better than the best baselines in every language.The reported improvements over baselines exceed 1 for Czech and Latin and are at least .5 for German and Hungarian.Source: §5 · p. 5
- Adding predicted morphological attributes reduces lemmatization performance in every language except Spanish, while preliminary experiments indicate correct attributes substantially improve ambiguous cases.German number can distinguish singular Raps (“canola”) from plural-looking Rap (“rap”).Source: §5 · p. 5
- Lemma accuracy improvements for the best models over the best baseline are >1 (German, Spanish, Hungarian) and >2 (Czech, Latin), with larger gains on unknown forms.Unknown-form improvements are >1 (English), >5 (German, Spanish, Hungarian), and >12 (Czech, Latin).Source: §5 · p. 5
6 Conclusion
LEMMING combines modular lemmatization with joint modeling of lemmata and morphological tags, without morphological dictionaries or analyzers. The paper reports state-of-the-art token-based lemmatization on six languages and benefits from joint modeling for both tasks.
6 Conclusion
- LEMMING supports arbitrary global lemma features and joint modeling of lemmata and morphological tags.It is trainable on corpora annotated with gold-standard tags and lemmata, without morphological dictionaries or analyzers.Source: §6 · p. 5
- LEMMING sets the new state of the art in token-based lemmatization across six languages.Source: §6 · p. 5
- Jointly modeling lemmatization and tagging benefits both tasks.Source: §6 · p. 5
A Tree Extraction
LEMMING extracts edit trees from form–lemma pairs by finding a longest common substring and recursively representing the surrounding transformations. The tree stores prefix and suffix lengths rather than the shared substring itself, allowing it to apply to related forms.
A Tree Extraction
- Edit trees store prefix and suffix lengths rather than the longest common substring, allowing the tree for umgeschaut to transform umgebaut into umbauen.Source: A Tree Extraction · p. 8, Figure 3 · p. 8
- The extraction process finds the longest common substring and recursively models the paired prefixes and suffixes, using a substitution node when no common substring exists.Source: A Tree Extraction · p. 8
- For umgeschaut→umschauen, the tree recursively transforms umge→um and t→en around the shared stem schau.Source: A Tree Extraction · p. 8, Figure 3 · p. 8
B Tree Application
Tree application recursively processes the form’s prefix and suffix, then combines the resulting strings with the unchanged middle. An application can fail when the required prefix or suffix transformation does not fit or match.
B Tree Application
- The application procedure creates prefix and suffix substrings, checks whether their lengths fit, and combines the transformed parts with the middle substring.Source: B Tree Application · p. 8
- A prefix or suffix subtree that cannot apply returns ⊥; applying this tree to einbauen fails because the expected ge does not match the form’s prefix.Source: B Tree Application · pp. 8–9
- Applying the edit tree to angebaut transforms ange to an and t to en, yielding the correct lemma anbauen.Source: B Tree Application · pp. 8–9
C Development Results
Table A1 reports development accuracies for baseline and LEMMING configurations.
C Development Results
- Table A1 reports general-token and unknown-form accuracy for baselines and LEMMING feature-set and tagger configurations.Source: Table A1 · p. 9
D Test Results
The supplied results passages identify test-accuracy comparisons and dataset statistics, but do not provide accuracy values. They also identify a development-accuracy comparison using gold tags.
D Test Results
- Table A2 reports general-token and unknown-form test accuracy for baselines and LEMMING feature-set and tagger configurations.Source: Table A2 · p. 10
- The supplied language-code list is cs, de, es, hu, and la.Source: D Test Results · p. 11
- Table A3 compares LEMMING development accuracy with and without morphological attributes when gold tags are used.Source: Table A3 · p. 11
- The listed dataset statistics cover sentence, token, POS-tag, morphological-tag, unknown-form, and unknown-lemma counts or rates.Source: Table A4 · p. 11