Source-linked AI summary
Morphological Word Embeddings
Ryan Cotterell, Hinrich Schütze
TL;DR
Word embeddings typically treat words holistically, while linguistic similarity includes morphology and morphologically rich languages require word-internal structure. Morph-LBL extends the log-bilinear model to predict morphological tags alongside words using annotated data. Its embeddings outperform two baselines on both evaluation metrics, and the results indicate that contextual similarity can preserve morphological relationships without word-internal features.
Problem
Word embeddings typically treat words holistically despite linguistic similarity having multiple axes, and exploiting word-internal structure is necessary for processing morphologically rich languages.
Source: §1 · p. 1Method
Morph-LBL extends the log-bilinear model with a multi-task objective that predicts the next word and morphological tag using raw text and morphologically annotated data.
Source: §1 · p. 1Results
Morph-LBL outperformed the original LBL and WORD2VEC under both evaluation metrics, and robustly maintained morphological relationships using distributional signatures without word-internal features.
Source: §6.3 · p. 5, §7 · p. 5Takeaways & Limitations
The results indicate that large corpora can yield a large portion of morphology through contextual similarity, without word-internal features.
Source: §6.3 · p. 5The model uses contextual signatures without integrating orthographic features, which the authors say are also requisite for a strong model.
Source: §7 · p. 5
Abstract
from arXiv · showhide
Linguistic similarity is multi-faceted. For instance, two words may be similar with respect to semantics, syntax, or morphology inter alia. Continuous word-embeddings have been shown to capture most of these shades of similarity to some degree. This work considers guiding word-embeddings with morphologically annotated data, a form of semi-supervised learning, encouraging the vectors to encode a word's morphology, i.e., words close in the embedded space share morphological features. We extend the log-bilinear model to this end and show that indeed our learned embeddings achieve this, using German as a case study.
1 Introduction
Word embeddings capture multiple kinds of linguistic similarity but typically represent words holistically, a limitation for morphologically rich languages. This work guides embeddings toward morphological similarity by jointly training on raw text and morphologically tagged data.
1 Introduction
- Morphologically rich languages require word-internal structure to be exploited, whereas holistic embeddings can be adequate for languages with few morphemes per word.The paper distinguishes semantic, syntactic, and morphological similarity as separate axes.Source: §1 · p. 1
- The paper aims to guide embeddings so that words close in the embedded space are morphologically related, rather than conflating that similarity with semantic or syntactic similarity.The introduction illustrates the distinction with topical, syntactic, and morphological relations among “ice,” “cold,” “fire,” and “icy.”Source: §1 · p. 1
- Morph-LBL extends the log-bilinear model with a multi-task objective that predicts the next word and morphological tag from raw text and annotated data.The authors report that the resulting embeddings better preserve morphological relationships in a German case study.Source: §1 · p. 1
2 Related Work
Prior work incorporates morphology into language models through factors, suffix classes, segmentation, character n-grams, and morpheme embeddings. The paper also situates its task in computational morphology, where rich inflectional systems require fine-grained tags and distributional context can provide morphological cues.
2.1 Morphology in Language Modeling
- Earlier language-modeling approaches incorporate morphology through factored tiers, suffix-based classes, morpheme segmentation, character n-grams, or summed morpheme embeddings.The surveyed approaches include Müller and Schütze (2011), Luong et al. (2013), Qiu et al. (2014), dos Santos and Zadrozny (2014), and Botha and Blunsom (2014).Source: §2.1 · p. 2
2.2 Computational Morphology
- Morphologically rich languages such as German require tags for attributes including case, gender, and number, producing larger tag sets than English.The cited tag-set example gives 137 tags for English and 970 for Czech (Hajič, 2000).Source: §2.2 · p. 2
- Contextual distributional similarity is also a cue to morphology, since words in related syntactic contexts can share properties such as genitive case.The authors describe isolating components of contextual signatures that predict morphology; they also note that word-internal information such as the suffix “ed” can signal a tag.Source: §2.2 · p. 2
3 Log-Bilinear Model
The log-bilinear model learns features along with weights rather than relying on hand-crafted features. In language modeling, it uses a word’s history to define an energy function for the model.
3 Log-Bilinear Model
- The log-bilinear model learns its features together with their weights instead of using hand-crafted features.The paper presents it as a generalization of the loglinear model.Source: §3 · p. 2
- In the language-modeling setting, the model represents a word w and its history h through an energy function sθ.The parameters include context-specific weights, context and target word embeddings, and a bias term.Source: §3 · pp. 2–3
4 Morph-LBL
Morph-LBL jointly predicts the next word and its morphological tag, using a hand-crafted feature vector for the tag. With partially annotated data, it marginalizes tags for unannotated examples while still using raw text to steer the embeddings.
4 Morph-LBL
- Morph-LBL jointly predicts the next word and its morphological tag given a history, adding a tag-feature vector and weight matrix to the model.The tag vector represents the morphological tag, and the matrix supplies additional weights.Source: §4 · p. 3
- A morphological tag is represented by binary features for its sub-tag units, with only units present in that tag activated.For example, ADJ.NOM.SG.FEM has four non-zero feature components.Source: §4 · p. 3
4.1 Semi-Supervised Learning
- Partially annotated training data lets Morph-LBL use raw text while marginalizing tags for examples without annotation.The authors state that only a partial set of labeled data is needed to steer the embeddings toward morphological properties.Source: §4.1 · p. 3
- The t-SNE projection shows clusters by morphological tag and coarse-grained part of speech, alongside cross-POS similarity for the genitive adjective sozialen and noun Friedens.The caption relates the latter pair to the frequent German phrase “sozialen Friedens” (“social peace”).Source: Figure 1 · p. 3
5 Evaluation
The evaluation asks whether embedding-space neighbors are morphologically related and introduces MORPHOSIM to quantify morphological similarity among them.
5 Evaluation
- The evaluation tests whether words that are similar in embedding space are morphologically related.Qualitative evaluation is also reported as indicating this relationship.Source: §5 · p. 3
5.1 MorphoSim
- MORPHOSIM scores morphological-tag similarity among embedding neighbors using Hamming distance, as a soft version of k-NN.The neighbors can be selected as the k nearest words by cosine distance.Source: §5.1 · pp. 3–4
6 Experiments and Results
The experiments compare Morph-LBL with unsupervised LBL and WORD2VEC on German, testing whether its embeddings encode morphology. Morph-LBL generally performs better under both evaluation metrics, including for relationships inferred from distributional context.
6 Experiments and Results
- The German TIGER case study compares Morph-LBL with fully unsupervised LBL and WORD2VEC; Morph-LBL receives annotations for the first 100k of 200k training words.All models use the first 200k words of TIGER; the embeddings have 200 dimensions.Source: §6 · p. 4
6.1 Experiment 1: Morphological Content
- Morph-LBL embeddings store the correct morphological analysis at a much higher rate than LBL and WORD2VEC, including when the tag was unseen during training.WORD2VEC trained on about 45 million tokens predicted the correct tag with about 22% accuracy on TIGER types.Source: §6.1 · pp. 4–5
6.2 Experiment 2: MORPHODIST
- Morph-LBL generally has better MORPHOSIM values than the baselines on words whose tags were not observed during training, with little change across k values.The figure reports k ∈ {5, 10, 25, 50}; lower MORPHOSIM values indicate morphologically closer neighbors.Source: §6.2 · p. 5, Figure 2 · p. 5
6.3 Discussion
- Morph-LBL outperforms LBL and WORD2VEC under both evaluation metrics, and robustly maintains morphological relationships using distributional signatures alone.The authors interpret this as showing that large corpora contain contextual similarity from which a large portion of morphology can be extracted.Source: §6.3 · p. 5
- The authors attribute Morph-LBL’s advantage to its annotated training data, while LBL’s local history is more suited to syntactic embeddings than WORD2VEC’s often non-local skip-grams.The comparison is described as unsurprising because Morph-LBL receives morphological annotations during training.Source: §6.3 · p. 5
7 Conclusion and Future Work
Morph-LBL combines morphologically annotated data with raw text to preserve morphological relationships and outperformed two baselines trained on the same corpus. The authors identify character-level orthographic features as future work and are interested in applying the embeddings to morphological tagging.
7 Conclusion and Future Work
- Morph-LBL preserves morphological relationships by combining morphologically annotated data with raw text, and it handily outperformed two baselines trained on the same corpus.The authors describe it as a model for semi-supervised induction of morphologically guided embeddings.Source: §7 · p. 5
- Character-level features remain future work, although orthographic features are described as requisite for a strong model alongside contextual signatures.The authors note that loving and eating share the likely context after is/are as well as the orthographic ending ing.Source: §7 · p. 5
- The authors are interested in applying the embeddings to morphological tagging and suggest they could improve downstream tagging by leveraging raw text.They contrast this potential with state-of-the-art morphological taggers, without reporting downstream tagging results here.Source: §7 · p. 5