Source-linked AI summary
The Interplay of Semantics and Morphology in Word Embeddings
Oded Avraham, Yoav Goldberg
TL;DR
The paper asks how word-representation components affect semantic and morphological similarity, and compares configurations of morphology-driven models. It finds a trade-off between the two measures, with lemma-based configurations favored for semantic similarity.
Problem
The paper asks how semantic and morphological similarity relate in morphology-driven representations, and whether different component combinations suit different performance goals.
Source: §1 · p. 1Method
The authors compare morphology-driven model configurations by varying representation components and evaluating semantic and morphological similarity across common and rare words.
Source: §1 · p. 1Results
Takeaways & Limitations
Users should include morphological information when their task requires morphological similarity that cannot be handled by other means.
Source: §4 · p. 5
Abstract
from arXiv · showhide
We explore the ability of word embeddings to capture both semantic and morphological similarity, as affected by the different types of linguistic properties (surface form, lemma, morphological tag) used to compose the representation of each word. We train several models, where each uses a different subset of these properties to compose its representations. By evaluating the models on semantic and morphological measures, we reveal some useful insights on the relationship between semantics and morphology.
1 Introduction
The paper examines how representation components shape semantic and morphological similarity, and reports a trade-off between the two. It compares model configurations and evaluates both aspects, focusing on inflectional morphology in Modern Hebrew.
1 Introduction
- The study asks whether representation components should vary with the goal, including semantic or morphological performance and common or rare words.It specifically asks whether excluding morphology could improve semantic performance and what surface form contributes.Source: §1 · p. 1
- The authors compare component-controlled configurations, evaluating semantic and morphological performance separately on common and rare words, with a focus on inflection rather than derivation.Their rationale is that inflections preserve lexical meaning while changing grammatical categories; derivations can substantially change meaning.Source: §1 · p. 2
- They use Modern Hebrew datasets and a morphological analyzer to represent words through lemmas and morphological tags, distinguishing inflectional from derivational processes.Because Hebrew morphology is non-concatenative, their approach does not split words into base and affixes; they also compare against a character n-gram model.Source: §1 · p. 2
- Models that excel at semantic similarity perform poorly on morphological similarity, revealing a trade-off between the two measures.The paper presents configuration strengths and weaknesses to help users choose according to their needs.Source: §1 · p. 2
- Prior morphology-driven work mostly used base-plus-affix or surface-plus-base-plus-affix configurations, while compositional models were reported to improve semantic performance, especially for rare words.The paper also frames base-form components as mostly semantic and affixes as mostly morphological.Source: §1 · pp. 1–2
- The paper presents its work as the first evaluation of both semantic and morphological performance, using Modern Hebrew because a reliable semantic-similarity dataset is available.The authors believe their conclusions hold more generally.Source: §1 · p. 2
2 Models
The model generalizes skip-gram and fastText by composing word representations from explicit linguistic properties. Varying the included properties gives finer control over which information contributes to each representation.
2 Models
- The model extends fastText, which extends skip-gram; skip-gram scores focus-word and context-word pairs and maximizes an objective using context words and sampled negatives.The loss is log-sigmoid, and the score is defined using focus-word and context-word embeddings.Source: §2 · p. 2
- Each word representation is the sum of vectors for selected linguistic properties: surface form, lemma, and morphological tag.The paper trains models with different subsets of these properties.Source: §2 · pp. 2–3
- The model replaces fastText’s character n-gram set with explicit linguistic properties, enabling finer control over representation content.fastText itself replaces a word vector with vectors for the character n-grams appearing in the word, which approximate morphemes.Source: §2 · pp. 2–3
3 Experiments and Results
Across the tested configurations, semantic and morphological performance trade off: lemma helps semantics, tags help morphology, and using both gives an intermediate compromise.
Experiments
- The models use W, L, WL, WM, and WLM representations alongside fastText n-grams, and are evaluated on semantic and morphological datasets.Training uses Hebrew Wikipedia (about 4M sentences), a window of 2 words on each side, and 200 dimensions; a disambiguator assigns tags and an inflection dictionary supplies lemmas.Source: §3 · p. 3
Semantic Evaluation Measure
- Semantic evaluation uses AG confidence-weighted triple ranking, with SemanticSim (1,819 triples) and SemanticSimRare (163 positive-versus-random triples for targets occurring fewer than 100 times).AG target groups include similar positives, related-but-dissimilar distractors, and unrelated randoms; the rare-word dataset uses only positive-versus-random comparisons.Source: §3 · p. 3
Morphological Evaluation Measure
- MorphoSim scores average morphological compatibility among a word’s top 10 neighbors, derived from MorphoDist’s count of incompatible tag components.Scores are averaged over 100 randomly chosen words, or over 35 SemanticSimRare target words for the rare-word benchmark.Source: §3 · pp. 3–4
Quantitative Results
- Semantic and morphological performance trade off: lemma improves semantics but hurts morphology, while tags do the reverse; LM, WLM, and n-grams are intermediate.The common practice of using both components is a compromise: those models are neither best nor worst on any measure.Source: §3 · p. 4
- For rare words, WL is 16.3% better on SSR and 11.9% worse on MSR than W; WM is 21.8% worse on SSR and 75.7% better on MSR.For common words, WL is 1.7% better on SS and 3.8% worse on MS than W; WM is 2.8% worse on SS and 44.9% better on MS.Source: §3 · p. 4
- For semantic similarity, L clearly outperforms the other models on rare words, while common words show a small drop versus WL and WLM, which also include surface form.The authors attribute the common-word difference to semantic information in morphological templates, such as gender-linked associations.Source: §3 · p. 4
4 Conclusions
The conclusion frames representation design as a trade-off: lemma-based configurations are better for semantic similarity, while morphological components are warranted only when a task needs them.
4 Conclusions
- Users should consider the trade-off between representation components when choosing a model configuration.The conclusion presents extrinsic evaluation across downstream applications as future work for identifying which tasks benefit from morphology versus a pure semantic model.Source: §4 · pp. 4–5
- Lemma alone or with the surface form performs better for semantic similarity than configurations combining semantic and morphological components.Table 2 reports results for semantic and morphological similarity, including rare-word measures.Source: §4 · p. 5, Table 2 · p. 5
- Excluding the morphological component lowers morphological similarity, but it is needed only when a task requires it and other means cannot handle it.The authors suggest evaluating models in downstream applications to identify which tasks benefit from morphological information.Source: §4 · p. 5