Source-linked AI summary
Hierarchical Embeddings for Hypernymy Detection and Directionality
Kim Anh Nguyen, Maximilian Köper, Sabine Schulte im Walde, Ngoc Thang Vu
TL;DR
Hypernymy methods have struggled to distinguish the relation from other semantic relations and to learn its directionality rather than memorize prototypical hypernyms. HyperVec learns embeddings that strengthen hypernym-pair similarity while encoding a hierarchy between hypernyms and hyponyms. On benchmark datasets, it outperforms prior measures and embedding models, with evidence of generalization from small training sets.
Problem
Standard distributional vectors can be insufficient to distinguish hypernymy from paradigmatic relations, while prior supervised methods may memorize prototypical hypernyms rather than learn the relation.
Source: §3.1.2 · p. 3, §6 · p. 9Method
HyperVec learns hierarchical embeddings that bring hypernym pairs closer while encoding a distributional hierarchy used for hypernymy detection and directionality.
Source: §1 · p. 2, §3.1.2 · p. 3Results
Takeaways & Limitations
The experiments support learning the hypernymy relation rather than merely memorizing prototypical hypernyms.
Source: §6 · p. 9The unsupervised measure applies when the resource is used only to learn hierarchical embeddings.
Source: §3.2 · p. 5
Abstract
from arXiv · showhide
We present a novel neural model HyperVec to learn hierarchical embeddings for hypernymy detection and directionality. While previous embeddings have shown limitations on prototypical hypernyms, HyperVec represents an unsupervised measure where embeddings are learned in a specific order and capture the hypernym$-$hyponym distributional hierarchy. Moreover, our model is able to generalize over unseen hypernymy pairs, when using only small sets of training data, and by mapping to other languages. Results on benchmark datasets show that HyperVec outperforms both state$-$of$-$the$-$art unsupervised measures and embedding models on hypernymy detection and directionality, and on predicting graded lexical entailment.
1 Introduction
Hypernymy is an asymmetric semantic relation with applications in several NLP tasks, but existing methods have difficulty reliably distinguishing it from other relations and may memorize prototypical hypernyms. HyperVec learns hierarchical embeddings for detection and directionality, and benchmark results show improvements over prior measures and embedding models.
1 Introduction
- Hypernymy is asymmetric: a hyponym implies its hypernym, but not vice versa, and the relation supports taxonomy creation, textual entailment recognition, and text generation.Source: §1 · p. 1
- Prior work found no single unsupervised measure consistently distinguishes hypernymy from other semantic relations, while supervised methods may memorize prototypical hypernyms (Shwartz et al., 2017).Source: §1 · p. 1
- HyperVec learns hierarchical embeddings to distinguish hypernymy from other relations and identify which term is the hypernym.It strengthens the similarity of hypernym pairs while encoding a distributional hierarchy between hypernyms and hyponyms.Source: §1 · p. 2
- HyperVec assumes that shared context words distinctive of a hyponym lie closer to that hyponym than to its hypernym.For example, flap is expected to be closer to bird than to animal.Source: §1 · p. 2
- Benchmark results show HyperVec outperforming state-of-the-art measures and previous embedding models.The model is evaluated on supervised and unsupervised detection and directionality, graded lexical entailment, and generalization by mapping to German and Italian.Source: §1 · p. 2
2 Related Work
Prior work approaches hypernymy with unsupervised directional measures, supervised classifiers, and embeddings specialized for hypernymy. These methods use distributional features or learned representations, with limitations in capturing asymmetry and generalizing beyond training pairs.
Unsupervised hypernymy measures
- Unsupervised measures use distributional inclusion or informativeness hypotheses to score hypernymy, commonly ranking pairs with Average Precision (Weeds and Weir, 2003; Weeds et al., 2004; Clarke, 2009; Kotlerman et al., 2010; Lenci and Benotto, 2012; Santus et al., 2014).Visual-property methods have also been used to estimate concept generality.Source: §2 · p. 2
Supervised hypernymy methods
- Supervised methods combine word vectors, such as by concatenation or difference, and classify hypernymy with SVM or logistic regression.Because standard embeddings are trained for similar, symmetric vectors, whether these methods learn hypernymy’s asymmetry is unclear.Source: §2 · p. 2
Hypernymy-specific embeddings
- Yu et al. (2015) and Tuan et al. (2016) learn hypernymy-specific embeddings, but Tuan et al.’s method still does not determine pair directionality.Yu et al. use pre-extracted taxonomic pairs without contexts; Tuan et al. add sentence context, while Vendrov et al. (2016) encode order in learned representations.Source: §2 · pp. 2–3
3 Hierarchical Embeddings
HyperVec learns hierarchical embeddings with Skip-gram and context-based objectives, then applies an unsupervised measure for hypernymy detection and directionality. The design addresses the difficulty of separating hypernymy from other paradigmatic relations, including meronymy.
3.1 Learning Hierarchical Embeddings
- The model learns from WordNet noun and verb hypernym–hyponym pairs, including direct and indirect relations, while excluding pairs present in evaluation datasets.The embeddings combine Skip-gram and the hierarchical objectives.Source: §3.1 · p. 3, §3.1.2 · p. 4
3.1.1 Skip-gram Model
- Skip-gram with negative sampling trains on observed word–context pairs, using their counts and sampled negative contexts.The corpus words and contexts are drawn from their respective vocabularies.Source: §3.1.1 · p. 3
3.1.2 Hierarchical Hypernymy Model
- HyperVec orders pair similarities so hypernym pairs score above other relations and represents hierarchy through larger hypernym than hyponym vector norms.For example, animal–frog is assigned a higher cosine score than the co-hyponymy pair eagle–frog.Source: §3.1.2 · p. 3
- Two context-based objectives pull hypernym and hyponym vectors together around distinctive shared contexts; the margin helps learn hierarchy and avoid minimizing synonymy or antonymy.The objectives update a hypernym and its hyponyms simultaneously.Source: §3.1.2 · p. 4
3.2 Unsupervised Hypernymy Measure
- HyperScore combines cosine similarity and vector magnitudes to detect hypernymy and infer direction, assuming the hypernym vector has a larger magnitude than the hyponym vector.The measure is unsupervised when the resource is used only to learn the hierarchical embeddings.Source: §3.2 · pp. 4–5
4 Experiments
The experiments evaluate HyperVec on unsupervised and supervised hypernymy detection, directionality, and graded lexical entailment. Across the reported benchmark tasks, HyperScore and HyperVec outperform the compared measures or models, with results varying by task and baseline.
4 Experiments
- The paper evaluates HyperVec on unsupervised and supervised hypernymy detection, directionality, and graded lexical entailment.It also assesses generalization through small hypernymy seed sets and mapping to German and Italian in the broader experimental program described here.Source: §4 · p. 5
4.1 Experimental Settings
- The experimental settings train HyperVec and default SGNS on ENCOW14A, an approximately 14.5-billion-token corpus, using 100-dimensional embeddings and a window size of 5.The settings also specify 15 negative samples, learning rate 0.025, and threshold θ = 0.05.Source: §4.1 · p. 5
4.2 Unsupervised Hypernymy Detection and Directionality
- The experiments compare ranking retrieval, which scores hypernymy above other relations, with classification, which tests detection and directionality.Ranking retrieval uses Average Precision; classification evaluates both binary and three-way setups.Source: §4.2 · p. 5, §4.2.1 · p. 5
4.2.1 Ranking Retrieval
- On BLESS, HyperScore improves AP by 40% over the best baseline for ranking hypernymy against all other relations; it improves on single-relation comparisons except BLESS event relations.The authors suggest the event-relation exception may reflect adjective–noun part-of-speech differences that prevent HyperVec from establishing a hierarchy.Source: Table 2 · p. 5, §4.2.1 · p. 6
4.2.2 Classification
- HyperVec outperforms the other methods on all three unsupervised classification tasks, while unmodified SGNS cannot solve any of them.The tasks include directionality, binary hypernymy detection, and three-way classification of hypernymy, reversed hypernymy, and other relations.Source: §4.2.2 · pp. 6–7, Table 3 · p. 6
4.3 Supervised Hypernymy Detection
- On supervised detection, HyperVec slightly outperforms Tuan et al. on BLESS, matches their ENTAILMENT result, and significantly improves over Yu et al.The supplied results describe accuracy on BLESS and ENTAILMENT.Source: Table 4 · p. 7, §4.3 · p. 7
4.4 Graded Lexical Entailment
- HyperScore significantly outperforms state-of-the-art measures and embedding models on graded lexical entailment, exceeding PARAGRAM by .22 and FR by .27.The authors attribute this result to the hierarchy distinguishing hyponym–hypernym pairs from reversed pairs; duck–animal scores 3.02 versus 0.30 for animal–duck.Source: Table 5 · p. 8, §4.4 · p. 8
5 Generalizing Hypernymy
HyperVec is tested for generalization from a 200-concept seed set and through mapping German and Italian representations into English HyperVec space. The results support seed-based generalization and higher hypernymy scores from mapped representations in both languages.
5 Generalizing Hypernymy
- 0.39, 0.448, and 0.585 AP on EVALution, LenciBenotto, and Weeds, respectively, with HyperScore outperforming baselines across all three datasets.For graded lexical entailment, HyperScore reaches ρ = 0.30, outperforming all models except PARAGRAM at ρ = 0.32.Source: §5 · p. 8
- The seed experiment trains HyperVec from 200 BLESS concepts, using their WordNet hyponyms, and reports generalization across datasets.The concepts were selected as distinct, unambiguous nouns, balanced between living and non-living entities, and grouped into 17 broader classes.Source: §5 · p. 8
- A least-squares mapping trained on Europarl word alignments projects German and Italian source vectors into English HyperVec space without language-specific WordNet tuning.The alignments provide 25,547 German–English and 47,475 Italian–English word pairs; the learned mapping can project every source-vocabulary word.Source: §5 · pp. 8–9
- In German and Italian, mapped representations yield higher hypernymy scores than default SGNS, which does not score hypernymy higher except for Italian Hyp/Ant.Table 6 reports AP results across languages and compares SGNS with projected representations.Source: §5 · p. 9, Table 6 · p. 9
6 Conclusion
HyperVec learns hierarchical embeddings for hypernymy and strengthens hypernymy similarity while capturing distributional hierarchy. With HyperScore, the experiments report improvements over state-of-the-art measures and generalization beyond prototypical hypernyms.
6 Conclusion
- HyperVec learns hierarchical embeddings that strengthen hypernymy similarity and capture the distributional hierarchy of hypernymy.The conclusion presents HyperVec as a neural model for hypernymy.Source: §6 · p. 9
- HyperScore experiments show significant improvements against state-of-the-art measures.Source: §6 · p. 9
- The experiments indicate that HyperVec can generalize hypernymy and learn the relation rather than memorize prototypical hypernyms.Source: §6 · p. 9