Source-linked AI summary
Improving Hypernymy Detection with an Integrated Path-based and Distributional Method
Vered Shwartz, Yoav Goldberg, Ido Dagan
TL;DR
Hypernymy detection requires identifying a specific lexical-semantic relation using complementary distributional and path-based evidence. The paper presents HypeNET, which encodes dependency paths with an LSTM and integrates distributional signals. The path-based model matches state-of-the-art distributional methods, while the integrated model improves by up to 14 F1 points.
Problem
Hypernymy detection is important for NLP, but path-based methods suffer from sparse features and limited recall while distributional methods can be less precise for specific relations.
Method
HypeNET encodes dependency paths with an LSTM and integrates the resulting path-based representation with distributional signals.
Results
The path-based model matches state-of-the-art distributional methods, and the integrated model yields an improvement of up to 14 F1 points over each individual model.
Takeaways & Limitations
Neural path representations improve generalization by capturing semantically similar paths, while combining path-based and distributional signals provides complementary information.
Takeaways & Limitations
Under a lexical split, learning only general paths can cause over-generalization, and the authors question whether the both-terms-unseen setting is common enough to justify this setup.
Abstract
from arXiv · showhide
Detecting hypernymy relations is a key task in NLP, which is addressed in the literature using two complementary approaches. Distributional methods, whose supervised variants are the current best performers, and path-based methods, which received less research attention. We suggest an improved path-based algorithm, in which the dependency paths are encoded using a recurrent neural network, that achieves results comparable to distributional methods. We then extend the approach to integrate both path-based and distributional signals, significantly improving upon the state-of-the-art on this task.
1 Introduction
Hypernymy detection uses distributional and path-based evidence, but path-based methods face sparse features and limited recall. HypeNET encodes dependency paths with an LSTM, matches state-of-the-art distributional methods, and gains up to 14 F1 points by integrating both signals.
- Hypernymy detection matters for NLP because identifying that y is a hypernym of x supports applications such as question answering.
- Distributional methods use separate term contexts, whereas path-based methods use lexico-syntactic paths connecting joint occurrences of x and y.
- Individual dependency paths create a huge, sparse feature space because lexically different paths can express the same hypernymy relation.
- Path-based methods generally underperform distributional methods because requiring joint occurrences limits recall, while distributional methods can be less precise for specific relations.
- HypeNET encodes dependency paths with an LSTM, improving path-based performance to a level comparable with state-of-the-art distributional methods.The neural representation captures semantic as well as syntactic similarities between paths.
- Up to 14 F1 points separate the combined model from each individual model, supporting complementary information from distributional and path-based signals.
2 Background
Hypernymy detection combines distributional representations of separate term contexts with path-based representations of connecting dependency paths. Prior work addresses path sparsity through path generalization, while recurrent networks motivate HypeNET's path encoding.
- Distributional Methods: Distributional methods infer hypernymy from the separate corpus contexts of the two terms, using unsupervised measures or supervised classifiers over pair representations.
- Path-based Methods: Path-based methods infer hypernymy from lexico-syntactic paths connecting joint term occurrences, including patterns such as Y such as X.
- Path-based Methods: Path-based feature spaces are sparse because similar paths vary lexically; PATTY generalizes path words using POS tags, ontological types, or wildcards.
- Relation Classification: Relation classification uses dependency paths to identify relations expressed in a sentence, whereas hypernymy detection targets generic lexical-semantic relations across many contexts.
- Neural Path Encoding: Recurrent neural networks, including LSTMs, process dependency paths edge-by-edge and have improved performance in relation classification.
- Relation Classification: Relation classification represents a pair with one dependency path, while hypernymy detection represents it with the multiset of all co-occurring corpus paths.
3 LSTM-based Hypernymy Detection
HypeNET represents dependency paths with an LSTM, pools the resulting path vectors for term-pair classification, and integrates word embeddings to combine path-based and distributional signals.
- 3.1 Path-based Network: HypeNET encodes each dependency path as an ordered sequence of edge vectors with an LSTM, rather than treating the entire path as one feature.Each edge combines lemma, part-of-speech, dependency label, and direction embeddings.
- 3.1 Path-based Network: The model averages path embeddings for each term-pair, weighting paths by corpus frequency, then classifies whether y is a hypernym of x.A single-layer network produces the binary classification decision.
- 3.2 Integrated Network: The path-based network is extended by concatenating x and y word embeddings with the averaged path vector.This gives each pair both distributional features and path-based features.
- Implementation Details: Higher-dimensional embeddings do not improve performance and increase training runtime.The comparison concerns the embedding dimensionality used during training.
4 Dataset
The dataset uses distant supervision from multiple knowledge resources, related semantic pairs as negatives, and both random and vocabulary-disjoint evaluation splits.
- Dataset Construction: The dataset extracts hypernymy relations from WordNet, DBPedia, Wikidata, and additional knowledge resources using distant supervision.The approach addresses the small size of existing hypernymy datasets.
- Dataset Construction: Related term-pairs, including hyponymy pairs, serve as negative instances, with a 1:4 positive-to-negative ratio.Using related negatives tests discrimination between hypernymy and other semantic relatedness.
- Dataset Splits: The primary dataset uses a random split of 70% training, 25% testing, and 5% validation instances.These proportions define the standard random-split evaluation setting.
- Dataset Splits: A lexical split assigns distinct vocabularies to the train, test, and validation sets to investigate behavior on unseen terms.The authors contrast this with random splitting, where previously observed terms may provide useful prior information.
- Dataset Splits: The lexical split discards many pairs consisting of cross-set terms.This is an explicit scope limitation of the lexical-split construction.
5 Baselines
The baselines cover path-based and distributional approaches, with experiments reproducing prior methods and adapting SLQS to the paper’s dataset. The authors also evaluate generalized paths and supervised embedding-based representations.
- Evaluation setup: The comparison replicates path-based and distributional baselines under a common evaluation setup.All methods use the May 2015 Wikipedia dump, while baseline results are replicated because prior studies used different datasets and corpora.
- Path-based baselines: Snow represents term pairs with dependency paths, retaining the 100,000 most informative paths for logistic regression.The implementation extracts shortest paths of four edges or fewer, including paths with satellite edges.
- Path-based baselines: Generalization replaces path edges with part-of-speech tags and wild cards, producing up to 2,093,220 features.The baseline includes the original paths and applies feature selection after generating the powerset of generalizations.
- Distributional baselines: SLQS is an entropy-based unsupervised measure for hypernymy detection, adapted here because vanilla settings performed poorly on a dataset containing rare terms.The adaptation tunes the positive-class threshold and context limit, omits cosine multiplication, and lemmatizes terms.
- Distributional baselines: The authors report that SLQS is designed primarily to classify specificity levels among related terms rather than hypernymy specifically.This characterization is presented as an interpretation of the method’s performance on the paper’s dataset.
- Distributional baselines: Supervised distributional baselines represent term pairs using concatenation, difference, or dot-product embeddings with several classifiers.The compared classifiers include logistic regression, SVM, and an RBF-kernel SVM.
6 Results
HypeNET’s path-based model improves substantially over prior path-based baselines and reaches state-of-the-art performance, while integrating distributional features yields statistically significant further gains. The results also identify recall gains from neural path generalization and errors from over-generalization and meronym confusion.
- Path-based results: HypeNET Path-based significantly improves recall over both path-based baselines, with slightly lower precision.The authors attribute the recall boost to better path generalization.
- Distributional results: 38% of random-split and 48% of lexical-split false positives were holonym-meronym pairs in a sample of 50 pairs per split.The authors link the low precision of unsupervised SLQS to difficulty distinguishing hypernyms from meronyms.
- Integrated results: HypeNET Integrated and HypeNET Path-based, as well as the supervised distributional method, differ substantially on both splits with p-value 1% in paired t-tests.The integrated model’s gains support the complementarity of path-based and distributional signals.
- Split analysis: HypeNET’s performance also decreases on the lexical split, which the authors attribute to over-generalization rather than lexical memorization.The same reduction had previously been observed for supervised distributional methods.
7 Analysis
The analysis examines how HypeNET generalizes dependency paths and characterizes its false positives and false negatives. Errors often involve semantically similar relations, ambiguous annotations, noisy generalization, or sparse corpus evidence.
- Qualitative Analysis of Learned Paths: HypeNET makes finer-grained path generalizations than Snow’s method, avoiding both verbatim-path recall limits and overly broad generalized paths.Snow’s generalized paths can become non-indicative, whereas HypeNET provides a better midpoint.
- Qualitative Analysis of Learned Paths: On the random split, HypeNET learns specific paths such as X is Y published and X is Y produced, but on the lexical split it learns only the general X is Y path.The general X is Y path is noisy and can occur in ad-hoc contexts without expressing generic hypernymy.
- Error Analysis: More than 20% of false-positive errors come from confusing synonymy with hypernymy, which the analysis identifies as difficult to distinguish.The false-positive analysis groups semantic relations found in the resources used to construct the dataset.
- Error Analysis: 30% of false-positive term-pairs are reversed hypernym-hyponym pairs, often involving near-synonyms whose direction is unclear.The example fiction and story illustrates that resource annotations may encode a disputed level of generality.
- Error Analysis: False negatives commonly involve few corpus co-occurrences, infrequent terms, rare senses, or incorrect positive annotations.Table 7 defines the low-co-occurrence threshold as fewer than 25 occurrences, compared with 99.7 average co-occurrences for true positives.
- Error Analysis: Hypernymy-like relations were annotated as negative because the dataset retained only indisputable hypernymy relations from its resources.Examples include (Goethe, occupation, novelist) and (Homo, subdivisionRanks, species).
8 Conclusion
HypeNET improves path-based hypernymy detection through neural path representations and combines path-based and distributional signals for further gains. Its recall improvement comes from generalizing semantically similar paths, while the integrated model adds 14 F1 points.
- HypeNET uses LSTM-based path representations to outperform prior path-based methods and match previously superior distributional methods.
- The increase in recall results from generalizing semantically similar paths rather than making no generalizations or over-generalizing paths.
- 14 additional F1 points result from integrating distributional signals into the path-based network.
- The integrated results demonstrate that path-based and distributional approaches are complementary.
- The architecture appears applicable to multi-class classification of term-pairs across multiple semantic relations in future work.
Appendix A Best Hyper-parameters
Table 8 presents the best hyper-parameters for every model. The selected settings yield the highest F1 score on the validation set.
- Table 8 displays the chosen hyper-parameters of each method.
- The listed settings are selected to yield the highest F1 score on the validation set.
- The table is titled “The best hyper-parameters in every model.”