Source-linked AI summary
Hearst Patterns Revisited: Automatic Hypernym Detection from Large Text Corpora
Stephen Roller, Douwe Kiela, Maximilian Nickel
TL;DR
Hypernym detection methods include pattern-based approaches, which can be sparse, and distributional approaches. This paper compares simple pattern-based models with DIH methods across hypernymy tasks and finds that pattern-based methods consistently outperform them, with low-rank embeddings improving coverage.
Problem
Hearst patterns can be sparse because terms must co-occur in exactly the right configuration, while distributional alternatives use representations based on corpus distributions.
Source: §1 · p. 1, §2.1 · p. 2Method
The paper evaluates Hearst-pattern models and their low-rank embeddings against DIH-based methods across hypernymy tasks.
Source: §1 · p. 1, §2.1 · p. 2Results
Pattern-based methods substantially outperform DIH-based methods on challenging benchmarks; on BLESS detection, average precision is 0.76 versus 0.19.
Source: §3.3 · p. 4, §4 · p. 5Takeaways & Limitations
Hearst patterns provide important contextual constraints for hypernymy detection that DIH models do not yet capture.
Source: §4 · p. 5
Abstract
from arXiv · showhide
Methods for unsupervised hypernym detection may broadly be categorized according to two paradigms: pattern-based and distributional methods. In this paper, we study the performance of both approaches on several hypernymy tasks and find that simple pattern-based methods consistently outperform distributional methods on common benchmark datasets. Our results show that pattern-based models provide important contextual constraints which are not yet captured in distributional methods.
1 Introduction
Pattern-based methods exploit lexico-syntactic patterns but suffer sparsity, while distributional methods represent words from corpus distributions. This paper compares these approaches and reports that simple pattern-based models outperform DIH methods, with low-rank embeddings mitigating sparsity.
1 Introduction
- Hearst patterns identify is-a relations from specific lexico-syntactic configurations, but exact-match requirements make them sparse.Words must co-occur in exactly the right configuration for a relation to be detected.Source: §1 · p. 1
- Distributional representations alleviate pattern sparsity by representing words from corpus distributions, while DIH-inspired measures compare narrow-term contexts with broader-term contexts.The Distributional Inclusion Hypothesis proposes that contexts of a narrow term should be a subset of those of a broader term.Source: §1 · p. 1
- Pattern-based methods use manually curated patterns for high-precision extractions, whereas DIH methods rely on unconstrained word co-occurrences in large corpora.Both approaches rely on co-occurrences, but differ in how they select contexts.Source: §1 · p. 1
- The paper evaluates simple pattern-based models on modern, large corpora and compares them with DIH-based methods.Source: §1 · p. 1
- Simple pattern-based methods consistently outperform specialized DIH methods on several difficult hypernymy tasks, and low-rank embeddings improve their performance by remedying sparsity.The results also identify important contextual constraints in Hearst patterns that distributional methods do not yet capture.Source: §1 · p. 1
2 Models
The paper compares simple pattern-based approaches with distributional methods for hypernym detection. Pattern-based models use Hearst-pattern extractions, frequency-adjusted scores, and low-rank embeddings; distributional models rely on word-context features and alternative generality measures.
2 Models
- The comparison uses relatively simple pattern-based approaches so their performance can be directly compared with DIH-based methods.Source: §2 · p. 2
2.1 Pattern-based Hypernym Detection
- Hearst-pattern counts or extraction probabilities directly predict hypernymy, while PPMI adjusts for word-frequency skew in those extractions.The text notes that raw extraction probabilities can favor more frequent hypernyms, such as country over republic for France.Source: §2.1 · p. 2
- Because PPMI cannot handle missing pairs, truncated-SVD embeddings support predictions for unseen pairs and give similar representations to words sharing Hearst patterns.The factorization uses the PPMI matrix; the paper also considers factorizing occurrence probabilities, related to Cederberg and Widdows (2003).Source: §2.1 · p. 2
2.2 Distributional Hypernym Detection
- Distributional methods include WeedsPrec and invCL, which measure feature inclusion, with invCL also accounting for contexts used by the broader term but not the narrower term.Source: §2.2 · pp. 2–3
- The evaluation also includes SLQS-cos, which combines relative generality with cosine similarity, and cosine similarity as a baseline.SLQS uses the median entropy of a term’s top N contexts, with N selected during hyperparameter selection.Source: §2.2 · p. 3
3 Evaluation
The evaluation compares relatively simple pattern-based and DIH-based models across detection, direction prediction, and graded entailment. Pattern-based methods outperform distributional models in nearly all cases, with important coverage limits on some datasets.
3.1 Tasks
- Detection uses BLESS, LEDS, EVAL, SHWARTZ, and WBLESS; direction prediction uses BLESS, WBLESS, and BIBLESS; graded entailment uses HYPERLEX’s 2,163 noun pairs.Detection is evaluated with Average Precision, direction prediction with accuracy, and graded entailment with Spearman’s rank correlation.Source: §3.1 · pp. 3–4
3.2 Experimental Setup
- Hearst patterns extracted from Gigaword and Wikipedia yielded roughly 4.5M matched pairs; the DIH baseline uses Shwartz et al. (2017)’s sparse dependency-context space.SVD rank was selected on validation data; the distributional space contains 218K words over 732K context dimensions.Source: §3.2 · p. 4
3.3 Results
- Pattern-based approaches substantially outperform all three distributional models in nearly all cases; BLESS average precision is 0.76 vs 0.19, and WBLESS scores are 0.96 vs. 0.69.At least one sparse model also outperforms all distributional measures in 7 of 9 datasets.Source: §3.3 · pp. 4–5, Table 2 · p. 5
- Sparse models face SHWARTZ’s long tail and EVAL’s OOV verbs and adjectives, yet at least one sparse model beats all distributional measures in 7 of 9 datasets.The patterns do not capture the verbs and adjectives that contribute to EVAL’s out-of-vocabulary penalty.Source: §3.3 · p. 5
4 Conclusion
The paper compares Hearst pattern-based and DIH-based methods for hypernym detection. It finds that patterns outperform DIH methods on several challenging benchmarks, while embeddings alleviate sparsity and patterns capture contexts not yet modeled by DIH.
4 Conclusion
- Pattern-based methods substantially outperform DIH-based methods on several challenging hypernymy benchmarks.The conclusion describes the comparison as one of relative performance between Hearst pattern-based and DIH-based methods.Source: §4 · p. 5
- Embedding methods alleviate sparsity concerns in pattern-based approaches and substantially improve coverage.Source: §4 · p. 5
- Hearst patterns provide important contexts for detecting hypernymy relations that DIH models do not yet capture.Source: §4 · p. 5