Source-linked AI summary
Inferring Concept Hierarchies from Text Corpora via Hyperbolic Embeddings
Matt Le, Stephen Roller, Laetitia Papaxanthos, Douwe Kiela, Maximilian Nickel
TL;DR
Inferring is-a relations from text is challenging because Hearst patterns miss relations and may produce noisy extractions. The paper combines Hearst patterns with hyperbolic embeddings to infer missing relations and improve hierarchy consistency. It reports better benchmark performance than non-hyperbolic baselines, alongside substantially lower embedding dimensionality.
Problem
Hearst patterns can miss true is-a relations, while concept hierarchies are useful for reasoning and structural insight across domains.
Source: §1 · p. 1Method
The method builds a graph from Hearst-pattern extractions and embeds it in hyperbolic space, using entailment cones to model transitivity.
Source: §1 · pp. 1–2Results
HYPECONES achieves a relative improvement typically over 40% over the best non-hyperbolic model and improves consistency through transitivity.
Source: §4 · p. 8Takeaways & Limitations
Hyperbolic geometry supports efficient, taxonomically consistent embeddings for predicting hypernymy relations from text.
Source: §1 · p. 2, §5 · p. 8Hearst-pattern matches are sparse, and broader low-precision patterns can introduce false positives, inconsistencies, and cycles.
Source: §3.1 · p. 4
Abstract
from arXiv · showhide
We consider the task of inferring is-a relationships from large text corpora. For this purpose, we propose a new method combining hyperbolic embeddings and Hearst patterns. This approach allows us to set appropriate constraints for inferring concept hierarchies from distributional contexts while also being able to predict missing is-a relationships and to correct wrong extractions. Moreover -- and in contrast with other methods -- the hierarchical nature of hyperbolic space allows us to learn highly efficient representations and to improve the taxonomic consistency of the inferred hierarchies. Experimentally, we show that our approach achieves state-of-the-art performance on several commonly-used benchmarks.
1 Introduction
The paper targets unsupervised inference of concept hierarchies from text by combining Hearst-pattern extractions with hyperbolic embeddings. The approach uses hyperbolic geometry to recover missing relations, improve taxonomic consistency, and represent hierarchies efficiently.
Motivation
- Concept hierarchies support high-level reasoning and provide structural insights into domain concept spaces.The introduction points to applications across biology, medicine, and artificial intelligence.Source: §1 · p. 1
Approach
- The method extracts potential is-a relations with Hearst patterns, builds a directed weighted graph, then embeds it to infer missing relations and remove wrong extractions.Hearst patterns can miss or incorrectly extract relations because terms must co-occur in the right pattern.Source: §1 · pp. 1–2
Consistency
- Hyperbolic embeddings of Hearst graphs enforce transitivity across the embedding space, improving taxonomic consistency.The constraint supports is-a inference when a relation follows from two existing relations.Source: §1 · p. 2
Efficiency
- Hyperbolic embeddings reduce dimensionality by over an order of magnitude while outperforming SVD-based methods.The paper identifies latent hierarchy and heavy-tailed degree distributions as properties of Hearst graphs that make low-dimensional hyperbolic representations useful.Source: §1 · p. 2
Interpretability
- Hyperbolic embeddings encode semantic similarity through distance and hierarchy through embedding norms, revealing term generality.The paper illustrates the resulting hierarchical structure with a two-dimensional embedding of the Hearst graph, while using higher dimensions for final embeddings.Source: §1 · p. 2
2 Related Work
Prior work addresses hypernym detection with pattern-based and distributional methods, while taxonomy induction builds richer graph structures. The paper situates its approach among these lines of work by combining Hearst-pattern evidence with embeddings that model hierarchy.
Hypernym detection
- Hearst-pattern methods suffer missing extractions, while distributional approaches alleviate sparsity but need specialized measures to distinguish lexical relations.Pattern methods may improve coverage through search engines or web-scale corpora, but with precision trade-offs.Source: §2 · p. 2
- Pattern-based methods can outperform DIH-based methods on challenging hypernymy benchmarks, while large-corpus pattern extraction and embeddings help address sparsity.The paper builds on this finding by replacing prior embeddings with ones having naturally hierarchical geometry.Source: §2 · p. 2
Taxonomy induction
- Hypernym detection alone does not produce rich taxonomic graphs, making taxonomy induction a parallel and complementary task.Prior work grows taxonomic graphs, predicts undiscovered edges, or prunes noisy extractions toward a strict hierarchy.Source: §2 · p. 3
Embeddings
- Some graph-embedding approaches recover hierarchical structure, but generally require extensive supervision and cannot learn taxonomies from only unstructured noisy data.The surveyed approaches include order embeddings, hyperbolic embeddings, box-lattice structures, and Gaussian word embeddings.Source: §2 · p. 3
3 Methods
The method builds a weighted Hearst graph from text-extracted is-a relations, then jointly embeds its terms in hyperbolic space. It uses hierarchical geometry and entailment cones to represent possible parent-child relations while addressing sparsity and consistency challenges.
3.1 Hearst Graph
- Hearst patterns extract is-a relations between noun phrases, which are represented as a directed graph whose edge weights count extraction frequency.The graph is built from unstructured text using a limited set of patterns.Source: §3.1 · pp. 3–4
- Long-tailed pattern matches can leave the graph extremely sparse, while broader, low-precision patterns risk false positives, inconsistencies, and cycles.Many true relations may never appear in the corpus; polysemy can contribute to cycles.Source: §3.1 · p. 4
3.2 Hyperbolic Embeddings
- Hyperbolic space represents hierarchical structure through distance: points near the origin are relatively close to others, while points near the boundary are farther apart.The paper describes this locality behavior as corresponding to shortest paths in trees.Source: §3.2 · p. 4
- For each possible parent v, an entailment cone defines a region whose points are treated as children, with wider cones for more general points near the origin.The embedding jointly optimizes term representations to explain the observed Hearst patterns.Source: §3.2 · p. 5
- The method develops entailment cones in the Lorentz model, addressing the Poincaré ball’s numerical instability near its boundary, and trains with Riemannian SGD.The two models are equivalent, allowing optimization in Lorentz space and analysis or visualization in the Poincaré ball.Source: §3.2 · pp. 5–6
4 Experiments
The experiments evaluate hyperbolic embeddings of Hearst-pattern graphs on hypernymy benchmarks and WORDNET subtree reconstruction. HYPECONES achieves state-of-the-art results on seven of nine benchmarks and typically improves reconstruction over the best non-hyperbolic model by over 40%.
Hypernymy Tasks
- HYPECONES achieves state-of-the-art results on seven of nine benchmarks, with substantial gains on detection and directionality.On HYPERLEX, it outperforms other embedding-based approaches but does not match count-based methods.Source: Table 2 · p. 7, §4 · p. 8
Corpora and Preprocessing
- The final corpus contains circa 4.5M matched pairs, 431K unique pairs, and 243K unique terms; text was lemmatized and POS-tagged using CoreNLP 3.8.0.Source: §4 · p. 7
Hypernymy Tasks
- Detection uses five benchmarks, while directionality and graded entailment use BIBLESS and HYPERLEX; the WORDNET reconstruction covers animal, plant, and vehicle subtrees.Detection benchmarks include negative pairs such as co-hyponyms, antonyms, meronyms, and adjectival relations.Source: §4 · pp. 7–8
Reconstruction
- Reconstruction improves by typically over 40% relative to the best non-hyperbolic model, and transitivity helps recover missing relations.For example, the model infers (male horse, is-a, equine) from the relations to horse and from horse to equine.Source: Table 3 · p. 8, §4 · p. 8
Hypernymy Tasks
- HYPECONES uses 20 dimensions across tasks, compared with 300 dimensions for the best SVD-based results.The paper presents this as a reduction in parameters by over an order of magnitude.Source: §4 · p. 8
5 Conclusion
The paper combines Hearst patterns with hyperbolic embeddings to infer concept hierarchies from large text corpora. It reports state-of-the-art performance on commonly used hypernymy benchmarks, alongside more consistent and lower-dimensional embeddings.
Conclusion
- The approach combines Hearst patterns and hyperbolic embeddings to infer concept hierarchies from large text corpora.It jointly embeds terms to explain extracted Hearst patterns and introduces a Lorentz-model method for computing entailment cones.Source: §5 · p. 8
- The resulting embeddings achieve state-of-the-art performance on commonly used hypernymy benchmarks.The paper also reports improved consistency and substantially reduced dimensionality relative to SVD-based methods.Source: §5 · p. 8